Text segmentation method, device, electronic device and computer readable medium

Abstract

The embodiments of the present disclosure disclose a text segmentation method, device, electronic device and computer-readable medium. A specific implementation of the method includes: creating a first initial vocabulary instance; for each target text corresponding to the target task information, executing a vocabulary instance information replacement step: obtaining a vocabulary instance information sequence corresponding to a category information sequence, wherein each category information in the category information sequence is the category information corresponding to the target text; according to the order of each vocabulary instance information in the vocabulary instance information sequence, sequentially replacing the vocabulary instance information of the first initial vocabulary instance to segment the target text. This implementation can segment text quickly and efficiently.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a text segmentation method, device, electronic device, and computer-readable medium. Background Art

[0002] In the field of natural language, text segmentation can be to split a sentence or an article into individual words. Since there are many industry-specific words in various industries, it is often necessary to maintain multiple word libraries for more accurate word segmentation. For using multiple word libraries to segment the target text, the usual method is: during the scanning process of the target text, load the corresponding word library for different stages in the target text to perform word segmentation.

[0003] However, when using the above method to segment the target text, the following technical problems often occur:

[0004] First, when using a word library for word segmentation, it is often necessary to load the word library into memory. Since Chinese word libraries are generally large, and target text segmentation often requires loading multiple word libraries, there is a problem of long loading time and low efficiency of target text segmentation.

[0005] Second, loading multiple lexicons also takes up a lot of memory resources. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose a text segmentation method, an apparatus, an electronic device, and a computer-readable medium to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a text segmentation method, comprising: creating a first initial vocabulary instance; for each target text corresponding to the target task information, executing a vocabulary instance information replacement step: obtaining a vocabulary instance information sequence corresponding to a category information sequence, wherein each category information in the above category information sequence is the category information corresponding to the above target text; according to the order of each vocabulary instance information in the above vocabulary instance information sequence, replacing the vocabulary instance information of the above first initial vocabulary instance in sequence to segment the above target text.

[0009] Optionally, before executing the text segmentation step for each target text corresponding to the target task information, the method further includes: determining whether there is a target text to be segmented in the pre-acquired target text set; in response to determining that there is a target text to be segmented in the target text set, screening out at least one target text to be segmented from the target text set.

[0010] Optionally, the method further includes: performing data grouping processing on the at least one target text to obtain at least one target text group; establishing task information corresponding to each target text group in the at least one target text group to obtain a task information set.

[0011] Optionally, the above-mentioned screening out at least one target text to be segmented from the above-mentioned target text set includes: obtaining a historical text set that has been segmented within a target time period; determining the same texts between the above-mentioned target text set and the above-mentioned historical text set to obtain at least one same text; removing the above-mentioned at least one same text from the above-mentioned target text set to obtain a target text set after removal as the above-mentioned at least one target text.

[0012] Optionally, the target text is segmented by replacing the vocabulary instance information of the first initial vocabulary instance in sequence according to the order of each vocabulary instance information in the vocabulary instance information sequence, including: for each vocabulary instance information in the vocabulary instance information sequence, executing the following target text segmentation steps: changing the vocabulary instance information of the current first initial vocabulary instance to the above vocabulary instance information to obtain a changed first initial vocabulary instance; initializing the changed first initial vocabulary instance; and using the initialized first initial vocabulary instance to segment the corresponding sub-text in the target text.

[0013] Optionally, the above-mentioned obtaining of the vocabulary instance information sequence corresponding to the category information sequence includes: obtaining the vocabulary instance information sequence corresponding to the category information sequence from a target variable, wherein the above-mentioned target variable is a variable sent by the vocabulary instance processing end, and the above-mentioned target variable stores a vocabulary instance information set in a first preset key-value pair format, and each vocabulary instance information in the above-mentioned vocabulary instance information set has corresponding category information.

[0014] Optionally, the vocabulary instance information set in the above-mentioned target variable is generated through the following steps: creating an initial vocabulary instance; loading each word in at least one general vocabulary acquired in advance into the above-mentioned second initial vocabulary instance to obtain a vocabulary instance after the first loading; based on the above-mentioned vocabulary instance after the first loading, loading each word in at least one category vocabulary to be loaded into the corresponding vocabulary instance in the target container to obtain each vocabulary instance after the second loading, wherein the above-mentioned target container stores each vocabulary instance in a second preset key-value pair format; determining the vocabulary instance information of each of the above-mentioned vocabulary instances after the second loading to obtain the above-mentioned vocabulary instance information set.

[0015] Optionally, the above-mentioned method, based on the above-mentioned vocabulary instance after the first loading, loads each word in at least one category vocabulary to be loaded into the corresponding vocabulary instance in the target container to obtain each vocabulary instance after the second loading, including: for each word in the above-mentioned at least one category vocabulary to be loaded, performing the following word loading steps: determining the category information of the above-mentioned word; determining whether there is a vocabulary instance with the above-mentioned category information as the key in the above-mentioned target container; in response to determining that there is a vocabulary instance with the above-mentioned category information as the key in the above-mentioned target container, loading the above-mentioned word into the target vocabulary instance in the above-mentioned target container, wherein the above-mentioned target vocabulary instance is a vocabulary instance with the above-mentioned category information as the key; and determining each loaded vocabulary instance as the above-mentioned vocabulary instance after the second loading.

[0016] Optionally, after determining whether there is a vocabulary instance with the category information as the key in the target container, the method further includes: in response to determining that there is no vocabulary instance with the category information as the key in the target container, creating key-value content with the category information as the key and the vocabulary instance after the first loading as the value in the target container; and loading the word into the vocabulary instance after the first loading in the key-value content.

[0017] Optionally, the above-mentioned vocabulary instance information is information with category information as a key and a target bigram as a value, wherein the above-mentioned target bigram includes: word list information and word frequency list information.

[0018] In a second aspect, some embodiments of the present disclosure provide a text segmentation device, comprising: a creation unit, configured to create a first initial vocabulary instance; a creation unit, configured to create a first initial vocabulary instance; an execution unit, configured to, for each target text corresponding to the target task information, perform a text segmentation step: for each target text corresponding to the target task information, perform a vocabulary instance information replacement step: obtain a vocabulary instance information sequence corresponding to a category information sequence, wherein each category information in the above-mentioned category information sequence is the category information corresponding to the above-mentioned target text; according to the order of each vocabulary instance information in the above-mentioned vocabulary instance information sequence, replace the vocabulary instance information of the first initial vocabulary instance in sequence to segment the above-mentioned target text.

[0019] Optionally, the device further comprises: determining whether there is a target text to be segmented in the pre-acquired target text set; in response to determining that there is a target text to be segmented in the target text set, screening out at least one target text to be segmented from the target text set.

[0020] Optionally, the device further comprises: performing data grouping processing on the at least one target text to obtain at least one target text group; establishing task information corresponding to each target text group in the at least one target text group to obtain a task information set.

[0021] Optionally, the above-mentioned device also includes: obtaining a historical text set that has been segmented within a target time period; determining the identical texts between the above-mentioned target text set and the above-mentioned historical text set to obtain at least one identical text; removing the above-mentioned at least one identical text from the above-mentioned target text set to obtain a target text set after removal as the above-mentioned at least one target text.

[0022] Optionally, the execution unit can be configured to: obtain a vocabulary instance information sequence corresponding to the category information sequence from a target variable, wherein the target variable is a variable sent by a vocabulary instance processing end, and the target variable stores a vocabulary instance information set in a first preset key-value pair format, and each vocabulary instance information in the vocabulary instance information set has corresponding category information.

[0023] Optionally, the execution unit can be configured to: for each word library instance information in the above-mentioned word library instance information sequence, perform the following target text segmentation steps: change the word library instance information of the current first initial word library instance to the above-mentioned word library instance information to obtain the changed first initial word library instance; initialize the above-mentioned changed first initial word library instance; use the initialized first initial word library instance to segment the corresponding sub-text in the above-mentioned target text.

[0024] Optionally, the vocabulary instance information set in the above-mentioned target variable is generated through the following steps: creating an initial vocabulary instance; loading each word in at least one general vocabulary acquired in advance into the above-mentioned second initial vocabulary instance to obtain a vocabulary instance after the first loading; based on the above-mentioned vocabulary instance after the first loading, loading each word in at least one category vocabulary to be loaded into the corresponding vocabulary instance in the target container to obtain each vocabulary instance after the second loading, wherein the above-mentioned target container stores each vocabulary instance in a second preset key-value pair format; determining the vocabulary instance information of each of the above-mentioned vocabulary instances after the second loading to obtain the above-mentioned vocabulary instance information set.

[0025] Optionally, the loading unit is configured to: for each word in the at least one category vocabulary to be loaded, perform the following word loading steps: determine the category information of the word; determine whether there is a vocabulary instance with the category information as the key in the target container; in response to determining that there is a vocabulary instance with the category information as the key in the target container, load the word into the target vocabulary instance in the target container, wherein the target vocabulary instance is a vocabulary instance with the category information as the key; determine each loaded vocabulary instance as the above-mentioned vocabulary instance after the second loading.

[0026] Optionally, the above-mentioned device also includes: in response to determining that there is no vocabulary instance with the above-mentioned category information as the key in the above-mentioned target container, creating new key-value content with the above-mentioned category information as the key and the above-mentioned vocabulary instance after the first loading as the value in the above-mentioned target container; and loading the above-mentioned word into the above-mentioned vocabulary instance after the first loading in the above-mentioned key-value content.

[0027] Optionally, the above-mentioned vocabulary instance information includes: word list information corresponding to the vocabulary instance and word frequency list information corresponding to the vocabulary instance.

[0028] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0029] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.

[0030] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the text segmentation method of some embodiments of the present disclosure can segment the target text quickly and efficiently. Specifically, the reason for the inability to efficiently segment the target text is that the Chinese word library is generally large, and the target text segmentation often requires loading multiple word libraries, resulting in a long loading time and low efficiency of target text segmentation. In addition, loading multiple word libraries also takes up a lot of memory resources, resulting in low efficiency of subsequent word segmentation. Based on this, the text segmentation method of some embodiments of the present disclosure can first create a first initial word library instance as a basic word library instance for the generation of the first initial word library instance after subsequent changes. Then, for each target text corresponding to the target task information, the word library instance information replacement step is performed: the word library instance information sequence corresponding to the category information sequence is obtained, so as to be used for the subsequent replacement of the word library instance information of the first initial word library instance. Among them, each category information in the above-mentioned category information sequence is the category information corresponding to the above-mentioned target text. According to the order of each word library instance information in the above word library instance information sequence, the word library instance information of the first initial word library instance is replaced in turn to perform word segmentation on the above target text. Here, by changing the word library instance information, the first initial word library instance can be transformed into a word library instance corresponding to the category information. Without loading the word library instances corresponding to each category information, it is only necessary to transform the word library instance information of the first initial word library instance, and the corresponding word library instance can be used for subsequent word segmentation, which solves the problem of long loading time caused by loading multiple word libraries. It also solves the problem that loading multiple word libraries also occupies a large amount of memory resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0032] Figure 1 is a schematic diagram of an application scenario of a text segmentation method according to some embodiments of the present disclosure;

[0033] Figure 2 is a flowchart of some embodiments of the text segmentation method according to the present disclosure;

[0034] Figure 3 is a flowchart of some other embodiments of the text segmentation method according to the present disclosure;

[0035] Figure 4 is a schematic diagram of grouping at least one target text in some embodiments of the text segmentation method according to the present disclosure;

[0036] Figure 5 is a schematic diagram of the structure of some embodiments of the text segmentation device according to the present disclosure;

[0037] Figure 6 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0038] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0039] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0040] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0041] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0042] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0043] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0044] Figure 1 It is a schematic diagram of an application scenario of a text segmentation method according to some embodiments of the present disclosure.

[0045] exist Figure 1In the application scenario, the electronic device 101 can first create a first initial vocabulary instance 102. Then, for each target text corresponding to the target task information 103, perform the vocabulary instance information replacement step: the first step is to obtain the vocabulary instance information sequence 106 corresponding to the category information sequence 105. Among them, each category information in the above-mentioned category information sequence 105 is the category information corresponding to the above-mentioned target text 1041. In this application scenario, the text set 104 corresponding to the above-mentioned target task information 103 includes: target text 1041, target text 1042, and target text 1043. The above-mentioned category information sequence 105 includes: category information 1051, category information 1052, and category information 1053. The vocabulary instance information sequence 106 includes: vocabulary instance information 1061, vocabulary instance information 1062, and vocabulary instance information 1063. In the second step, the word instance information of the first initial word instance is replaced in sequence according to the order of each word instance information in the above word instance information sequence 106 to perform word segmentation on the above target text 1041. In this application scenario, the word instance information of the first initial word instance 102 is replaced with word instance information 1061 to obtain the first initial word instance 107. The word instance information 1061 of the first initial word instance 107 is replaced with word instance information 1062 to obtain the first initial word instance 108. The word instance information 1062 of the first initial word instance 108 is replaced with word instance information 1063 to obtain the first initial word instance 109. According to the first initial word instance 107, the first initial word instance 108 and the first initial word instance 109, the above target text 1041 is segmented.

[0046] It should be noted that the above-mentioned electronic device 101 can be hardware or software. When the electronic device is hardware, it can be implemented as a distributed cluster consisting of multiple server terminals or terminal devices, or it can be implemented as a single server terminal or a single terminal device. When the electronic device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0047] Here, the electronic device 101 may also be a task execution end (executor end) in the computing engine Spark, wherein Spark may be a fast and general computing engine designed for large-scale data processing.

[0048] It should be understood that Figure 1 The number of electronic devices in the embodiment is only for illustration. Any number of electronic devices may be provided according to implementation requirements.

[0049] Continue to refer Figure 2, shows a process 200 of some embodiments of the text segmentation method according to the present disclosure. The text segmentation method comprises the following steps:

[0050] Step 201: Create a first initial vocabulary instance.

[0051] In some embodiments, the execution body of the above-mentioned text segmentation method (for example Figure 1 The electronic device shown in the figure can create a first initial word library instance. The word library instance can be a dictionary tree (Trie tree). For example, the first initialized word library instance can be a Jieba word library instance.

[0052] As an example, the above execution subject can call the target interface to create a Jieba vocabulary instance.

[0053] Step 202: for each target text corresponding to the target task information, perform the word library instance information replacement step:

[0054] The target task information may be information characterizing the target task. For example, the target task information may be identification information of the target task. The target task information corresponds to a plurality of target texts. The target task corresponding to the target task information is a task of performing word segmentation processing on the plurality of target texts. As an example, the target text may be the title information of an item.

[0055] Step 2021, obtaining the vocabulary instance information sequence corresponding to the category information sequence.

[0056] In some embodiments, the execution subject may obtain the vocabulary instance information sequence corresponding to the category information sequence by wired or wireless means. Wherein, each category information in the category information sequence is the category information corresponding to the target text. There is one-to-one corresponding vocabulary instance information for each category information in the category information sequence, so there is a corresponding vocabulary instance information sequence for the category information sequence. Wherein, the category information may be three-level category information. The vocabulary instance information may be the key information of the vocabulary instance. For example, the vocabulary identification information of the vocabulary instance.

[0057] Here, for each category information in the category information sequence, corresponding vocabulary instance information is established respectively, so that the subsequent target text segmentation is more accurate and efficient.

[0058] As an example, the following table shows:

[0059] First category information First word library instance information Second category information Second vocabulary instance information Third category information Third word library instance information

[0060] It should be noted that there is at least one item information in the target text. The order in which the at least one item information appears in the target text is different. According to the order in which the at least one item information appears in the target text, the at least one item information is sorted to obtain an item information sequence. Since each item information has corresponding category information, the item information sequence has a corresponding category information sequence.

[0061] In some optional implementations of some embodiments, the execution subject may obtain the vocabulary instance information sequence corresponding to the category information sequence from the target variable. The target variable is a variable sent by the vocabulary instance processing end. The target variable stores the vocabulary instance information set in a first preset key-value pair format. Each vocabulary instance information in the vocabulary instance information set has corresponding category information.

[0062] Here, the target variable can be a variable in Spark. The target variable can distribute the data in the word library instance processing end to each Executor end. The word library instance processing end can be the driver end in Spark. The first preset key-value pair format can be a format with category information as the key and word library instance information as the value. The driver end can be the driver of Spark, which is the process of executing the main method in the development program.

[0063] Optionally, the vocabulary instance information set in the above target variable is generated by the following steps:

[0064] In the first step, the word library instance processing terminal may create a second initial word library instance, wherein the second initialization word library instance may be a Jieba word library instance.

[0065] In the second step, the above-mentioned word library instance processing terminal can load each word in the at least one general word library obtained in advance into the above-mentioned second initial word library instance to obtain the word library instance after the first loading. Among them, each general word library in the above-mentioned at least one general word library can be a general word library related to the target field. For example, for the e-commerce field, each word in the general word library can be a common word with a high frequency of use in the e-commerce field.

[0066] In the third step, the above-mentioned word library instance processing terminal can load each word in at least one category word library to be loaded into the corresponding word library instance in the target container through various methods according to the above-mentioned word library instance after the first loading, and obtain each word library instance after the second loading. Among them, the above-mentioned target container stores each word library instance in the second preset key-value pair format. The above-mentioned second preset key-value pair format can be a format with category information as the key and word library instance as the key.

[0067] As an example, the following table shows:

[0068] First category information First word library example Second category information Second vocabulary example Third category information Third vocabulary example

[0069] In the fourth step, the thesaurus instance processing terminal may determine the thesaurus instance information of each of the thesaurus instances after the second loading, and obtain the thesaurus instance information set.

[0070] As an example, the thesaurus instance processing end may determine the thesaurus instance information of each of the thesaurus instances after the second loading by querying, and obtain the thesaurus instance information set.

[0071] Optionally, the above method of loading each word in at least one category word library to be loaded into a corresponding word library instance in the target container according to the above first loaded word library instance to obtain each second loaded word library instance includes:

[0072] For each word in the at least one to-be-loaded category vocabulary, the following word loading steps are performed:

[0073] In the first sub-step, the vocabulary instance processing terminal may determine the category information of the words.

[0074] As an example, the database instance processing end may determine the category information of the word by querying.

[0075] In the second sub-step, the vocabulary instance processing terminal may determine whether there is a vocabulary instance with the category information as a key in the target container.

[0076] In the third sub-step, in response to determining that there is a word library instance with the category information as a key in the target container, the word library instance processing terminal may load the word into a target word library instance in the target container, wherein the target word library instance is a word library instance with the category information as a key.

[0077] In a fourth sub-step, the above-mentioned word library instance processing end may determine each loaded word library instance as each of the above-mentioned second loaded word library instances.

[0078] Optionally, after determining whether there is a vocabulary instance with the category information as a key in the target container, the step further includes:

[0079] In the first sub-step, in response to determining that there is no vocabulary instance with the category information as the key in the target container, the vocabulary instance processing end may create a key-value content with the category information as the key and the vocabulary instance after the first loading as the value in the target container.

[0080] As an example, the content stored in the above-mentioned target container includes {[first category information, first vocabulary instance], [second category information, second vocabulary instance], [third category information, third vocabulary instance]}. The above-mentioned category information may be fourth category information. Since there is no vocabulary instance with the above-mentioned category information as the key in the target, the above-mentioned vocabulary instance processing end may create a key-value content with the above-mentioned category information as the key and the above-mentioned vocabulary instance after the first loading as the value in the above-mentioned target container. That is, it can be obtained that the content stored in the target container includes {[first category information, first vocabulary instance], [second category information, second vocabulary instance], [third category information, third vocabulary instance], [fourth category information, vocabulary instance after the first loading]}.

[0081] In a second sub-step, the word library instance processing terminal may load the word into the word library instance after the first loading in the key value content.

[0082] Step 2022: According to the order of each word library instance information in the above word library instance information sequence, the word library instance information of the first initial word library instance is replaced in sequence to perform word segmentation on the above target text.

[0083] In some embodiments, the execution subject may replace the word library instance information of the first initial word library instance in sequence according to the order of each word library instance information in the word library instance information sequence to segment the target text. The first initial word library instance after replacing the word library instance information may segment part of the target text.

[0084] In some optional implementations of some embodiments, the step of replacing the word library instance information of the first initial word library instance in sequence according to the order of each word library instance information in the word library instance information sequence to segment the target text may include the following steps:

[0085] For each word library instance information in the above word library instance information sequence, the above execution subject may perform the following target text segmentation steps:

[0086] In a first sub-step, the execution subject may change the vocabulary instance information of the current first initial vocabulary instance to the vocabulary instance information to obtain the changed first initial vocabulary instance.

[0087] In a second sub-step, the execution subject may initialize the modified first initial vocabulary instance.

[0088] As an example, the execution entity may mark the initialized state (initialized) in the modified first initial vocabulary instance as true.

[0089] In the third sub-step, the execution subject may use the initialized first initial vocabulary instance to segment the corresponding sub-text in the target text.

[0090] It should be noted that the target text may be composed of various sub-texts, and the above-mentioned sub-texts may be divided according to the appearance positions of various item information in the target text.

[0091] In some optional implementations of some embodiments, the word library instance information includes: word list information corresponding to the word library instance and word frequency list information corresponding to the word library instance. Among them, the word library instance can be a dictionary tree. The word list information corresponding to the word library instance can be the address information of the word list corresponding to the dictionary tree. The last word list stores each word corresponding to the dictionary tree. The word frequency list information corresponding to the word library instance can be the address information of the word frequency list corresponding to the dictionary tree. The above-mentioned word frequency list stores the usage frequency of each word in the word list.

[0092] The above-mentioned embodiments of the present disclosure have the following beneficial effects: the text segmentation method of some embodiments of the present disclosure can segment the target text quickly and efficiently. Specifically, the reason for the inability to efficiently segment the target text is that the Chinese word library is generally large, and the target text segmentation often requires loading multiple word libraries, resulting in a long loading time and low efficiency of target text segmentation. In addition, loading multiple word libraries also takes up a lot of memory resources, resulting in low efficiency of subsequent word segmentation. Based on this, the text segmentation method of some embodiments of the present disclosure can first create a first initial word library instance as a basic word library instance for the generation of the first initial word library instance after subsequent changes. Then, for each target text corresponding to the target task information, the word library instance information replacement step is performed: the word library instance information sequence corresponding to the category information sequence is obtained, so as to be used for the subsequent replacement of the word library instance information of the first initial word library instance. Among them, each category information in the above-mentioned category information sequence is the category information corresponding to the above-mentioned target text. According to the order of each word library instance information in the above word library instance information sequence, the word library instance information of the first initial word library instance is replaced in turn to perform word segmentation on the above target text. Here, by changing the word library instance information, the first initial word library instance can be transformed into a word library instance corresponding to the category information. Without loading the word library instances corresponding to each category information, it is only necessary to transform the word library instance information of the first initial word library instance, and the corresponding word library instance can be used for subsequent word segmentation, which solves the problem of long loading time caused by loading multiple word libraries. It also solves the problem that loading multiple word libraries also occupies a large amount of memory resources.

[0093] Further references Figure 3 , shows a process 300 of another embodiment of the text segmentation method according to the present disclosure. The text segmentation method comprises the following steps:

[0094] Step 301 : Determine whether there is a target text to be segmented in the pre-acquired target text set.

[0095] In some embodiments, an execution entity (e.g. Figure 1 The electronic device shown) can determine whether there is a target text to be segmented in the pre-acquired target text set.

[0096] As an example, the execution subject may determine whether there is a target text to be segmented in the target text set according to the upload time point (or online time) of each target text in the target text set.

[0097] In response to determining that there is a target text in the target text set whose upload time is later than the target time point, the execution entity may determine that there is a target text to be segmented in the target text set.

[0098] In response to determining that there is a target text in the target text set whose upload time is earlier than the target time point, the execution entity may determine that there is no target text to be segmented in the target text set.

[0099] Step 302 , in response to determining that there is a target text to be segmented in the target text set, at least one target text to be segmented is selected from the target text set.

[0100] In some embodiments, in response to determining that there is a target text to be segmented in the target text set, the execution entity may filter out at least one target text to be segmented in the target text set.

[0101] As an example, the execution subject may filter out at least one target text uploaded later than the target time point from the target text set as the at least one target text to be segmented.

[0102] In some optional implementations of some embodiments, the step of selecting at least one target text to be segmented from the target text set may include the following steps:

[0103] In the first step, the execution subject may obtain a historical text set that has been segmented within a target time period. The target time period may be a time period before the corresponding time of the target text set. The corresponding time of the target text set may be the time when the execution subject obtains the target text set.

[0104] In the second step, the execution subject may determine the same texts between the target text set and the historical text set, and obtain at least one same text;

[0105] In a third step, the execution subject may remove the at least one identical text from the target text set to obtain a target text set after removal as the at least one target text.

[0106] Step 303: performing data grouping processing on the at least one target text to obtain at least one target text group.

[0107] In some embodiments, the execution entity may perform data grouping processing on the at least one target text to obtain at least one target text group.

[0108] As an example, the execution entity may utilize a consistent hashing algorithm to perform data grouping processing on the at least one target text to obtain at least one target text group.

[0109] As another example, the execution subject may evenly group at least one target text to obtain at least one target text group, wherein the number of target texts in each target text group is the same.

[0110] Step 304: Establish task information corresponding to each target text group in the at least one target text group to obtain a task information set.

[0111] In some embodiments, the execution subject may establish task information corresponding to each target text group in the at least one target text group to obtain a task information set, wherein the task information set includes the target task information, that is, the target task information is any task information in the task information set.

[0112] like Figure 4 As shown, at least one target text 401 may include: target text 4011, target text 4012, target text 4013, target text 4014, target text 4015, target text 4016. At least one target text group includes: target text group 402, target text group 403. Target text group 402 includes: target text 4011, target text 4012, target text 4013. Target text 403 includes: target text 4014, target text 4015, target text 4016. The task information corresponding to target text group 402 may be first task information 404. The task information corresponding to target text group 403 may be first task information 405.

[0113] Step 305: Create a first initial vocabulary instance.

[0114] Step 306: for each target text corresponding to the target task information, perform the word library instance information replacement step:

[0115] Step 3061, obtaining the vocabulary instance information sequence corresponding to the category information sequence.

[0116] Step 3062: According to the order of each word library instance information in the above word library instance information sequence, the word library instance information of the first initial word library instance is replaced in sequence to perform word segmentation on the above target text.

[0117] In some embodiments, the specific implementation of steps 301-306 and the technical effects thereof can be referred to in Figure 2 The steps 201-202 in the corresponding embodiment are not described in detail here.

[0118] from Figure 3 It can be seen that Figure 2 Compared with the description of some corresponding embodiments, Figure 3 The process 300 of the text segmentation method in some corresponding embodiments further highlights the specific steps of screening out at least one target text to be segmented and grouping at least one target text. Therefore, the scheme described in these embodiments performs segmentation on at least one target text to be segmented through multi-tasking, which greatly improves the efficiency of segmentation and reduces the waste of computing resources.

[0119] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a text segmentation device. These device embodiments are similar to Figure 2 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0120] like Figure 5 As shown, a text segmentation device 500 includes: a creation unit 501 and an execution unit 503. The creation unit 501 is configured to create a first initial word library instance; the execution unit 502 is configured to perform a text segmentation step for each target text corresponding to the target task information: for each target text corresponding to the target task information, perform a word library instance information replacement step: obtain a word library instance information sequence corresponding to a category information sequence, wherein each category information in the above category information sequence is the category information corresponding to the above target text; according to the order of each word library instance information in the above word library instance information sequence, replace the word library instance information of the first initial word library instance in sequence to segment the above target text.

[0121] In some optional implementations of some embodiments, the apparatus 500 further includes: a determination unit and a screening unit (not shown in the figure). The determination unit may be configured to: determine whether there is a target text to be segmented in the pre-acquired target text set. The screening unit may be configured to: in response to determining that there is a target text to be segmented in the target text set, screen out at least one target text to be segmented from the target text set.

[0122] In some optional implementations of some embodiments, the apparatus 500 further includes: a group processing unit and an establishment unit (not shown). The group processing unit may be configured to: perform data group processing on the at least one target text to obtain at least one target text group. The establishment unit may be configured to: establish task information corresponding to each target text group in the at least one target text group to obtain a task information set.

[0123] In some optional implementations of some embodiments, the screening unit in the above-mentioned device 500 can be further configured to: obtain a historical text set that has been segmented within a target time period; determine the identical texts between the above-mentioned target text set and the above-mentioned historical text set to obtain at least one identical text; remove the above-mentioned at least one identical text from the above-mentioned target text set to obtain a target text set after removal as the above-mentioned at least one target text.

[0124] In some optional implementations of some embodiments, the acquisition unit in the above-mentioned device 500 can be further configured to: obtain a vocabulary instance information sequence corresponding to the category information sequence from the target variable, wherein the above-mentioned target variable is a variable sent by the vocabulary instance processing end, and the above-mentioned target variable stores a vocabulary instance information set in a first preset key-value pair format, and each vocabulary instance information in the above-mentioned vocabulary instance information set has corresponding category information.

[0125] In some optional implementations of some embodiments, the word segmentation unit in the above-mentioned device 500 can be further configured to: for each word library instance information in the above-mentioned word library instance information sequence, perform the following target text segmentation steps: change the word library instance information of the current first initial word library instance to the above-mentioned word library instance information to obtain the changed first initial word library instance; initialize the above-mentioned changed first initial word library instance; use the initialized first initial word library instance to segment the corresponding sub-text in the above-mentioned target text.

[0126] In some optional implementations of some embodiments, the vocabulary instance information set in the above-mentioned target variable is generated by the following steps: creating a second initial vocabulary instance; loading each word in at least one general vocabulary acquired in advance into the above-mentioned second initial vocabulary instance to obtain a vocabulary instance after the first loading; according to the above-mentioned vocabulary instance after the first loading, loading each word in at least one category vocabulary to be loaded into the corresponding vocabulary instance in the target container to obtain each vocabulary instance after the second loading, wherein the above-mentioned target container stores each vocabulary instance in a second preset key-value pair format; determining the vocabulary instance information of each vocabulary instance after the second loading to obtain the above-mentioned vocabulary instance information set.

[0127] In some optional implementations of some embodiments, the loading unit in the above-mentioned device 500 can be configured to: for each word in the above-mentioned at least one category vocabulary to be loaded, perform the following word loading steps: determine the category information of the above-mentioned word; determine whether there is a vocabulary instance with the above-mentioned category information as the key in the above-mentioned target container; in response to determining that there is a vocabulary instance with the above-mentioned category information as the key in the above-mentioned target container, load the above-mentioned word into the target vocabulary instance in the above-mentioned target container, wherein the above-mentioned target vocabulary instance is a vocabulary instance with the above-mentioned category information as the key; determine each loaded vocabulary instance as the above-mentioned each vocabulary instance after the second loading.

[0128] In some optional implementations of some embodiments, the device 500 further includes: a key-value content creation unit and a word loading unit (not shown in the figure). The key-value content creation unit may be configured to: in response to determining that there is no word library instance with the category information as the key in the target container, create a key-value content with the category information as the key and the first loaded word library instance as the value in the target container. The word loading unit may be configured to: load the word into the first loaded word library instance in the key-value content.

[0129] In some optional implementations of some embodiments, the above-mentioned vocabulary instance information includes: word list information corresponding to the vocabulary instance and word frequency list information corresponding to the vocabulary instance.

[0130] In some optional implementations of some embodiments, the above-mentioned device 500 further includes: It can be understood that the units recorded in the device 500 are the same as those in the reference Figure 2 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 500 and the units included therein, and will not be described in detail here.

[0131] Reference below Figure 6 , which shows an electronic device (eg, Figure 1A schematic diagram of the structure of an electronic device (electronic device in FIG. 600). Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0132] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0133] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0134] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0135] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0136] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0137] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: creates a first initial vocabulary instance; for each target text corresponding to the target task information, performs a vocabulary instance information replacement step: obtains a vocabulary instance information sequence corresponding to a category information sequence, wherein each category information in the category information sequence is the category information corresponding to the target text; replaces the vocabulary instance information of the first initial vocabulary instance in sequence according to the order of each vocabulary instance information in the vocabulary instance information sequence to segment the target text.

[0138] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0140] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes a creation unit and an execution unit. The names of these units do not constitute a limitation on the units themselves in some cases, for example, the creation unit may also be described as a "unit for creating a first initial vocabulary instance".

[0141] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0142] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A text segmentation method, comprising: Creating a first initial vocabulary instance; For each target text corresponding to the target task information, perform the vocabulary instance information replacement step: Acquire a vocabulary instance information sequence corresponding to a category information sequence, wherein each category information in the category information sequence is the category information corresponding to the target text; According to the order of each word library instance information in the word library instance information sequence, the word library instance information of the first initial word library instance is replaced in sequence to segment the target text, including: for each word library instance information in the word library instance information sequence, the following target text segmentation steps are performed: the word library instance information of the current first initial word library instance is changed to the word library instance information to obtain a changed first initial word library instance; initialization processing is performed on the changed first initial word library instance; and the initialized first initial word library instance is used to segment the corresponding subtext in the target text.

2. The method according to claim 1, wherein: Before performing the replacement of the vocabulary instance information for each target text corresponding to the target task information, the method further includes: Determine whether there is a target text to be segmented in the pre-acquired target text set; In response to determining that there is a target text to be segmented in the target text set, at least one target text to be segmented is selected from the target text set.

3. The method according to claim 2, wherein: The method further comprises: Performing data grouping processing on the at least one target text to obtain at least one target text group; Task information corresponding to each target text group in the at least one target text group is established to obtain a task information set.

4. The method according to claim 2, wherein: The step of selecting at least one target text to be segmented from the target text set includes: Obtain a historical text set that has been segmented within the target time period; Determine the same texts between the target text set and the historical text set to obtain at least one same text; The at least one identical text is removed from the target text set to obtain a target text set after removal as the at least one target text.

5. The method according to claim 1, wherein: The step of obtaining the vocabulary instance information sequence corresponding to the category information sequence includes: A vocabulary instance information sequence corresponding to a category information sequence is obtained from a target variable, wherein the target variable is a variable sent by a vocabulary instance processing end, and the target variable stores a vocabulary instance information set in a first preset key-value pair format, and each vocabulary instance information in the vocabulary instance information set has corresponding category information.

6. The method according to claim 5, wherein: The vocabulary instance information set in the target variable is generated by the following steps: Creating a second initial vocabulary instance; Loading each word in at least one pre-acquired general vocabulary into the second initial vocabulary instance to obtain a vocabulary instance after the first loading; According to the first loaded vocabulary instance, each word in at least one to-be-loaded category vocabulary is loaded into a corresponding vocabulary instance in a target container to obtain each second loaded vocabulary instance, wherein the target container stores each vocabulary instance in a second preset key-value pair format; The vocabulary instance information of each vocabulary instance after the second loading is determined to obtain the vocabulary instance information set.

7. The method according to claim 6, wherein: The step of loading each word in at least one category word library to be loaded into a corresponding word library instance in a target container according to the first loaded word library instance to obtain each second loaded word library instance includes: For each word in the at least one to-be-loaded category vocabulary, the following word loading steps are performed: Determine category information of the word; Determine whether there is a word library instance with the category information as a key in the target container; In response to determining that a word library instance with the category information as a key exists in the target container, loading the word into a target word library instance in the target container, wherein the target word library instance is a word library instance with the category information as a key; The loaded vocabulary instances are determined as the second loaded vocabulary instances.

8. The method according to claim 7, wherein: After determining whether there is a word library instance with the category information as a key in the target container, the method further includes: In response to determining that the target container does not have a word library instance with the category information as a key, creating a key-value content with the category information as a key and the first loaded word library instance as a value in the target container; The word is loaded into the first loaded word library instance in the key value content.

9. The method according to claim 1, wherein: The word library instance information includes: word list information corresponding to the word library instance and word frequency list information corresponding to the word library instance.

10. A text segmentation device, comprising: A creating unit configured to create a first initial vocabulary instance; The execution unit is configured to perform a text segmentation step for each target text corresponding to the target task information: for each target text corresponding to the target task information, perform a vocabulary instance information replacement step: obtain a vocabulary instance information sequence corresponding to the category information sequence, wherein each category information in the category information sequence is the category information corresponding to the target text; replace the vocabulary instance information of the first initial vocabulary instance in sequence according to the order of each vocabulary instance information in the vocabulary instance information sequence to segment the target text, including: for each vocabulary instance information in the vocabulary instance information sequence, perform the following target text segmentation step: change the vocabulary instance information of the current first initial vocabulary instance to the vocabulary instance information to obtain the changed first initial vocabulary instance; initialize the changed first initial vocabulary instance; and use the initialized first initial vocabulary instance to segment the corresponding subtext in the target text.

11. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

12. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images