Word segmentation method, device, electronic device and readable storage medium

Abstract

The present application provides a word segmentation method, device, electronic device and readable storage medium, and relates to the field of computer technology. The method comprises: obtaining a candidate word segmentation set corresponding to each character in the text to be segmented from the text to be segmented, wherein the candidate word segmentation set corresponding to any character includes a candidate word segmentation with the character as the starting character, and when multiple characters are included in the candidate word segmentation, the multiple characters are continuous characters in the text to be segmented; according to the candidate word segmentation set corresponding to each character, through dynamic programming, with the maximum product of the string lengths of each candidate word segmentation included in the candidate word segmentation result as the goal, traverse each character in the text to be segmented in order from back to front, obtain the candidate word segmentation result, and then determine the target word segmentation result, and the sum of the candidate word segmentations in any candidate word segmentation result constitutes the text to be segmented. In this way, there is no need to use a training set to train the word segmentation model, and the accuracy of the word segmentation result can be guaranteed at the same time.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a word segmentation method, device, electronic device and readable storage medium. Background Art

[0002] Word segmentation is very important in information retrieval, text mining and other fields. Currently, the commonly used word segmentation methods include model-based word segmentation. Model-based word segmentation has the advantage of good fitting, but it is often unreliable when used in contextual scenarios not included in the training set, and the corpus annotation requirements are high. Summary of the invention

[0003] The embodiments of the present application provide a word segmentation method, device, electronic device and readable storage medium, which can train the word segmentation model without using a training set while ensuring the accuracy of the word segmentation results, and have the characteristics of good effect and low cost.

[0004] The embodiments of the present application can be implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a word segmentation method, the method comprising:

[0006] From the text to be segmented, a candidate segmentation set corresponding to each character in the text to be segmented is obtained, wherein the candidate segmentation set corresponding to any character includes a candidate segmentation with the character as the starting character, and when the candidate segmentation includes multiple characters, the multiple characters are consecutive characters in the text to be segmented;

[0007] According to the candidate segmentation set corresponding to each character, through dynamic programming, with the goal of maximizing the product of the string lengths of each candidate segmentation included in the candidate segmentation result, traverse each character in the to-be-segmented text in a back-to-front order to obtain the candidate segmentation result, wherein the sum of the candidate segmentations in any one of the candidate segmentation results constitutes the to-be-segmented text;

[0008] A target word segmentation result is determined based on the obtained candidate word segmentation results.

[0009] In a second aspect, an embodiment of the present application provides a word segmentation device, the device comprising:

[0010] A word segmentation set obtaining module is used to obtain, from a text to be segmented, a candidate word segmentation set corresponding to each character in the text to be segmented, wherein the candidate word segmentation set corresponding to any character includes a candidate word segmentation with the character as a starting character, and when the candidate word segmentation includes multiple characters, the multiple characters are consecutive characters in the text to be segmented;

[0011] A processing module is used to traverse each character in the to-be-segmented text in a backward order according to a candidate segmentation set corresponding to each character by a dynamic programming method, with the goal of maximizing the product of the string lengths of each candidate segmentation included in the candidate segmentation result, to obtain the candidate segmentation result, wherein the sum of the candidate segmentations in any one of the candidate segmentation results constitutes the to-be-segmented text;

[0012] The result determination module is used to determine the target word segmentation result according to the obtained candidate word segmentation result.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the word segmentation method described in the aforementioned embodiment.

[0014] In a fourth aspect, an embodiment of the present application provides a readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the word segmentation method as described in the aforementioned embodiment is implemented.

[0015] The word segmentation method, device, electronic device and readable storage medium provided by the embodiment of the present application can firstly obtain a candidate word segmentation set corresponding to each character in the text to be segmented based on the text to be segmented, wherein the candidate word segmentation set corresponding to any character includes a candidate word segmentation with the character as the starting character, and when the candidate word segmentation includes multiple characters, the multiple characters are continuous characters in the text to be segmented; then, according to the obtained candidate word segmentation set corresponding to each character, through dynamic programming, with the maximum product of the string lengths of each candidate word segmentation included in the candidate word segmentation result as the goal, traverse each character in the text to be segmented in order from back to front, so as to obtain the candidate word segmentation result, wherein the sum of the candidate word segmentations in any candidate word segmentation result constitutes the above-mentioned text to be segmented; finally, the target word segmentation result can be obtained according to the above-mentioned candidate word segmentation result. Thus, a word segmentation result with better word segmentation effect can be obtained without using a training set to train the word segmentation model, and the word segmentation effect can still be guaranteed when applied in a vertical search scenario. This method has the characteristics of good effect and low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A block diagram of an electronic device provided in an embodiment of the present application;

[0018] Figure 2 A flowchart of a word segmentation method provided in an embodiment of the present application;

[0019] Figure 3 for Figure 2 A schematic flow chart of the sub-steps included in step S120;

[0020] Figure 4 for Figure 2 A schematic flow chart of a sub-step included in step S130;

[0021] Figure 5 for Figure 2 A schematic flow chart of another sub-step included in step S130;

[0022] Figure 6 One of the block diagrams of the word segmentation device provided in the embodiment of the present application;

[0023] Figure 7 The second block diagram of the word segmentation device provided in the embodiment of the present application.

[0024] Icon: 100 - electronic device; 110 - memory; 120 - processor; 130 - communication unit; 200 - word segmentation device; 201 - vocabulary generation module; 210 - word segmentation set acquisition module; 220 - processing module; 230 - result determination module. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0026] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0027] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0028] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0029] In addition to model-based word segmentation, word segmentation can also be performed based on statistics. Statistical word segmentation requires a word library, which includes word segmentation and word frequency. First, all possible word segmentation combinations are found from the text, and the global optimal word segmentation combination is obtained by calculating statistical values ​​with the help of greedy thinking, dynamic programming thinking, etc. This is more suitable for word segmentation of short sentences in vertical business.

[0030] Jieba word segmentation is a statistical word segmentation method. Jieba word segmentation is a relatively common open source word segmentation tool. It is a dynamic programming word segmentation tool that aims to maximize the joint probability of sequences. It assumes that no matter how they are segmented, any token is independent of each other, but this assumption is obviously not reasonable. In addition, this method requires a word list with carefully formulated word frequencies. Since the calculation rules of word frequencies are complex and special, the word list is not easy to expand. In vertical search scenarios, Jieba word segmentation is not ideal because a large number of domain word sets (such as albums, sounds, and anchor names) are not included in the word list.

[0031] In order to alleviate the current situation of unsatisfactory word segmentation effect and high cost of word segmentation method, the embodiments of the present application provide a word segmentation method, device, electronic device and readable storage medium, using the idea of ​​dynamic programming, and selecting the "most even segmentation" quantization value as the optimization target in the iterative formula. The set of segmentation schemes with the largest target value is the global optimal solution. In this way, there is no need to spend a lot of cost on training to obtain the word segmentation model. At the same time, word segmentation can be achieved by using a word list without word frequency, thereby greatly reducing the cost of the word list and can be replaced at any time according to the scene, solving the traditional drawbacks of the major open source word segmentation scenarios. This method has the characteristics of low cost, wide applicability and good effect.

[0032] The word segmentation method provided in this application is a dynamic programming algorithm with the quantitative value of "most even segmentation" as the optimization target. "Most even segmentation" aims to make each segmentation combination (i.e., word segmentation) in the optimal token as close to 3 as possible. It can be proved that when a positive integer is split into several positive integers, the solution with the largest product of the number of splits in all the segmentation combinations will definitely not split 1. Furthermore, if 1 is not split, 3 must be split as much as possible to maximize the product. In fact, according to the semantic characteristics of Chinese words, words with a length of 2 and 3 are already able to clearly express the meaning, and word segmentation processing often hopes to split each semantic component (note that the word segmentation here does not include the named body recognition task), so it must be a good segmentation solution to split short sentences into Chinese words with a length of 2 to 4 characters as much as possible.

[0033] If the text "Guo Degang Yu Qian's Cross Talk Selection" can be split into the following two possibilities (spaces represent splits), the product of the lengths of the tokens of "Guo Degang Yu Qian's Cross Talk Selection" is 20, while the product of "Guo Degang Yu Qian's Cross Talk Selection" is 24. Obviously, the latter has a larger product value and a better splitting scheme. Such an optimization goal is very suitable for the search field, because when building an index, the search needs to split the text as finely as possible while ensuring that the token has semantics. Of course, it is understandable that it can also be applied to other fields.

[0034] Please refer to Figure 1 , Figure 1 A block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be, but is not limited to, a smart phone, a computer, a server, etc. The electronic device 100 may include a memory 110, a processor 120, and a communication unit 130. The memory 110, the processor 120, and the communication unit 130 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines.

[0035] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0036] The processor 120 is used to read / write data or programs stored in the memory 110 and execute corresponding functions. For example, the memory 110 stores a word segmentation device 200, and the word segmentation device 200 includes at least one software function module that can be stored in the memory 110 in the form of software or firmware. The processor 120 executes various functional applications and data processing by running software programs and modules stored in the memory 110, such as the word segmentation device 200 in the embodiment of the present application, that is, implementing the word segmentation method in the embodiment of the present application.

[0037] The communication unit 130 is used to establish a communication connection between the electronic device 100 and other communication terminals through a network, and to send and receive data through the network.

[0038] It should be understood that Figure 1 The structure shown is only a schematic diagram of the structure of the electronic device 100. The electronic device 100 may also include Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown. Figure 1 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0039] Please refer to Figure 2 , Figure 2 A flow chart of a word segmentation method provided in an embodiment of the present application. The method can be applied to the electronic device 100. The method can be applied to the search field, or other fields that require word segmentation processing. The specific process of the word segmentation method is described in detail below. In this embodiment, the method may include steps S110 to S130.

[0040] Step S110, obtaining a candidate word segmentation set corresponding to each character in the text to be segmented from the text to be segmented.

[0041] In this embodiment, the text to be segmented is a text that needs to be segmented, which can be determined in combination with actual needs. The text to be segmented can be a Chinese text or a text in other languages. The text to be segmented includes multiple characters, and for each character, a candidate segmentation set corresponding to each character can be obtained from the text to be segmented. Among them, the candidate segmentation set corresponding to any one of the characters includes at least one candidate segmentation with the character as the starting character. That is, the candidate segmentation set corresponding to each position in the text to be segmented is obtained, and the candidate segmentation set corresponding to a position includes a candidate segmentation with the character at that position as the starting character.

[0042] One candidate word segmentation may include one or more characters. When the candidate word segmentation includes multiple characters, the positions of the multiple characters in the text to be segmented are continuous, that is, the multiple characters are continuous characters in the text to be segmented.

[0043] Optionally, as a possible implementation method, a preset word library may be obtained in advance, and the preset word library includes a plurality of preset word segmentations. The method for obtaining the preset word library may be determined in combination with actual needs. For example, the word library of other word segmenters may be used as the preset word library. Alternatively, only the word segmentations included in the word library of other word segmenters may be used to form the preset word library; or a plurality of preset word segmentations may be obtained from a preset data source to form the preset word library, and the method for obtaining the plurality of preset word segmentations may be based on manual annotation, etc. In this way, the word library may not have word frequency, which is convenient for expansion, the cost of the word list may be greatly reduced, and it may be replaced at any time according to the scenario.

[0044] According to the preset word library, for each character in the text to be segmented, at least one character string starting with the character and the same as one of the preset word segments can be found from the text to be segmented as the candidate word segmentation corresponding to the character. For any character in the text to be segmented, a specific character string is found from the text to be segmented as the candidate word segmentation corresponding to the character, and the specific character string starts with the character and is the same as one of the preset word segments. In this way, it is convenient to quickly obtain a candidate word segmentation set corresponding to each character in the text to be segmented according to the preset word library, and the candidate word segments in the candidate word segmentation set can all form words.

[0045] It is understandable that the above is only an example of a method for obtaining a candidate word segmentation set, and the candidate word segmentation set can also be obtained in other ways.

[0046] Step S120, according to the candidate word segmentation set corresponding to each character, through dynamic programming, with the goal of maximizing the product of the string lengths of each candidate word segmentation included in the candidate word segmentation result, traverse each character in the to-be-segmented text in a backward order to obtain the candidate word segmentation result.

[0047] Based on the idea of ​​dynamic programming, the character string length product of each candidate word segmentation included in the candidate word segmentation result can be maximized, and the characters in the text to be segmented can be traversed in order from back to front. During the traversal process, based on the candidate word segmentation set corresponding to the traversed characters, the optimal local word segmentation result corresponding to the position of the traversed characters in the text to be segmented is obtained. After traversing all characters, the optimal local word segmentation result corresponding to the last traversed position (that is, the position of the last traversed character, which can also be called the position of the first character in the text to be segmented) can be used as the candidate word segmentation result. Among them, the sum of the candidate word segmentations in any one of the candidate word segmentation results constitutes the text to be segmented.

[0048] In this embodiment, Figure 3 The method shown above obtains the optimal local word segmentation result corresponding to a traversed position. Please refer to Figure 3 , Figure 3 for Figure 2 Schematic diagram of the flow of sub-steps included in step S120. In this embodiment, step S120 may include sub-steps S121 to S122.

[0049] Sub-step S121, traverses the candidate participles in the candidate participle set corresponding to the traversed current character, and obtains the current candidate partial participle result and the current candidate partial product corresponding to the current candidate participle according to the string length of the current candidate participle corresponding to the traversed current character, and the optimal partial participle result and the maximum partial product corresponding to the target position.

[0050] The currently traversed character can be used as the current character, and for the current character, the candidate participles included in the candidate participle set corresponding to the current character can be traversed. The currently traversed candidate participle can be used as the current candidate participle, and the current candidate partial participle result and the current candidate partial product corresponding to the current candidate participle can be obtained according to the character string length of the current candidate participle, the optimal partial participle result and the maximum partial product corresponding to the target position.

[0051] The target position is the first position after the last character of the current candidate word segmentation in the text to be segmented. The target position is the position that has been traversed when traversing the characters. Assuming that the position of the current character in the text to be segmented is i, and the string length of the current candidate word segmentation is length, the target position is i+length. i decreases as the characters are traversed from back to front.

[0052] The maximum local product corresponding to the target position is the product of the string lengths of the candidate word segments included in the optimal local word segmentation result corresponding to the maximum local product. A position may correspond to one local optimal solution or multiple local optimal solutions, and the number of local optimal solutions is determined by the actual situation. A local optimal solution may include an optimal local word segmentation result and a maximum local product, and the maximum local product is the product of the string lengths of the candidate word segments included in the optimal local word segmentation result.

[0053] Among them, when i+length is greater than the maximum number of positions of the text to be segmented, for example, when the position of the last character of the current candidate segmentation in the text to be segmented is the end position of the text to be segmented, the last position of the text to be segmented can be used as the target position. When calculating the optimal local segmentation result and the maximum local product corresponding to the last position in the text to be segmented, that is, when calculating the optimal local segmentation result and the maximum local product corresponding to the first character in the process of traversing the characters, the character can be used as the optimal local segmentation result corresponding to itself, and the product of an initial value and the character string length 1 is used as the maximum local product corresponding to the character.

[0054] By the above traversal processing of the candidate participles in the candidate participle set corresponding to the current character, the current candidate local participle results and the current candidate local products corresponding to each candidate participle in the candidate participle set corresponding to the current character can be obtained.

[0055] Sub-step S122, taking the largest current candidate local product as the current maximum local product corresponding to the current character position, and taking the current candidate local word result corresponding to the current maximum local product as the current optimal local word result corresponding to the current character position.

[0056] When the current candidate partial word segmentation results and the current candidate partial products corresponding to each candidate word segmentation in the candidate word segmentation set corresponding to the current character are obtained, the largest current candidate partial product can be determined by comparison. The largest current candidate partial product can be used as the current maximum partial product corresponding to the position of the current character, that is, as the optimal partial word segmentation result corresponding to the currently traversed character position. The current candidate partial word segmentation result corresponding to the current maximum partial product is used as the current optimal partial word result corresponding to the position of the current character, wherein the product of the string lengths of the candidate word segments included in the current candidate partial word segmentation result corresponding to the current maximum partial product is the current maximum partial product.

[0057] Each position in the text to be segmented can be traversed in order from back to front. When a position is traversed, the above sub-steps S121 to S122 are executed according to the character corresponding to the position, so as to obtain the optimal local segmentation result and the maximum local product corresponding to the position. When the last character in the text to be segmented is traversed, the optimal local segmentation result obtained at this time can be used as the candidate segmentation result. That is, the optimal local segmentation result corresponding to the position of the first character in the text to be segmented is the candidate segmentation result.

[0058] The above dynamic programming process can be expressed by the following iterative formula:

[0059]

[0060] Among them, dp[i] represents the maximum local product corresponding to the i-th position in the text to be segmented, and length k Indicates the string length of the kth candidate word in the candidate word set corresponding to the character at the i-th position. Since the characters are traversed from back to front, i decreases with iteration.

[0061] This formula means: when looping from large to small to i, assuming that the character at position i is the starting character, there are m segmentation candidates (that is, the candidate segmentation set corresponding to the character at position i includes m candidate segmentations), and the length of each candidate string is expressed as length k It means that length k The local optimal solution dp[i+length k ] are multiplied together, and the candidate with the largest product is the local optimal solution dp[i] of state i.

[0062] Step S130: determining a target word segmentation result according to the obtained candidate word segmentation results.

[0063] Optionally, as a possible implementation, when the candidate word segmentation results are obtained, all the obtained candidate word segmentation results can be directly used as the target word segmentation results. In this way, the target word segmentation results can be quickly obtained.

[0064] Optionally, as another possible implementation, when there are multiple candidate word segmentation results, Figure 4 The target word segmentation result is obtained in the manner shown. Figure 4 , Figure 4 for Figure 2 Schematic diagram of a flow chart of a sub-step included in step S130. In this embodiment, step S130 may include sub-steps S131 to S132.

[0065] Sub-step S131, obtaining the word segmentation granularity of each of the candidate word segmentation results.

[0066] Sub-step S132, selecting the candidate word segmentation result corresponding to the minimum word segmentation granularity as the target word segmentation result.

[0067] In this embodiment, the word segmentation granularity of each candidate word segmentation result can be determined according to the number of candidate word segmentations included in the candidate word segmentation result and / or the character string length of each candidate word segmentation. The word segmentation granularity is used to indicate the granularity of the word segmentation splitting. That is, the word segmentation granularity of a candidate word segmentation result is used to indicate the granularity of the word segmentation splitting method represented by the candidate word segmentation result. For example, the word segmentation granularity is determined according to the number of candidate word segmentations included in the candidate word segmentation result, and the word segmentation granularity is inversely correlated with the number of candidate word segmentations included in the candidate word segmentation result.

[0068] Then, the minimum word segmentation granularity can be determined by comparison, and the candidate word segmentation result corresponding to the minimum word segmentation granularity can be used as the target word segmentation result. In this way, the candidate word segmentation result with finer granularity can be used as the target word segmentation result.

[0069] For example, the inverse of the number of candidate segmentation words included in the candidate segmentation result is used as the segmentation granularity of the candidate segmentation result, so that the candidate segmentation result with the largest number of candidate segmentation words can be used as the target segmentation result.

[0070] Optionally, as another possible implementation, when there are multiple candidate word segmentation results, Figure 5 The target word segmentation result is obtained in the manner shown. Figure 5 , Figure 5 for Figure 2 Schematic diagram of another sub-step included in step S130. In this embodiment, step S130 may include sub-steps S135 to S136.

[0071] Sub-step S135, according to the preset segmentations and word frequencies included in the preset word library, obtain the sum of the word frequencies of the segmentations included in each of the candidate segmentation results as the total number of word frequencies corresponding to each of the candidate segmentation results.

[0072] Sub-step S136, selecting the candidate word segmentation result corresponding to the maximum total word frequency as the target word segmentation result.

[0073] In this embodiment, the preset word library includes a plurality of preset word segments and word frequencies corresponding to each preset word segment. The preset word library may be a word library of a conventional word segmenter, or a word library obtained by other means. Optionally, when a word library is obtained by other means as the preset word library, the electronic device 100 may obtain the word library in advance based on the other means, or other devices may obtain the word library in advance by the other means, and send the word library to the electronic device 100 as the preset word library. The preset word library may also be used in step S110 to obtain a candidate word segmentation set.

[0074] As a possible implementation method, the preset word library can be obtained in the following manner: obtain multiple preset word segments from a preset data source; count the number of occurrences of each preset word segment in the preset data source, and use the number of occurrences corresponding to each preset word segment as the word frequency of the preset word segment. Among them, the multiple preset word segments can be obtained from a preset data source with marked word segments, or can be obtained by matching in the preset data source based on other word libraries. In this way, the preset word library can be quickly obtained without determining the word frequency based on specific complex rules, and it is convenient to expand the word library.

[0075] For each candidate word segmentation result, the word frequency of the preset word segmentation that is the same as each candidate word segmentation in the candidate word segmentation result can be searched from the preset word library, and the obtained word frequencies are accumulated to obtain the total word frequency corresponding to the candidate word segmentation result. When the total word frequency corresponding to each candidate word segmentation result is obtained, the maximum total word frequency can be determined by comparison, and the candidate word segmentation result corresponding to the maximum total word frequency is used as the target word segmentation result. In this way, it is convenient to obtain the target word segmentation result with the same conventional word segmentation habit, and the calculation amount of calculating the total word frequency is small, which is convenient to ensure efficiency.

[0076] The following is an example to illustrate the above word segmentation method.

[0077] The iterative formula used to implement dynamic rules is:

[0078]

[0079] Let i be the text subscript, and text[i:] represents the text substring with the character at position i as the first character in the text to be segmented. Suppose the text to be segmented is "Guo Degang and Yu Qian's Crosstalk Selections".

[0080] STEP 1: Find all the candidates that can form a word in the text, and group them with i as the first character.

[0081] Through STEP1, we can get the following candidate word set:

[0082] i=0: Guo Degang, Guo Degang Yu Qian, Guo

[0083] When i = 1: Degang, De

[0084] When i = 2: Gang

[0085] When i = 3: Yu Qian, Yu Qian's cross-talk, Yu

[0086] When i = 4: Qian

[0087] When i = 5: Cross-talk, Selected Cross-talk, Xiang

[0088] When i = 6: Sheng

[0089] When i = 7: Selected, Jing

[0090] When i = 8: Xuan

[0091] STEP2: Traverse text from the end to the front, that is, loop from the largest to i, and obtain dp[i] from the previous iteration formula. After the loop ends, a detailed route of the optimal path scheme is obtained.

[0092] STEP3: From this route, find the global optimal segmentation scheme to obtain the final word segmentation result.

[0093] Set the initial value of dp[end]. The following is an explanation with the initial value of dp[end] being 1. Start looping from i = 8. Then, the optimal local word segmentation results and the maximum local products corresponding to each position will be obtained:

[0094] dp[8] = 1 * length(Xuan) = 1

[0095] dp[7] = max(1 * length(Selected), dp[8] * length(Jing)) = 2

[0096] dp[6] = max(dp[7] * length(Sheng)) = 2

[0097] dp[5] = max(1 * length(Selected Cross-talk), dp[7] * length(Cross-talk), dp[6] * length(Xiang)) = 4

[0098] dp[4] = max(dp[5] * length(Qian)) = 4

[0099] dp[3] = max(dp[7] * length(Yu Qian's cross-talk), dp[5] * length(Yu Qian), dp[4] * length(Yu)) = 8

[0100] dp[2] = max(dp[3] * length(Gang)) = 8

[0101] dp[1] = max(dp[3] * length("德纲"), dp[2] * length("德")) = 16

[0102] dp[0] = max(dp[5] * length("郭德纲于谦"), dp[3] * length("郭德纲"), dp[1] * length("郭")) = 24

[0103] For example, the optimal partial word segmentation results corresponding to position 5 are: "相声精选" (Selected Cross Talks); "精选相声" (Cross Talks Selection), and the corresponding maximum partial product is 4.

[0104] After obtaining the route, it is possible to determine the segmentation combinations with the maximum dp value as follows:

[0105] 郭德纲于谦相声精选 —— dp = 24

[0106] 郭德纲于谦相声精选 —— dp = 24

[0107] 郭德纲于谦相声精选 —— dp = 24

[0108] At this time, a large number of candidate solutions have been eliminated, and three solutions with equal optimized quantization values have been obtained. In this case, it is possible to compare which segmentation combination is finer, so as to select "郭德纲于谦相声精选" (Selected Cross Talks by Guo Degang and Yu Qian) as the optimal solution, that is, as the target word segmentation result. It is also possible to comprehensively consider the word frequencies of each token in each segmentation combination, and obtain a set of solutions with the largest word frequency. In this way, if the word frequency of "相声精选" (Selected Cross Talks) is greater than the sum of the word frequencies of "相声" (Cross Talks) and "精选" (Selection), then "郭德纲于谦相声精选" (Selected Cross Talks by Guo Degang and Yu Qian) can be selected as the optimal solution, that is, as the target word segmentation result.

[0109] In this way, dynamic programming and a custom iterative formula can be used to achieve global optimal word segmentation, which has the advantages of good effect, high efficiency, and low cost. It can be applied in vertical search scenarios, including index word segmentation, query understanding word segmentation, etc.

[0110] To execute the corresponding steps in the above embodiments and various possible ways, the following provides an implementation manner of a word segmentation device 200. Optionally, the word segmentation device 200 may adopt the Figure 1 device structure of the electronic device 100 shown above. Further, please refer to Figure 6 , Figure 6 which is one of the block diagrams of the word segmentation device 200 provided by the embodiments of the present application. It should be noted that for the word segmentation device 200 provided in this embodiment, its basic principle and the technical effects produced are the same as those in the above embodiments. For the sake of brief description, for the parts not mentioned in this embodiment, reference may be made to the corresponding content in the above embodiments. The word segmentation device 200 may include: a word segmentation set obtaining module 210, a processing module 220, and a result determination module 230.

[0111] The word segmentation set acquisition module 210 is used to obtain a candidate word segmentation set corresponding to each character in the text to be segmented from the text to be segmented. The candidate word segmentation set corresponding to any character includes a candidate word segmentation with the character as the starting character, and when the candidate word segmentation includes multiple characters, the multiple characters are consecutive characters in the text to be segmented.

[0112] The processing module 220 is used to traverse the characters in the text to be segmented from back to front in a dynamic programming manner according to the candidate segmentation set corresponding to each character, with the goal of maximizing the product of the string lengths of each candidate segmentation included in the candidate segmentation result, to obtain the candidate segmentation result. The sum of the candidate segmentations in any one of the candidate segmentation results constitutes the text to be segmented.

[0113] The result determination module 230 is used to determine a target word segmentation result according to the obtained candidate word segmentation results.

[0114] Optionally, in this embodiment, the word segmentation set obtaining module 210 may obtain the candidate word segmentation set based on a preset word library, and / or the result determination module 230 may determine the target word segmentation result based on a preset word library.

[0115] Please refer to Figure 7 , Figure 7 The second block diagram of the word segmentation device 200 provided in the embodiment of the present application. In this embodiment, the word segmentation device 200 may further include a word library generation module 201.

[0116] The word library generation module 201 is used to: obtain a plurality of the preset participles from a preset data source; count the number of occurrences of each of the preset participles in the preset data source, and use the number of occurrences corresponding to each of the preset participles as the word frequency of the preset participle.

[0117] Optionally, the above modules can be stored in the form of software or firmware. Figure 1 The memory 110 shown in the figure or the operating system (OS) of the electronic device 100 is fixed, and can be used by Figure 1 Meanwhile, the data and program codes required for executing the above modules may be stored in the memory 110.

[0118] An embodiment of the present application also provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the word segmentation method is implemented.

[0119] In summary, the embodiments of the present application provide a word segmentation method, device, electronic device and readable storage medium. First, based on the text to be segmented, a candidate word segmentation set corresponding to each character in the text to be segmented can be obtained from the text to be segmented, wherein the candidate word segmentation set corresponding to any character includes a candidate word segmentation with the character as the starting character, and when the candidate word segmentation includes multiple characters, the multiple characters are continuous characters in the text to be segmented; then, according to the obtained candidate word segmentation set corresponding to each character, through a dynamic programming method, with the goal of maximizing the product of the string lengths of each candidate word segmentation included in the candidate word segmentation result, each character in the text to be segmented is traversed in order from back to front, so as to obtain a candidate word segmentation result, wherein the sum of the candidate word segmentations in any candidate word segmentation result constitutes the text to be segmented; finally, the target word segmentation result can be obtained according to the obtained candidate word segmentation result. Therefore, it is possible to obtain a better segmentation result without using a training set to train the segmentation model. When applied in a vertical search scenario, the segmentation effect can still be guaranteed. This method has the characteristics of good effect and low cost.

[0120] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0121] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0122] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0123] The above description is only an optional embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A word segmentation method, characterized in that: The method comprises: From the text to be segmented, a candidate segmentation set corresponding to each character in the text to be segmented is obtained, wherein the candidate segmentation set corresponding to any character includes a candidate segmentation with the character as the starting character, and when the candidate segmentation includes multiple characters, the multiple characters are consecutive characters in the text to be segmented; According to the candidate segmentation set corresponding to each character, through dynamic programming, with the goal of maximizing the product of the string lengths of each candidate segmentation included in the candidate segmentation result, traverse each character in the to-be-segmented text in a back-to-front order to obtain the candidate segmentation result, wherein the sum of the candidate segmentations in any one of the candidate segmentation results constitutes the to-be-segmented text; Determine a target word segmentation result according to the obtained candidate word segmentation results; Among them, the candidate word segmentation set corresponding to each character is dynamically programmed to maximize the product of the string lengths of each candidate word segmentation included in the candidate word segmentation result, and each character in the to-be-segmented text is traversed in order from back to front to obtain the candidate word segmentation result, including: Traversing the candidate participles in the candidate participle set corresponding to the traversed current character, obtaining the current candidate partial participle result and the current candidate partial product corresponding to the current candidate participle according to the string length of the current candidate participle corresponding to the traversed current character, and the optimal partial participle result and the maximum partial product corresponding to the target position, wherein the target position is the first position after the position of the last character of the current candidate participle in the text to be segmented, and the maximum partial product corresponding to the target position is the product of the string lengths of the candidate participles included in the optimal partial participle result corresponding to the maximum partial product; When the current candidate local products corresponding to each candidate participle in the candidate participle set corresponding to the current character are obtained, the largest current candidate local product among the current candidate local products corresponding to each candidate participle in the candidate participle set corresponding to the current character is used as the current maximum local product corresponding to the position of the current character, and the current candidate local participle result corresponding to the current maximum local product is used as the current optimal local participle result corresponding to the position of the current character, wherein the optimal local participle result corresponding to the position of the first character in the text to be segmented is the candidate participle result.

2. The method according to claim 1, characterized in that In the case where there are multiple candidate word segmentation results, determining a target word segmentation result according to the obtained candidate word segmentation results includes: Obtaining the word segmentation granularity of each of the candidate word segmentation results, wherein the word segmentation granularity is used to indicate the coarseness or fineness of the word segmentation splitting; The candidate word segmentation result corresponding to the minimum word segmentation granularity is selected as the target word segmentation result.

3. The method according to claim 2, characterized in that The word segmentation granularity is determined according to the number of candidate word segmentations included in the candidate word segmentation result, and the word segmentation granularity is inversely correlated with the number of candidate word segmentations included in the candidate word segmentation result.

4. The method according to claim 1, characterized in that: In the case where there are multiple candidate word segmentation results, determining a target word segmentation result according to the obtained candidate word segmentation results includes: According to the preset segmentation words and word frequencies included in the preset word library, the sum of the word frequencies of the segmentation words included in each of the candidate segmentation results is obtained as the total number of word frequencies corresponding to each of the candidate segmentation results; The candidate word segmentation result corresponding to the maximum total word frequency is selected as the target word segmentation result.

5. The method according to claim 4, characterized in that The preset word library is obtained in the following way: Obtaining a plurality of the preset word segments from a preset data source; The number of occurrences of each of the preset participles in the preset data source is counted, and the number of occurrences corresponding to each of the preset participles is used as the word frequency of the preset participle.

6. The method according to claim 1, characterized in that The preset word library includes a plurality of preset word segments, and the step of obtaining a candidate word segmentation set corresponding to each character in the text to be segmented from the text to be segmented includes: According to the preset word library, for each character in the text to be segmented, at least one character string starting with the character and being the same as one of the preset segmented words is found from the text to be segmented as the candidate segmented word corresponding to the character.

7. A word segmentation device, characterized in that: The device comprises: A word segmentation set obtaining module is used to obtain, from a text to be segmented, a candidate word segmentation set corresponding to each character in the text to be segmented, wherein the candidate word segmentation set corresponding to any character includes a candidate word segmentation with the character as a starting character, and when the candidate word segmentation includes multiple characters, the multiple characters are consecutive characters in the text to be segmented; A processing module is used to traverse each character in the to-be-segmented text in a backward order according to a candidate segmentation set corresponding to each character by a dynamic programming method, with the goal of maximizing the product of the string lengths of each candidate segmentation included in the candidate segmentation result, to obtain the candidate segmentation result, wherein the sum of the candidate segmentations in any one of the candidate segmentation results constitutes the to-be-segmented text; A result determination module, used to determine a target word segmentation result based on the obtained candidate word segmentation results; The processing module is specifically used to: traverse the candidate word segments in the candidate word segmentation set corresponding to the traversed current character, and obtain the current candidate partial word segmentation result and the current candidate partial product corresponding to the current candidate word segment according to the string length of the current candidate word segment corresponding to the traversed current character, and the optimal partial word segmentation result and the maximum local product corresponding to the target position, wherein the target position is the first position after the position of the last character of the current candidate word segment in the text to be segmented, and the maximum local product corresponding to the target position is the candidate word segment included in the optimal partial word segmentation result corresponding to the maximum local product. The product of the character string length of the selected word segmentation; when the current candidate local products corresponding to each candidate participle in the candidate participle set corresponding to the current character are obtained, the largest current candidate local product among the current candidate local products corresponding to each candidate participle in the candidate participle set corresponding to the current character is used as the current maximum local product corresponding to the position of the current character, and the current candidate local participle result corresponding to the current maximum local product is used as the current optimal local participle result corresponding to the position of the current character, wherein the optimal local participle result corresponding to the position of the first character in the text to be segmented is the candidate participle result.

8. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the word segmentation method described in any one of claims 1 to 6.

9. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the word segmentation method according to any one of claims 1 to 6 is implemented.

Images