Variable length word segmentation method, device and electronic device

Abstract

The present application provides a variable-length word segmentation method, device and electronic device. After obtaining the text to be processed, the text to be processed is segmented to obtain a corresponding directed acyclic graph, which contains multiple paths, each path contains multiple nodes, and each node corresponds to a segmentation. For each path, based on the set length parameter and the segmentations corresponding to each node contained in the path, the path score of the path is obtained. Based on the path scores of each path, the optimal path is determined, and the segmentation result is obtained based on the optimal path. This solution adopts the method of generating a directed acyclic graph containing multiple paths, and obtains the segmentation result under the guidance of the set length parameter, and can obtain an optimized segmentation result on the basis of meeting the required segmentation granularity.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a variable-length word segmentation method, device and electronic device. Background Art

[0002] Chinese natural language processing involves information processing at multiple levels, including word processing, phrase processing, sentence processing, etc., and all of these processes require text segmentation. Chinese word segmentation processing usually has different adaptability in terms of granularity, scene, style, etc. In the existing conventional word segmentation processing methods, the general goal is to make the word segmentation results as close to actual usage habits as possible.

[0003] However, in some special application scenarios, the granularity of word segmentation may be of greater concern, so that the word segmentation results with granularity that meets the requirements can be used for purposes such as comparison. In the existing conventional word segmentation processing method, the purpose of word segmentation is set to fit the actual language habits, which makes it difficult to make the granularity of the word segmentation results meet the requirements. Summary of the invention

[0004] The purpose of the present application includes, for example, providing a variable-length word segmentation method, device and electronic device, which can obtain optimized word segmentation results on the basis of meeting the required word segmentation granularity.

[0005] The embodiments of the present application can be implemented as follows:

[0006] In a first aspect, the present application provides a variable-length word segmentation method, the method comprising:

[0007] Obtaining a text to be processed, wherein the text to be processed includes a plurality of words;

[0008] Performing word segmentation processing on the text to be processed to obtain a corresponding directed acyclic graph, wherein the directed acyclic graph includes multiple paths, each of the paths includes multiple nodes, and each of the nodes corresponds to a word segmentation;

[0009] For each path, based on the set length parameter and the word segments corresponding to each node included in the path, the path score of the path is obtained;

[0010] Based on the path scores of the paths, an optimal path is determined, and a word segmentation result is obtained based on the optimal path.

[0011] In an optional implementation, the step of obtaining the path score of the path based on the set length parameter and the word segments corresponding to each node included in the path includes:

[0012] Determine the word length of the segmentation corresponding to each node included in the path;

[0013] Compare the word length of each node with the set length parameter to obtain the node score corresponding to each node;

[0014] The node scores corresponding to the nodes included in the path are accumulated to obtain the path score of the path.

[0015] In an optional implementation manner, the step of comparing the word length of each node with a set length parameter to obtain a node score corresponding to each node includes:

[0016] When the word length of a node is equal to the set length parameter, the node score corresponding to the node is set to 0;

[0017] When the word length of a node is greater than the set length parameter, the node score corresponding to the node is set to a preset positive number;

[0018] When the word length of a node is less than a set length parameter, the difference between the word length and the length parameter is used as the node score corresponding to the node.

[0019] In an optional implementation manner, the step of determining the optimal path based on the path scores of the paths includes:

[0020] The path scores of the paths are compared, and the path with the largest path score is taken as the optimal path.

[0021] In an optional implementation manner, the step of comparing the path scores of the paths and taking the path with the largest path score as the optimal path includes:

[0022] Compare the path scores of the paths, and if there is only one path with the largest path score, take the path as the optimal path;

[0023] If there are multiple paths with the largest path scores, for each of the multiple paths, the word frequencies of the segmented words corresponding to each node contained in the path in the preset dictionary are obtained, and the optimal path is determined from the multiple paths according to the word frequencies of the nodes contained in each path.

[0024] In an optional implementation, the step of determining the optimal path from the multiple paths according to the word frequencies of the nodes included in each path includes:

[0025] For each path, calculate the sum of the word frequencies of all nodes included in the path. If there is only one path with the largest sum of word frequencies, then this path is taken as the optimal path.

[0026] If there are multiple paths with the largest sum of word frequencies, the word frequency variance is calculated based on the word frequencies corresponding to the nodes included in the paths with the largest sum of word frequencies, and the path with the smallest word frequency variance is taken as the optimal path.

[0027] In an optional implementation, the step of calculating the word frequency variance based on the word frequencies corresponding to the nodes included in the path with the largest sum of word frequencies includes:

[0028] For each path with the largest sum of word frequencies, obtain the word frequencies of the word segments corresponding to each node included in the path;

[0029] The word frequency variance corresponding to the path is calculated based on the number of nodes included in the path, the average word frequency of the word segments corresponding to all the nodes, and the word frequency of the word segments corresponding to each of the nodes.

[0030] In an optional implementation, the step of performing word segmentation processing on the text to be processed includes:

[0031] obtaining a reference dictionary, wherein the reference dictionary includes a plurality of reference words;

[0032] The text to be processed is segmented based on a plurality of reference words in the reference dictionary.

[0033] In a second aspect, the present application provides a variable-length word segmentation device, the device comprising:

[0034] An acquisition module, used for acquiring a text to be processed, wherein the text to be processed includes a plurality of words;

[0035] A word segmentation module is used to perform word segmentation processing on the text to be processed to obtain a corresponding directed acyclic graph, wherein the directed acyclic graph includes multiple paths, each of the paths includes multiple nodes, and each of the nodes corresponds to a word segmentation;

[0036] An obtaining module, for obtaining, for each path, a path score of the path based on a set length parameter and the word segments corresponding to each node included in the path;

[0037] The determination module is used to determine the optimal path based on the path scores of each of the paths, and obtain a word segmentation result based on the optimal path.

[0038] In a third aspect, the present application provides an electronic device comprising one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine-executable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-executable instructions to perform the method steps described in any one of the aforementioned embodiments.

[0039] The beneficial effects of the embodiments of the present application include, for example:

[0040] The present application provides a variable-length word segmentation method, device and electronic device. After obtaining the text to be processed, the text to be processed is segmented to obtain a corresponding directed acyclic graph, which contains multiple paths, each path contains multiple nodes, and each node corresponds to a segmentation. For each path, based on the set length parameter and the segmentations corresponding to each node contained in the path, the path score of the path is obtained. Based on the path scores of each path, the optimal path is determined, and the segmentation result is obtained based on the optimal path. This solution adopts the method of generating a directed acyclic graph containing multiple paths, and obtains the segmentation result under the guidance of the set length parameter, and can obtain an optimized segmentation result on the basis of meeting the required segmentation granularity. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A flow chart of a variable length word segmentation method provided in an embodiment of the present application;

[0043] Figure 2 for Figure 1 Flow chart of sub-steps included in step S102;

[0044] Figure 3 for Figure 1 Flow chart of sub-steps included in step S103;

[0045] Figure 4 for Figure 1 Flow chart of sub-steps included in step S104;

[0046] Figure 5 for Figure 4 Flow chart of sub-steps included in step S1043;

[0047] Figure 6 for Figure 5 Flow chart of sub-steps included in step S10433;

[0048] Figure 7 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0049] Figure 8 A functional module block diagram of a variable-length word segmentation device provided in an embodiment of the present application.

[0050] Icons: 110 - storage medium; 120 - processor; 130 - variable length word segmentation device; 131 - acquisition module; 132 - word segmentation module; 133 - acquisition module; 134 - determination module; 140 - communication interface. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0052] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0053] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0054] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.

[0055] See also Figure 1 , is a flow chart of a variable length word segmentation method provided in an embodiment of the present application. The method steps defined in the process related to the variable length word segmentation method can be implemented by an electronic device with text analysis and processing functions. Figure 1 The specific process shown is described in detail.

[0056] S101, obtaining a text to be processed, wherein the text to be processed includes a plurality of characters.

[0057] S102, performing word segmentation processing on the text to be processed to obtain a corresponding directed acyclic graph, wherein the directed acyclic graph includes multiple paths, each of the paths includes multiple nodes, and each of the nodes corresponds to a word segmentation.

[0058] S103: For each path, based on a set length parameter and word segments corresponding to each node included in the path, obtain a path score of the path.

[0059] S104: Determine an optimal path based on the path scores of the paths, and obtain a word segmentation result based on the optimal path.

[0060] In this embodiment, the text to be processed may be a text containing multiple characters, such as an article, a paragraph, or a sentence, etc. In the text to be processed, a single character, or two or more adjacent characters may form different words, so after the text to be processed is segmented, a variety of different results obtained by different segmentation methods can be obtained.

[0061] In this embodiment, in order to avoid some meaningless words or symbols from interfering with word segmentation, the text to be processed may be preprocessed before word segmentation. The preprocessing method may include removing punctuation marks in the text to be processed, removing function words in the text to be processed, such as relational words, auxiliary words, etc. In addition, onomatopeia, interjections, etc. in the text to be processed may also be removed.

[0062] In this embodiment, please refer to Figure 2 , when performing word segmentation, it can be implemented in the following ways:

[0063] S1021, obtaining a reference dictionary, wherein the reference dictionary includes a plurality of reference words.

[0064] S1022: Perform word segmentation processing on the text to be processed based on a plurality of reference words in the reference dictionary.

[0065] In this embodiment, the reference dictionary may be a dictionary belonging to the same field as the content in the text to be processed. For example, if the content of the text to be processed belongs to the medical field, the reference dictionary may be a dictionary containing multiple reference words in the medical field. If the content of the text to be processed belongs to the communication field, the reference dictionary may be a dictionary containing multiple reference words in the communication field.

[0066] Based on the field to which the content of the text to be processed belongs, a reference dictionary of the corresponding field is obtained to perform word segmentation on the text to be processed, which can improve the adaptability between the obtained word segmentation and the actual field.

[0067] In this embodiment, when the word segmentation processing is performed on the text to be processed based on the reference dictionary, an AC automaton (Aho-Corasick automaton) can be used to perform the word segmentation processing process, and the result obtained can be presented in the form of a directed acyclic graph. The obtained directed acyclic graph contains multiple paths, each path contains multiple nodes, and each path from the starting point to the end point can constitute a complete text to be processed. Among them, each node corresponds to a word segmentation.

[0068] For example, if the text to be processed is "commodity and service provider", the structure of the resulting directed acyclic graph can be shown in the following table 1:

[0069] Table 1

[0070] 0 start 1 business commodity 2 Taste 3 and kimono 4 Clothes Service, service provider 5 Service 6 business 7 end

[0071] That is, the obtained word segmentation may include "commodity", "and", "service provider", "and service", "business", etc. That is, a variety of word segmentation processing results that completely contain all the characters in the text to be processed can be obtained, for example, "commodity", "and", "service provider", or "commodity", "and service", "business", "business".

[0072] In the resulting directed acyclic graph, all nodes in each path can together constitute a complete text to be processed, but the number of nodes contained in each path and the specific word segmentations corresponding to the nodes may be different. For example, the nodes contained in one path can correspond to the word segmentations "commodity", "and", "service provider", while the nodes contained in another path can correspond to the word segmentations "commodity", "and service", "business", "business". It can be seen that these two paths can share the node "commodity".

[0073] In practical applications, in some scenarios, there may be requirements for the granularity of word segmentation. For example, when comparing texts, it is necessary to compare at a certain word segmentation granularity, such as 3, 4, 5, etc. Therefore, in some scenarios, it is necessary for the word segmentation results to fit the actual situation and for the word segmentation granularity to meet the requirements. However, in the prior art, the focus of word segmentation processing is often placed on how to make the word segmentation results as consistent as possible with the actual usage habits. The word segmentation results obtained by this method are difficult to meet certain special requirements.

[0074] Based on this, in this embodiment, after obtaining multiple paths, when calculating the path score of the path, it can be calculated based on the set length parameter and the segmentation corresponding to each node contained in the path. Among them, the set length parameter can be 3, 4, 5, etc., that is, the length parameter reflects the segmentation length standard. In this way, the path score of each path obtained is calculated under the guidance of the set segmentation length standard, and the path score can reflect the degree of consistency between the segmentation of each path and the required segmentation granularity.

[0075] On this basis, the optimal path can be determined according to the path scores of each path, and the nodes included in the optimal path and the segmentation of each node can be used as the final segmentation result.

[0076] In this way, the word segmentation results obtained can be word segmentations that conform to general usage habits obtained by word segmentation according to a reference dictionary, and the word segmentations obtained under the guidance of the set length parameter can meet the required word segmentation granularity.

[0077] In this embodiment, since each path contains multiple nodes, the final path score of the path should be determined by the comprehensive combination of the multiple nodes. Figure 3 In the above step S103, a possible implementation method of obtaining the path score of the path is as follows:

[0078] S1031, determining the word length of the segmentation corresponding to each node included in the path.

[0079] S1032, comparing the word length of each of the nodes with the set length parameter to obtain a node score corresponding to each of the nodes.

[0080] S1033: Accumulate the node scores corresponding to the nodes included in the path to obtain the path score of the path.

[0081] In this embodiment, based on the word segmentation obtained based on the reference dictionary, in order to select the word segmentation result that meets the required granularity, the word segmentation of each path can be evaluated from the perspective of the word segmentation length.

[0082] For each node included in the path, the word length of the word corresponding to the node can be obtained. The word length can be determined by the number of characters included in the word. For example, if the number of characters included in the word is 2, the word length is 2.

[0083] The node score of the node is determined by comparing the word length of each node and the set length parameter. In this embodiment, when determining the node score of the node, when the word length of the node is equal to the set length parameter, the node score corresponding to the node can be set to 0. When the word length of the node is greater than the set length parameter, the node score corresponding to the node can be set to a preset positive number, for example, the preset positive number can be 1. When the word length of the node is less than the set length parameter, the difference between the word length and the length parameter can be used as the node score corresponding to the node. For example, when the length parameter is 3 and the word length is 1, the node score of the node can be word length-length parameter=-2. In this case, the node score obtained is a negative number, which can be used as a penalty item in the evaluation process.

[0084] It should be noted that in the above-mentioned various situations, the specific node score determination method of the node can also adopt other methods, and because the change of each word segmentation will affect the word segmentation of other words, it is generally impossible for all nodes included in the path to be consistent with the length parameter, because the word segmentation also needs to be consistent with the reference word in the reference dictionary. Therefore, the specific node score determination method of the node can be satisfied as long as the path as a whole is more consistent with the required granularity, the higher the path score.

[0085] On the basis of the above, after obtaining the path scores of each path, the path scores of each path can be compared, and the path with the largest path score is taken as the optimal path.

[0086] In implementation, the path with the highest path score may be one or more. Figure 4 When determining the optimal path by comparing the path scores, it can be achieved in the following ways:

[0087] S1041, comparing the path scores of the paths.

[0088] S1042: If there is only one path with the largest path score, this path is used as the optimal path.

[0089] S1043, if there are multiple paths with the largest path scores, then for each of the multiple paths, obtain the word frequency of the segmented words corresponding to each node included in the path in the preset dictionary, and determine the optimal path from the multiple paths according to the word frequency of the nodes included in each path.

[0090] In this embodiment, if there is only one path with the largest path score determined based on the path scores, then this path can be used as the optimal path.

[0091] If there are multiple paths with the largest path scores, the optimal path can be determined from the multiple paths. As can be seen from the above, since the segmented words in the path affect each other, when one of the adjacent segmented words changes, it will affect the linkage change of the adjacent segmented words. Therefore, for each path, the word length corresponding to each node in the path cannot be kept consistent, that is, the word length of each node is uneven.

[0092] Furthermore, among the segmentations included in each path, some segmentations are obtained directly based on the words included in a preset dictionary, wherein the preset dictionary may be the reference dictionary mentioned above, and some segmentations may be automatically constructed after the adjacent segmentations based on the original order of words in the text.

[0093] The preset dictionary may also be a text, in which a certain word may appear multiple times. That is to say, the number of times the word corresponding to a certain word appears in the preset dictionary is the word frequency corresponding to the word. The greater the word frequency, the more commonly used the word is.

[0094] Therefore, in this embodiment, if the optimal path cannot be determined by the path score, for multiple paths with the highest path scores, the word frequencies of the groups corresponding to the nodes included in each path can be calculated to determine the optimal path based on the word frequencies.

[0095] See also Figure 5In this embodiment, when determining the optimal path based on word frequency, the following method can be used:

[0096] S10431, for each path, calculate the sum of the word frequencies of all nodes included in the path, and if there is only one path with the largest sum of word frequencies, then take this path as the optimal path.

[0097] S10432: If there is only one path with the largest sum of word frequencies, then this path is taken as the optimal path.

[0098] S10433, if there are multiple paths with the largest sum of word frequencies, the word frequency variance is calculated based on the word frequencies corresponding to the nodes included in the paths with the largest sum of word frequencies, and the path with the smallest word frequency variance is taken as the optimal path.

[0099] In this embodiment, when the sum of the word frequencies of multiple paths is the same, the idea of ​​calculating the variance can be used, because the variance is a measure of the degree of discreteness of a random brightening or a set of data. The smaller the variance, the lower the degree of discreteness. Therefore, the word frequency variance can be calculated based on the word frequency, and the path with the smallest word frequency variance is taken as the optimal path.

[0100] See also Figure 6 In this embodiment, when calculating the word frequency variance of each path, it can be achieved in the following way:

[0101] S104331. For each of the multiple paths, obtain the word frequency of the segmented words corresponding to each node included in the path.

[0102] S104332, calculating the word frequency variance corresponding to the path according to the number of nodes included in the path, the average word frequency of the word segments corresponding to all the nodes, and the word frequency of the word segments corresponding to each of the nodes.

[0103] In this embodiment, for any path among the multiple paths with the largest sum of word frequencies, if the path contains N nodes, that is, corresponds to N word segments, the word frequency of the i-th word segment can be represented by x i Based on the frequency of each word segment, the average frequency can be calculated. The word frequency variance σ of this path can be calculated by the following formula:

[0104]

[0105] In this embodiment, the word frequency variance is used to assist in determining the optimal path, and the optimal path can be evaluated from the perspectives of word segmentation commonness and word frequency uniformity, so that in the optimal path finally obtained, the word segmentation uniformity, word segmentation rationality, and the matching degree with the required granularity can all achieve good results.

[0106] By using the variable-length word segmentation method in this embodiment, the difficulty of uncontrollable results of traditional word segmentation methods can be addressed. Combine word frequency and word segmentation granularity to solve some mis-segmentation situations. For example, when segmenting the text "A car accident occurred in Audi Nissan, Puxi", the traditional dictionary-based word segmentation method will impose rules on Audi, Nissan, Audini, and Puxi to perform word segmentation. Based on the word segmentation method provided in this embodiment, you only need to set the length parameter to 2 to distinguish, and then select a reasonable path through Audi's high word frequency. For example, for the text "Promoting the Classic Anthology of Cialdini", if the length parameter is set to 3, then in the word segmentation results, Cialdini can be retained. In some online tests, the length parameter can be switched according to the context, so as to achieve the purpose of adapting a set of word segmentation to multiple scenarios.

[0107] See also Figure 7 , is an exemplary component diagram of an electronic device provided in an embodiment of the present application, and the electronic device may include a storage medium 110, a processor 120, a variable-length word segmentation device 130, and a communication interface 140. In the present embodiment, the storage medium 110 and the processor 120 are both located in the electronic device and the two are separately arranged. However, it should be understood that the storage medium 110 may also be independent of the electronic device and may be accessed by the processor 120 through a bus interface. Alternatively, the storage medium 110 may also be integrated into the processor 120, for example, it may be a cache and / or a general register.

[0108] The variable-length word segmentation device 130 can be understood as the above-mentioned electronic device, or the processor 120 of the electronic device, or can be understood as a software function module independent of the above-mentioned electronic device or processor 120 that implements the above-mentioned variable-length word segmentation method under the control of the electronic device.

[0109] like Figure 8 As shown, the variable length word segmentation device 130 may include an acquisition module 131, a word segmentation module 132, an acquisition module 133 and a determination module 134. The functions of each functional module of the variable length word segmentation device 130 are described in detail below.

[0110] An acquisition module 131 is used to acquire a text to be processed, wherein the text to be processed includes a plurality of characters;

[0111] It can be understood that the acquisition module 131 can be used to execute the above step S101. The detailed implementation of the acquisition module 131 can refer to the above content related to step S101.

[0112] The word segmentation module 132 is used to perform word segmentation processing on the text to be processed to obtain a corresponding directed acyclic graph, wherein the directed acyclic graph includes multiple paths, each of the paths includes multiple nodes, and each of the nodes corresponds to a word segmentation.

[0113] It can be understood that the word segmentation module 132 can be used to execute the above step S102. The detailed implementation of the word segmentation module 132 can refer to the above content related to step S102.

[0114] The obtaining module 133 is used to obtain the path score of each path based on the set length parameter and the word segments corresponding to each node included in the path.

[0115] It can be understood that the obtaining module 133 can be used to execute the above step S103. The detailed implementation of the obtaining module 133 can refer to the above content related to step S103.

[0116] The determination module 134 is used to determine the optimal path based on the path scores of the paths, and obtain a word segmentation result based on the optimal path.

[0117] It can be understood that the determination module 134 can be used to execute the above step S104. The detailed implementation of the determination module 134 can refer to the above content related to step S104.

[0118] In a possible implementation, the obtaining module 133 may be used to:

[0119] Determine the word length of the segmentation corresponding to each node included in the path;

[0120] Compare the word length of each node with the set length parameter to obtain the node score corresponding to each node;

[0121] The node scores corresponding to the nodes included in the path are accumulated to obtain the path score of the path.

[0122] In a possible implementation, the obtaining module 133 may be used to:

[0123] When the word length of a node is equal to the set length parameter, the node score corresponding to the node is set to 0;

[0124] When the word length of a node is greater than the set length parameter, the node score corresponding to the node is set to a preset positive number;

[0125] When the word length of a node is less than a set length parameter, the difference between the word length and the length parameter is used as the node score corresponding to the node.

[0126] In a possible implementation, the determination module 134 may be used to:

[0127] The path scores of the paths are compared, and the path with the largest path score is taken as the optimal path.

[0128] In a possible implementation, the determination module 134 may be used to:

[0129] Compare the path scores of the paths, and if there is only one path with the largest path score, take the path as the optimal path;

[0130] If there are multiple paths with the largest path scores, for each of the multiple paths, the word frequencies of the segmented words corresponding to each node contained in the path in the preset dictionary are obtained, and the optimal path is determined from the multiple paths according to the word frequencies of the nodes contained in each path.

[0131] In a possible implementation, the determination module 134 may be used to:

[0132] For each path, calculate the sum of the word frequencies of all nodes included in the path. If there is only one path with the largest sum of word frequencies, then this path is taken as the optimal path.

[0133] If there are multiple paths with the largest sum of word frequencies, the word frequency variance is calculated based on the word frequencies corresponding to the nodes included in the paths with the largest sum of word frequencies, and the path with the smallest word frequency variance is taken as the optimal path.

[0134] In a possible implementation, the determination module 134 may be used to:

[0135] For each of the multiple paths, obtain the word frequency of the segmented words corresponding to each node included in the path;

[0136] The word frequency variance corresponding to the path is calculated based on the number of nodes included in the path, the average word frequency of the word segments corresponding to all the nodes, and the word frequency of the word segments corresponding to each of the nodes.

[0137] In a possible implementation, the word segmentation module 132 may be used to:

[0138] obtaining a reference dictionary, wherein the reference dictionary includes a plurality of reference words;

[0139] The text to be processed is segmented based on a plurality of reference words in the reference dictionary.

[0140] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0141] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are executed, the variable-length word segmentation method provided in the above embodiment is implemented.

[0142] Specifically, the computer-readable storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the computer-readable storage medium is executed, the above variable-length word segmentation method can be executed. Regarding the process involved when the computer-readable storage medium and its executable instructions are executed, reference can be made to the relevant description in the above method embodiment, which will not be described in detail here.

[0143] In summary, the variable-length word segmentation method, device and electronic device provided in the embodiments of the present application, after obtaining the text to be processed, performs word segmentation processing on the text to be processed to obtain a corresponding directed acyclic graph, which contains multiple paths, each path contains multiple nodes, and each node corresponds to a word segmentation. For each path, based on the set length parameter and the word segmentation corresponding to each node contained in the path, the path score of the path is obtained. Based on the path score of each path, the optimal path is determined, and the word segmentation result is obtained based on the optimal path. This scheme adopts the method of generating a directed acyclic graph containing multiple paths, and obtains the word segmentation result under the guidance of the set length parameter, and can obtain an optimized word segmentation result on the basis of satisfying the required word segmentation granularity.

[0144] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A variable length word segmentation method, characterized in that: The method comprises: Obtaining a text to be processed, wherein the text to be processed includes a plurality of words; Performing word segmentation processing on the text to be processed to obtain a corresponding directed acyclic graph, wherein the directed acyclic graph includes multiple paths, each of the paths includes multiple nodes, and each of the nodes corresponds to a word segmentation; For each path, based on the set length parameter and the segmentation corresponding to each node included in the path, the path score of the path is obtained, wherein the length parameter reflects the segmentation length standard, and the path score reflects the degree of conformity between the segmentation of each path and the required segmentation granularity; Determine an optimal path based on the path scores of each of the paths, and obtain a word segmentation result based on the optimal path; The step of obtaining the path score of the path based on the set length parameter and the word segments corresponding to each node included in the path includes: Determine the word length of the segmented words corresponding to each node included in the path; compare the word length of each node with the set length parameter to obtain the node score corresponding to each node; accumulate the node scores corresponding to each node included in the path to obtain the path score of the path; The step of comparing the word length of each node with the set length parameter to obtain the node score corresponding to each node includes: When the word length of the node is equal to the set length parameter, the node score corresponding to the node is set to 0; when the word length of the node is greater than the set length parameter, the node score corresponding to the node is set to a preset positive number; when the word length of the node is less than the set length parameter, the difference between the word length and the length parameter is used as the node score corresponding to the node.

2. The variable length word segmentation method according to claim 1, characterized in that: The step of determining the optimal path based on the path scores of the paths comprises: The path scores of the paths are compared, and the path with the largest path score is taken as the optimal path.

3. The variable length word segmentation method according to claim 2, characterized in that: The step of comparing the path scores of the paths and taking the path with the largest path score as the optimal path includes: Compare the path scores of the paths, and if there is only one path with the largest path score, take the path as the optimal path; If there are multiple paths with the largest path scores, for each of the multiple paths, the word frequencies of the segmented words corresponding to each node contained in the path in the preset dictionary are obtained, and the optimal path is determined from the multiple paths according to the word frequencies of the nodes contained in each path.

4. The variable length word segmentation method according to claim 3, characterized in that: The step of determining the optimal path from the multiple paths according to the word frequencies of the nodes included in each path includes: For each path, calculate the sum of the word frequencies of all nodes included in the path. If there is only one path with the largest sum of word frequencies, then this path is taken as the optimal path. If there are multiple paths with the largest sum of word frequencies, the word frequency variance is calculated based on the word frequencies corresponding to the nodes included in the paths with the largest sum of word frequencies, and the path with the smallest word frequency variance is taken as the optimal path.

5. The variable length word segmentation method according to claim 4, characterized in that: The step of calculating the word frequency variance based on the word frequencies corresponding to the nodes included in the path with the largest sum of word frequencies includes: For each path with the largest sum of word frequencies, obtain the word frequencies of the word segments corresponding to each node included in the path; The word frequency variance corresponding to the path is calculated based on the number of nodes included in the path, the average word frequency of the word segments corresponding to all the nodes, and the word frequency of the word segments corresponding to each of the nodes.

6. The variable length word segmentation method according to any one of claims 1 to 5, characterized in that: The step of performing word segmentation processing on the text to be processed includes: obtaining a reference dictionary, wherein the reference dictionary includes a plurality of reference words; The text to be processed is segmented based on a plurality of reference words in the reference dictionary.

7. A variable length word segmentation device, characterized in that: Used to implement the variable-length word segmentation method according to any one of claims 1 to 6, the device comprises: An acquisition module, used for acquiring a text to be processed, wherein the text to be processed includes a plurality of characters; A word segmentation module is used to perform word segmentation processing on the text to be processed to obtain a corresponding directed acyclic graph, wherein the directed acyclic graph includes multiple paths, each of the paths includes multiple nodes, and each of the nodes corresponds to a word segmentation; An obtaining module, for obtaining, for each path, a path score of the path based on a set length parameter and the word segments corresponding to each node included in the path; The determination module is used to determine the optimal path based on the path scores of each of the paths, and obtain a word segmentation result based on the optimal path.

8. An electronic device, characterized in that: The electronic device comprises one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine executable instructions executable by the processor, and when the electronic device is running, the processor executes the machine executable instructions to perform the method steps described in any one of claims 1 to 6.

Images