Word segmentation labeling method and device, electronic equipment and storage medium

Abstract

The invention discloses a word segmentation labeling method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a text to be subjected to word segmentation; performing a word segmentation strategy based on a preset word list on the to-be-segmented text to obtain a first word segmentation path; performing a rule-based word segmentation strategy on the text to be subjected to word segmentation to obtain a second word segmentation path; performing a statistics-based word segmentation strategy on the text to be subjected to word segmentation to obtain a third word segmentation path; generating a plurality of candidate paths based on a preset dictionary and the text to be subjected to word segmentation; determining a plurality of word segmentation paths to be scored in the first word segmentation path, the second word segmentation path and the third word segmentation path based on the plurality of candidate paths; determining multi-dimensional scores of the plurality of word segmentation paths to be scored; according to the method, the target word segmentation path is determined in the multiple word segmentation paths to be scored based on the multi-dimensional score, and the optimal word segmentation annotation result is selected by performing score comparison on the screened word segmentation annotation results, so that the method is closer to actual requirements and semantics, and can adapt to more diversified text contents and application scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a word segmentation and labeling method and device, an electronic device and a storage medium. BACKGROUND

[0002] In the field of natural language processing, word segmentation is a technology of cutting a continuous text sequence into independent and meaningful language units, and is particularly important for languages without space separation such as Chinese. It is an important preprocessing link for subsequent semantic analysis, machine translation and other tasks. As a crucial basic task, the accuracy of word segmentation directly affects the performance of subsequent tasks such as part-of-speech tagging, named entity recognition, machine translation and the like.

[0003] At present, the mainstream word segmentation technology has a high dependence on a corpus. If the coverage of the corpus is not extensive enough, or the text is quite different from the actual application scenario, the word segmentation accuracy will decrease. At the same time, training a statistical model requires a large amount of computing resources and time. SUMMARY

[0004] In order to solve the existing technical problems, the present application provides a word segmentation and labeling method, device, electronic device and storage medium. The word segmentation and labeling is performed based on a preset word table strategy, a rule-based word segmentation strategy and a statistical word segmentation strategy respectively. Then, a plurality of candidate paths generated based on a preset dictionary are used to preliminarily screen the word segmentation paths. Finally, the screened word segmentation and labeling results are scored and compared, and the optimal word segmentation strategy and word segmentation and labeling result are selected, which is closer to the actual demand and semantics, and can adapt to more diverse text content and application scenarios.

[0005] In a first aspect, an embodiment of the present application provides a word segmentation and labeling method, which comprises: obtaining a text to be segmented; performing a word segmentation strategy based on a preset word table on the text to be segmented to obtain a first word segmentation path; performing a rule-based word segmentation strategy on the text to be segmented to obtain a second word segmentation path; performing a statistical word segmentation strategy on the text to be segmented to obtain a third word segmentation path; generating a plurality of candidate paths based on a preset dictionary and the text to be segmented; determining a plurality of to-be-scored word segmentation paths in the first word segmentation path, the second word segmentation path and the third word segmentation path based on the plurality of candidate paths; determining a multi-dimensional score of the plurality of to-be-scored word segmentation paths; determining a target word segmentation path in the plurality of to-be-scored word segmentation paths based on the multi-dimensional score.

[0006] In an alternative embodiment, the rule-based segmentation strategy on the text to be segmented to obtain a second segmentation path comprises: performing forward maximum matching rule segmentation on the text to be segmented to obtain a forward segmentation path; performing reverse maximum matching rule segmentation on the text to be segmented to obtain a reverse segmentation path; determining the second segmentation path based on the forward segmentation path and the reverse segmentation path.

[0007] In an alternative embodiment, the statistical-based segmentation strategy on the text to be segmented to obtain a third segmentation path comprises: determining a plurality of third sub-paths based on the text to be segmented; each third sub-path comprises a plurality of segmented words arranged in a preset order; obtaining large-scale corpus data; determining inter-segmented word following probability of each third sub-path based on the large-scale corpus data; the inter-segmented word following probability is used to represent the commonality of adjacent segmented words; determining the third segmentation path based on the inter-segmented word following probability in the plurality of third sub-paths.

[0008] In an alternative embodiment, the determination of a plurality of to-be-scored segmentation paths from the first segmentation path, the second segmentation path and the third segmentation path based on the plurality of candidate paths comprises: performing the following for each segmentation path in the first segmentation path, the second segmentation path and the third segmentation path: determining the segmentation path being performed as a current segmentation path; if there is a candidate path in the plurality of candidate paths that is identical to the current segmentation path, determining the current segmentation path as the to-be-scored segmentation path; or if there is no candidate path in the plurality of candidate paths that is identical to the current segmentation path, but there is a sub-word path of the current segmentation path, determining the sub-word path of the current segmentation path as the to-be-scored segmentation path; the sub-word path is a path after splitting the current segmentation path.

[0009] In an alternative embodiment, the determination of multi-dimensional scores of the plurality of to-be-scored segmentation paths comprises: determining a dictionary hit score of the plurality of to-be-scored segmentation paths based on the preset dictionary; determining a structure reward score of the plurality of to-be-scored segmentation paths based on the length of the segmented words of the plurality of to-be-scored segmentation paths; determine a grammatical fluency score of the plurality of to-be-scored segmented word paths based on part-of-speech of the plurality of to-be-scored segmented word paths; determine a statistical probability score of the plurality of to-be-scored segmented word paths based on a preset language model; determine a first weight of the dictionary hit score, a second weight of the structure reward score, a third weight of the grammatical fluency score, and a fourth weight of the statistical probability score; determine the multi-dimensional score of each of the to-be-scored segmented word paths based on the dictionary hit score, the first weight, the structure reward score, the second weight, the grammatical fluency score, the third weight, the statistical probability score, and the fourth weight of each of the to-be-scored segmented word paths.

[0010] In an optional embodiment, the determining the first weight of the dictionary hit score, the second weight of the structure reward score, the third weight of the grammatical fluency score, and the fourth weight of the statistical probability score comprises: obtaining the first weight, the second weight, the third weight, and the fourth weight by using a pre-trained multi-layer perception and a supervised learning test set.

[0011] In an optional embodiment, the obtaining the first weight, the second weight, the third weight, and the fourth weight by using the pre-trained multi-layer perception and the supervised learning test set comprises: obtaining the first initial weight, the second weight, the initial third weight, and the fourth weight by using the pre-trained multi-layer perception and the supervised learning test set; determine a text attribute of the to-be-segmented text; if the text attribute is technical text, increasing the first initial weight to obtain the first weight and decreasing the third initial weight to obtain the third weight, or if the text attribute is chat text, increasing the third initial weight to obtain the third weight and decreasing the first initial weight to obtain the first weight.

[0012] In a second aspect, the embodiments of the present application provide a segmented word labeling device, the device comprising: an acquisition module configured to acquire to-be-segmented text; a first segmentation module configured to perform a preset word table-based segmentation strategy on the to-be-segmented text to obtain a first segmented word path; a second segmentation module configured to perform a rule-based segmentation strategy on the to-be-segmented text to obtain a second segmented word path; a third segmentation module configured to perform a statistics-based segmentation strategy on the to-be-segmented text to obtain a third segmented word path; The path generation module is configured to generate a plurality of candidate paths based on a preset dictionary and the text to be segmented. The first determination module is configured to determine a plurality of to-be-scored segmented paths from the first segmented path, the second segmented path and the third segmented path based on the plurality of candidate paths. The second determination module is configured to determine a multi-dimensional score of the plurality of to-be-scored segmented paths. The third determination module is configured to determine a target segmented path from the plurality of to-be-scored segmented paths based on the multi-dimensional score.

[0013] In a third aspect, an electronic device is provided, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the segmentation and annotation method of the first aspect.

[0014] In a fourth aspect, a computer-readable storage medium is provided, which stores at least one instruction or at least one program. The at least one instruction or the at least one program is loaded and executed by a processor to implement the segmentation and annotation method of the first aspect.

[0015] In a fifth aspect, a computer program product or a computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium. The processor executes the computer instructions to enable the computer device to implement the segmentation and annotation method of the first aspect.

[0016] The segmentation and annotation method, the apparatus, the electronic device and the storage medium provided by the embodiments of the present application have the following technical effects: obtaining a text to be segmented; performing a segmentation strategy based on a preset word table on the text to be segmented to obtain a first segmentation path; performing a segmentation strategy based on a rule on the text to be segmented to obtain a second segmentation path; performing a segmentation strategy based on statistics on the text to be segmented to obtain a third segmentation path; generating a plurality of candidate paths based on a preset dictionary and the text to be segmented; determining a plurality of to-be-scored segmentation paths in the first segmentation path, the second segmentation path and the third segmentation path based on the plurality of candidate paths; determining a multi-dimensional score of the plurality of to-be-scored segmentation paths; and determining a target segmentation path in the plurality of to-be-scored segmentation paths based on the multi-dimensional score. In the embodiment of the present application, the segmentation annotation is performed based on the segmentation strategy based on the preset word table, the segmentation strategy based on the rule and the segmentation strategy based on the statistics, and then the preliminary screening of the segmentation path is performed based on the plurality of candidate paths generated based on the preset dictionary, and finally the segmented annotation results screened out are scored and compared, the optimal segmentation strategy and segmentation annotation result are selected, which is closer to the actual demand and semantics, and can adapt to more diversified text content and application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application; Figure 2 is a flowchart of a segmentation annotation method provided by an embodiment of the present application Figure 1 ; Figure 3 is a flowchart of a segmentation annotation method provided by an embodiment of the present application Figure 2 ; Figure 4 is a flowchart of a method for determining a plurality of to-be-scored segmentation paths provided by an embodiment of the present application; Figure 5 is a flowchart of a method for determining a multi-dimensional score provided by an embodiment of the present application; Figure 6 is a structural schematic diagram of a segmentation annotation device provided by an embodiment of the present application; Figure 7 is a hardware structure block diagram of a server of a segmentation annotation method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0019] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of the present application.

[0020] It should be noted that the terms "first", "second" and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in other than the order illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units need not be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0021] Please refer to Figure 1 , Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application, which includes a word segmentation and annotation system 101.

[0022] In an embodiment of the present application, the word segmentation and annotation system 101 obtains a text to be segmented; performs a word segmentation strategy based on a preset word table on the text to be segmented to obtain a first word segmentation path; performs a word segmentation strategy based on a rule on the text to be segmented to obtain a second word segmentation path; performs a word segmentation strategy based on statistics on the text to be segmented to obtain a third word segmentation path; generates a plurality of candidate paths based on a preset dictionary and the text to be segmented; determines a plurality of to-be-scored word segmentation paths in the first word segmentation path, the second word segmentation path and the third word segmentation path based on the plurality of candidate paths; determines a multi-dimensional score of the plurality of to-be-scored word segmentation paths; and determines a target word segmentation path in the plurality of to-be-scored word segmentation paths based on the multi-dimensional score.

[0023] In an embodiment of the present application, the word segmentation and annotation system 101 obtains a text to be segmented; performs a word segmentation strategy based on a preset word table on the text to be segmented to obtain a first word segmentation path; performs a word segmentation strategy based on a rule on the text to be segmented to obtain a second word segmentation path; performs a word segmentation strategy based on statistics on the text to be segmented to obtain a third word segmentation path; generates a plurality of candidate paths based on a preset dictionary and the text to be segmented; determines a plurality of to-be-scored word segmentation paths in the first word segmentation path, the second word segmentation path and the third word segmentation path based on the plurality of candidate paths; determines a multi-dimensional score of the plurality of to-be-scored word segmentation paths; and determines a target word segmentation path in the plurality of to-be-scored word segmentation paths based on the multi-dimensional score.

[0024] The following describes a specific embodiment of a word segmentation and annotation method according to an embodiment of the present application, Figure 2Figure 1 is a flowchart of a word segmentation and annotation method according to an embodiment of the present application Figure 1 The present specification provides method operation steps as embodiments or flowcharts, but more or fewer operation steps can be included based on routine or non-creative labor. The order of steps listed in the embodiments is only one of many possible execution orders, and does not represent the only execution order. In actual system or server product execution, the method order shown in the embodiments can be executed in sequence or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, as shown in Figure 2 The method can include the following steps: S201: obtaining a text to be segmented.

[0025] S202: performing a word segmentation strategy based on a preset word table on the text to be segmented to obtain a first word segmentation path.

[0026] S203: performing a rule-based word segmentation strategy on the text to be segmented to obtain a second word segmentation path.

[0027] S204: performing a statistical-based word segmentation strategy on the text to be segmented to obtain a third word segmentation path.

[0028] S205: generating a plurality of candidate paths based on a preset dictionary and the text to be segmented.

[0029] S206: determining a plurality of to-be-scored word segmentation paths from the first word segmentation path, the second word segmentation path, and the third word segmentation path based on the plurality of candidate paths.

[0030] S207: determining a multi-dimensional score of the plurality of to-be-scored word segmentation paths.

[0031] S208: determining a target word segmentation path from the plurality of to-be-scored word segmentation paths based on the multi-dimensional score.

[0032] Figure 3 Figure 2 is a flowchart of a word segmentation and annotation method according to an embodiment of the present application Figure 2 The method can include the following steps: S301: obtaining a text to be segmented.

[0033] In one possible embodiment, the text to be segmented is a text that needs to be segmented and annotated, and is usually a coherent text.

[0034] S302: performing a word segmentation strategy based on a preset word table on the text to be segmented to obtain a first word segmentation path.

[0035] In the embodiment of the present application, the word segmentation strategy based on the preset word table is a method of starting from the starting position of the text, scanning and comparing the word table, and preferentially selecting the longest matching word to segment the text. The preset word table can be a user input or a pre-set specific word table. This word table can be applicable to a certain professional field, or a word table customized based on specific task requirements. The word segmentation annotation system receives user input or pre-set preset word table, and performs word segmentation strategy based on the preset word table on the text to be segmented to obtain a first word segmentation path.

[0036] In a possible embodiment, taking the text to be segmented "married and unmarried" as an example, the word segmentation process is as follows: First, scan the preset word table to find the longest word starting with "conclusion" is "marriage", and segment out "marriage".

[0037] The remaining text is "of and unmarried", starting from "of", the longest match in the preset word table is "of". Segment out "of".

[0038] The remaining text is "and unmarried", starting from "and", scan the preset word table, find that "and" and "and" are both matching items. According to the "longest match" principle, select "and". Segment out "and".

[0039] The remaining text is "still not married", starting from "still", the longest match in the preset word table is "still not", and segment out "still not".

[0040] The remaining text is "married", starting from "conclusion", the longest match in the preset word table is "marriage", and segment out "marriage".

[0041] The remaining text is "of", the longest match in the preset word table is "of", and segment out "of".

[0042] The final word segmentation result is the first word segmentation path: marriage / of / and still not / marriage / of.

[0043] S303: performing a rule-based word segmentation strategy on the text to be segmented to obtain a second word segmentation path.

[0044] In the embodiment of the present application, the rule-based word segmentation strategy is a method of segmenting the text according to the pre-set matching direction and matching principle.

[0045] In a possible embodiment, the rules include forward maximum matching rules and reverse maximum matching rules, and the rule-based word segmentation strategy on the text to be segmented to obtain a second word segmentation path includes: S3031: performing word segmentation on the text to be segmented according to the forward maximum matching rule to obtain a forward word segmentation path.

[0046] S3032: performing reverse maximum matching rule on the text to be segmented to obtain a reverse segmentation path.

[0047] S3033: determining the second segmentation path based on the forward segmentation path and the reverse segmentation path.

[0048] Take "ping pong paddle auction finished" as an example, the segmentation process of the reverse maximum matching rule is as follows: Starting from "finished" at the end of the sentence, match to "finished", and then match to "finished".

[0049] The remaining "ping pong paddle", starting from "sell" and going forward, match to "auction".

[0050] The remaining "ping pong ball", starting from "ball" and going forward, the longest match is "ping pong ball".

[0051] The final result of the reverse segmentation path is: ping pong ball / auction / finished.

[0052] S304: performing statistical-based segmentation strategy on the text to be segmented to obtain a third segmentation path.

[0053] In the embodiments of the present application, the statistical-based segmentation strategy is a method of calculating the most probable segmentation path by using statistical information such as word frequency and co-occurrence probability in a large-scale corpus.

[0054] In one possible embodiment, the statistical-based segmentation strategy on the text to be segmented to obtain a third segmentation path includes: S3041: determining a plurality of third sub-paths based on the text to be segmented.

[0055] In the embodiments of the present application, the third sub-path is all possible segmentation paths of the text to be segmented, and each third sub-path includes a plurality of segmented words arranged in a predetermined order.

[0056] S3042: obtaining large-scale corpus data.

[0057] S3043: determining the inter-segmentation following probability of each third sub-path based on the large-scale corpus data.

[0058] In the embodiments of the present application, the inter-segmentation following probability is used to represent the commonality of adjacent segmented words. Assuming that AB / C are adjacent segmented words, the inter-segmentation following probability is the commonality of C following AB in the large-scale corpus data.

[0059] S3044: determining the third segmentation path based on the inter-segmentation following probability in the plurality of third sub-paths.

[0060] Take the example of "ping pong auction complete" to illustrate the process of word segmentation based on statistical word segmentation strategy: First, find all possible word segmentation paths, mainly including two results, "ping pong / racket / sell / complete / done" and "ping pong / auction / complete / done".

[0061] Then, according to the probability data, calculate: "ping pong" is followed by "auction" or "racket" is followed by "sell". The statistical data is likely to show that the probability of "racket" as a whole noun and the probability of connecting the verb "sell" after it are much higher than "ping pong" and "paddle" are separated, and the probability of "ping pong" followed by "auction".

[0062] The final result selects the word segmentation path with higher statistical probability: ping pong / racket / sell / complete / done.

[0063] A probability threshold can also be set to retain one or more word segmentation paths with higher statistical probability.

[0064] S305: generating a plurality of candidate paths based on a preset dictionary and the text to be segmented.

[0065] In an embodiment of the present application, the preset dictionary includes a general dictionary and a professional dictionary.

[0066] In one possible embodiment, a directed acyclic graph (DAG) containing all possible word segmentation combinations is constructed based on the preset dictionary and the text to be segmented, and all head-to-tail candidate paths are parsed from it.

[0067] S306: determining a plurality of to-be-scored word segmentation paths in the first word segmentation path, the second word segmentation path and the third word segmentation path based on the plurality of candidate paths.

[0068] Figure 4 is a flowchart of a method for determining a plurality of to-be-scored word segmentation paths provided by an embodiment of the present application. In one possible embodiment, the determination of a plurality of to-be-scored word segmentation paths in the first word segmentation path, the second word segmentation path and the third word segmentation path based on the plurality of candidate paths comprises: For each of the first word segmentation path, the second word segmentation path and the third word segmentation path, perform: S3061: determining the word segmentation path being executed as the current word segmentation path.

[0069] S3062: judging whether there is a candidate path identical to the current word segmentation path in the plurality of candidate paths, if yes, performing S3063; if not, performing S3064.

[0070] S3063: Determine the current segmentation path as the to-be-scored segmentation path.

[0071] S3064: Determine whether there is a candidate path in the plurality of candidate paths that is the same as the sub-word path of the current segmentation path, if yes, perform S3065; if no, perform S3066.

[0072] S3065: Determine the sub-word path of the current segmentation path as the to-be-scored segmentation path.

[0073] In the embodiments of the present application, the sub-word path is the path after the current segmentation path is split.

[0074] S3066: Discard the current segmentation path.

[0075] Specifically, for each current segmentation path, it is checked whether it exists in the DAG.

[0076] In a possible embodiment, if there is the candidate path in the plurality of candidate paths that is the same as the current segmentation path, the current segmentation path is determined as the to-be-scored segmentation path. That is, the current segmentation path is completely matched with one of the candidate paths, the candidate path is mapped to the DAG, and it is checked whether each word corresponds to an edge in the DAG, that is, the word is in the dictionary and the position is correct. For example: the path “pingpangqiu / paimai / wan / cheng / le” completely matches the edges 0->3, 3->5, 5->6, 6->7 in the DAG, and thus is retained.

[0077] In a possible embodiment, if there is no candidate path in the plurality of candidate paths that is the same as the current segmentation path, but there is the candidate path that is the sub-word path of the current segmentation path, the sub-word path of the current segmentation path is determined as the to-be-scored segmentation path.

[0078] That is, in the path “pingpang / qiupai / mai / wan / le”, “qiupai” may not be in the dictionary, and thus does not directly match the DAG, and needs to be further processed. If the path contains an out-of-vocabulary word (such as “qiupai”), the out-of-vocabulary word is split using the sub-word strategy, and then it is checked whether the split path is in the DAG. For example: “qiupai” is split into “qi” and “pai”, and a new path “pingpang / qi / pai / mai / wan / le” is obtained. It is checked whether the new path is in the DAG, and the DAG may have edges 2->3 (“qi”) and 3->4 (“pai”), but “pai” is followed by “mai” and may not be continuous, and the position needs to be verified. If the DAG has an edge 4->5 (“mai”), the path may be partially matched.

[0079] Further, if the proportion of out-of-vocabulary words in the current segmentation path exceeds a threshold (such as 30%), the current segmentation path is directly discarded, because the score may be very low.

[0080] In another possible embodiment, if there is no candidate path in the plurality of candidate paths that is the same as the current segmentation path, and there is no candidate path in the plurality of candidate paths that is the same as a subword path of the current segmentation path, the candidate path is directly discarded.

[0081] In the segmentation annotation system, all candidate paths based on a preset dictionary are used to preliminarily screen segmentation paths generated by three segmentation methods, i.e., segmentation based on a given word table, rule-based segmentation, and statistical-based segmentation, so as to quickly filter out invalid or unreasonable paths, and only keep paths consistent with the dictionary basis, thereby reducing the calculation burden in the subsequent scoring stage, and improving the efficiency and accuracy of segmentation.

[0082] S308: determining a target segmentation path in the plurality of to-be-scored segmentation paths based on the multidimensional scores.

[0083] In an embodiment of the present application, the to-be-scored segmentation path with the highest multidimensional score is selected as the target segmentation path.

[0084] Figure 5 is a flowchart of a method for determining a multidimensional score provided by an embodiment of the present application, which can include: S401: determining a lexicon hit score of the plurality of to-be-scored segmentation paths based on the preset dictionary.

[0085] In an embodiment of the present application, the lexicon hit score (Lexicon Score, LS) reflects the use of high-quality dictionaries, and the purpose is to preferentially adopt professional domain or more core words.

[0086] Specifically, the rule for determining the lexicon hit score is that a word hitting a professional dictionary obtains a high score (for example, +10 points), a word hitting a general dictionary obtains a basic score (for example, +3 points), and a single character or an unregistered word obtains a very low score (for example, +1 point), and the total score of the path is the sum of the scores of each word.

[0087] S402: determining a structure bonus score of the plurality of to-be-scored segmentation paths based on the segmentation length of the plurality of to-be-scored segmentation paths.

[0088] In an embodiment of the present application, the structure bonus score (Structure Bonus, SB) tends to be more complete expression, and the purpose is to reward the path using "ping pong ball" instead of "ping pong / ball" when other conditions are similar, so as to reduce ambiguity.

[0089] Specifically, the rule for determining the structure reward score is that a length of each word in the path is scored, and the longer the word, the more points are scored. For example, the reward score = log(word length).

[0090] S403: Determine a syntax fluency score of the plurality of to-be-scored segmented paths based on part-of-speeches of the plurality of to-be-scored segmented paths.

[0091] In the embodiment of the present application, the syntax fluency score (SS1) is used to evaluate whether the path conforms to the syntax structure, and the purpose is to exclude combinations that do not conform to the syntax, such as a "noun" followed by a "auxiliary word".

[0092] Specifically, the rule for determining the syntax fluency score is that the fluency of the path is calculated based on the probability of part-of-speech collocation, such as the probability of a "noun" followed by a "verb". The more reasonable the collocation, the higher the score.

[0093] S404: Determine a statistics probability score of the plurality of to-be-scored segmented paths based on a preset language model.

[0094] In the embodiment of the present application, the statistics probability score (SS2) is used to judge the naturalness of the word combination, and the purpose is to select the path that best conforms to the language habits of the public. For example, the model knows that "ping-pong ball / auction" is more common than "ball / sell".

[0095] Specifically, the rule for determining the syntax fluency score is that the possibility of the word sequence appearing in a large amount of real corpus is calculated using an N-gram language model. The more common the combination, the higher the score.

[0096] S405: Determine a first weight of the dictionary hit score, a second weight of the structure reward score, a third weight of the syntax fluency score, and a fourth weight of the statistics probability score.

[0097] In one possible embodiment, the determination of the first weight of the dictionary hit score, the second weight of the structure reward score, the third weight of the syntax fluency score, and the fourth weight of the statistics probability score includes: obtaining the first weight, the second weight, the third weight, and the fourth weight by using a pre-trained multi-layer perception and a supervised learning test set.

[0098] In a possible embodiment, the dimension weights w can be dynamically adjusted according to the number of words of the shortest candidate path. The weight parameter set is learned by a multi-layer perceptron using a dedicated supervised learning test set. The multi-layer perceptron (MLP) is the most basic and widely used neural network structure in deep learning, and its core value is to fit the nonlinear data distribution by stacking hidden layers, which lays the foundation for modern complex models (such as CNN / Transformer).

[0099] In a possible embodiment, the first weight, the second weight, the third weight and the fourth weight are obtained by a pre-trained multi-layer perceptron using a supervised learning test set. S4051: The first initial weight, the second weight, the initial third weight and the fourth weight are obtained by the pre-trained multi-layer perceptron using the supervised learning test set.

[0100] S4052: The text attribute of the text to be segmented is determined.

[0101] In the embodiment of the present application, the text attribute of the text to be segmented includes technical text and chat text.

[0102] S4053: It is judged whether the text attribute is technical text. If yes, S4054 is performed; if not, S4055 is performed.

[0103] S4054: The first weight is obtained by increasing the first initial weight, and the third weight is obtained by decreasing the initial third weight.

[0104] S4055: The third weight is obtained by increasing the third initial weight, and the first weight is obtained by decreasing the first initial weight.

[0105] In a possible embodiment, if the text attribute is technical text, the first initial weight related to the dictionary needs to be increased, the first weight is obtained by increasing the first initial weight, and the third weight is obtained by decreasing the initial third weight.

[0106] In another possible embodiment, if the text attribute is chat text, the third initial weight related to daily language needs to be increased, the third weight is obtained by increasing the third initial weight, and the first weight is obtained by decreasing the first initial weight.

[0107] S406: determining the multi-dimensional score of each of the to-be-scored segmented paths based on the dictionary hit score of each of the to-be-scored segmented paths, the first weight, the structure reward score, the second weight, the grammar well-formedness score, the third weight, the statistical probability score, and the fourth weight.

[0108] In a possible embodiment, the scores of the dimensions are weighted and summed by preset weights to obtain a final total score. Total score = w1*S_LS + w2*S_SB + w3*S_SS1 + w4*S_SS2. The path with the highest total score is the final segmentation result.

[0109] The embodiment of the present application also provides a segmentation and labeling device, Figure 6 is a structural schematic diagram of a segmentation and labeling device provided by the embodiment of the present application, as Figure 6 shown, the device 500 comprises: an acquisition module 501 configured to acquire a text to be segmented; a first segmentation module 502 configured to perform a segmentation strategy based on a preset word table on the text to be segmented to obtain a first segmented path; a second segmentation module 503 configured to perform a segmentation strategy based on a rule on the text to be segmented to obtain a second segmented path; a third segmentation module 504 configured to perform a segmentation strategy based on statistics on the text to be segmented to obtain a third segmented path; a path generation module 505 configured to generate a plurality of candidate paths based on a preset dictionary and the text to be segmented; a first determination module 506 configured to determine a plurality of to-be-scored segmented paths from the first segmented path, the second segmented path, and the third segmented path based on the plurality of candidate paths; a second determination module 507 configured to determine multi-dimensional scores of the plurality of to-be-scored segmented paths; a third determination module 508 configured to determine a target segmented path from the plurality of to-be-scored segmented paths based on the multi-dimensional scores.

[0110] In an optional embodiment, the device further comprises: a fourth segmentation module configured to perform segmentation on the text to be segmented by a forward maximum matching rule to obtain a forward segmented path; a fifth segmentation module configured to perform segmentation on the text to be segmented by a reverse maximum matching rule to obtain a reverse segmented path; a fourth determination module configured to determine the second segmented path based on the forward segmented path and the reverse segmented path.

[0111] In an optional embodiment, the device further comprises: a fifth determining module, configured to determine a plurality of third sub-paths based on the text to be segmented; each of the third sub-paths comprises a plurality of segmented words arranged in a preset order; a first obtaining module, configured to obtain large-scale corpus data; a sixth determining module, configured to determine an inter-segmented word following probability of each of the third sub-paths based on the large-scale corpus data; the inter-segmented word following probability is used to represent a common degree of adjacent segmented words; a seventh determining module, configured to determine the third segmented path from the plurality of third sub-paths based on the inter-segmented word following probability.

[0112] In an optional embodiment, the method further comprises: performing the following for each of the first segmented path, the second segmented path and the third segmented path: an eighth determining module, configured to determine the segmented path being performed as a current segmented path; a ninth determining module, configured to determine the current segmented path as the segmented path to be scored if there is a same candidate path as the current segmented path in the plurality of candidate paths, or determine a sub-word path of the current segmented path as the segmented path to be scored if there is no same candidate path as the current segmented path in the plurality of candidate paths, but there is a sub-word path of the current segmented path; the sub-word path is a path after the current segmented path is split.

[0113] In an optional embodiment, the method further comprises: a tenth determining module, configured to determine a dictionary hit score of the plurality of segmented paths to be scored based on the preset dictionary; an eleventh determining module, configured to determine a structure reward score of the plurality of segmented paths to be scored based on lengths of segmented words of the plurality of segmented paths to be scored; a twelfth determining module, configured to determine a grammatical fluency score of the plurality of segmented paths to be scored based on parts of speech of segmented words of the plurality of segmented paths to be scored; a thirteenth determining module, configured to determine a statistical probability score of the plurality of segmented paths to be scored based on a preset language model; a fourteenth determining module, configured to determine a first weight of the dictionary hit score, a second weight of the structure reward score, a third weight of the grammatical fluency score and a fourth weight of the statistical probability score; The fifteenth determining module is configured to determine the multi-dimensional score of each of the to-be-scored segmented word paths based on the dictionary hit score, the first weight, the structure reward score, the second weight, the syntax coherence score, the third weight, the statistical probability score, and the fourth weight.

[0114] In an alternative embodiment, the method further comprises: The sixteenth determining module is configured to obtain the first weight, the second weight, the third weight, and the fourth weight by using a pre-trained multi-layer perception and a supervised learning test set.

[0115] In an alternative embodiment, the method further comprises: The seventeenth determining module is configured to obtain the first initial weight, the second weight, the initial third weight, and the fourth weight by using the pre-trained multi-layer perception and the supervised learning test set. The eighteenth determining module is configured to determine a text attribute of the to-be-segmented text. The weight adjusting module is configured to increase the first initial weight to obtain the first weight and decrease the third initial weight to obtain the third weight if the text attribute is a technical text, or increase the third initial weight to obtain the third weight and decrease the first initial weight to obtain the first weight if the text attribute is a chatting text.

[0116] The device and method embodiments in the present application are based on the same application concept.

[0117] The method embodiments provided in the present application can be executed in a computer terminal, a server, or a similar computing device. Taking the case of running on a server as an example, Figure 7 is a hardware structure diagram of a server of a segmentation and labeling method provided in the present application. As shown in Figure 7As shown, the server 600 can vary greatly in configuration and performance, and can include one or more Central Processing Units (CPU) 610 (which can include, but is not limited to, processing devices such as microprocessors, MCUs, or programmable logic devices, FPGAs, etc.), memory 630 for storing data, one or more storage media 620 (such as one or more mass storage devices) for storing applications 623 or data 622. The memory 630 and the storage media 620 can be of the temporary or persistent storage variety. The programs stored in the storage media 620 can include one or more modules, each of which can include a series of instructions for operating the server. Further, the CPU 610 can be configured to communicate with the storage media 620 to execute the series of instructions in the storage media 620 on the server 600. The server 600 can also include one or more power supplies 660, one or more wired or wireless network interfaces 650, one or more input / output interfaces 640, and / or one or more operating systems 621, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0118] The input / output interface 640 can be configured to receive or transmit data via a network. Examples of the network can include a wireless network provided by a communication provider of the server 600. In one example, the input / output interface 640 includes a network interface controller (NIC) that can be connected to other network devices through a base station to communicate with the Internet. In one example, the input / output interface 640 can be a radio frequency (RF) module that is configured to communicate with the Internet through a wireless manner.

[0119] Those of ordinary skill in the art can understand that, Figure 7 The structure shown is merely illustrative and does not limit the structure of the electronic device described above. For example, the server 600 can include more or fewer components than those shown, or have a different configuration than that shown. Figure 7 For example, the server 600 can include more or fewer components than those shown, or have a different configuration than that shown. Figure 7 For example, the server 600 can include more or fewer components than those shown, or have a different configuration than that shown.

[0120] The electronic device provided by the embodiments of the present application includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to implement the data processing method described above.

[0121] The embodiment of the present application further provides a computer readable storage medium, which can be arranged in a server to store at least one instruction, at least one program, a code set or an instruction set related to the word segmentation and labeling method in the method embodiment, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to realize the word segmentation and labeling method.

[0122] Optionally, in the embodiment, the storage medium can be located in at least one of the network servers in the computer network. Optionally, in the embodiment, the storage medium can include but is not limited to a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk and various storage program code media.

[0123] As can be seen from the above embodiment of the word segmentation and labeling method, the device, the electronic equipment or the storage medium provided by the present application, the text to be segmented is acquired in the present application; a first segmentation path is obtained by performing a word segmentation strategy based on a preset word table on the text to be segmented; a second segmentation path is obtained by performing a rule-based word segmentation strategy on the text to be segmented; a third segmentation path is obtained by performing a statistical-based word segmentation strategy on the text to be segmented; a plurality of candidate paths are generated based on a preset dictionary and the text to be segmented; a plurality of to-be-scored segmentation paths are determined in the first segmentation path, the second segmentation path and the third segmentation path based on the plurality of candidate paths; a multi-dimensional score of the plurality of to-be-scored segmentation paths is determined; and a target segmentation path is determined in the plurality of to-be-scored segmentation paths based on the multi-dimensional score. In the embodiment of the present application, the word segmentation and labeling is performed based on the word segmentation strategy based on the preset word table, the rule-based word segmentation strategy and the statistical-based word segmentation strategy, respectively, and then the preliminary screening of the segmentation paths is performed based on the plurality of candidate paths generated based on the preset dictionary, and finally the scored comparison of the screened word segmentation and labeling results is performed to select the optimal word segmentation strategy and word segmentation and labeling result, which is closer to the actual demand and semantics and can adapt to more diversified text content and application scenarios.

[0124] It should be noted that the above-mentioned embodiments of the present application are in the order of description only, and do not represent the advantages and disadvantages of the embodiments. The above describes a specific embodiment of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be executed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are possible or can be advantageous.

[0125] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each of the embodiments focuses on the difference from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0126] A person of ordinary skill in the art can understand that all or part of the steps of the above embodiments can be completed by hardware, or by a program instructing the relevant hardware to complete, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0127] The above only describes the preferred embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A word segmentation and annotation method, characterized in that, include: Get the text to be segmented; The text to be segmented is subjected to a segmentation strategy based on a preset vocabulary to obtain the first segmentation path; The text to be segmented is subjected to a rule-based segmentation strategy to obtain a second segmentation path; A statistical-based word segmentation strategy is applied to the text to be segmented to obtain a third word segmentation path; Multiple candidate paths are generated based on a preset dictionary and the text to be segmented; Based on the multiple candidate paths, multiple word segmentation paths to be scored are determined among the first word segmentation path, the second word segmentation path, and the third word segmentation path; Determine the multidimensional score of the multiple word segmentation paths to be scored; The target word segmentation path is determined based on the multidimensional scoring among the multiple word segmentation paths to be scored.

2. The word segmentation and annotation method according to claim 1, characterized in that, The step of applying a rule-based word segmentation strategy to the text to be segmented to obtain a second word segmentation path includes: The text to be segmented is segmented using the forward maximum matching rule to obtain the forward segmentation path; The text to be segmented is segmented using the reverse maximum matching rule to obtain the reverse segmentation path; The second word segmentation path is determined based on the forward word segmentation path and the reverse word segmentation path.

3. The word segmentation and annotation method according to claim 1, characterized in that, The step of applying a statistical-based word segmentation strategy to the text to be segmented to obtain a third word segmentation path includes: Multiple third sub-paths are determined based on the text to be segmented; each third sub-path includes multiple word segments arranged in a preset order; Obtain large-scale corpus data; The inter-segmentation following probability of each of the third sub-paths is determined based on the large-scale corpus data; the inter-segmentation following probability is used to characterize the frequency of adjacent segments. The third word segmentation path is determined from the plurality of third sub-paths based on the inter-segmentation following probability.

4. The word segmentation and annotation method according to claim 1, characterized in that, The step of determining multiple word segmentation paths to be scored from the first word segmentation path, the second word segmentation path, and the third word segmentation path based on the multiple candidate paths includes: For each of the first, second, and third word segmentation paths, execute: The currently executing word segmentation path is determined as the current word segmentation path; If there is a candidate path among the multiple candidate paths that is the same as the current word segmentation path, the current word segmentation path is determined as the word segmentation path to be scored; or if there is no candidate path among the multiple candidate paths that is the same as the current word segmentation path, but there is a sub-word path that is the same as the current word segmentation path, the sub-word path of the current word segmentation path is determined as the word segmentation path to be scored; the sub-word path is the path after the current word segmentation path is split.

5. The word segmentation and annotation method according to claim 1, characterized in that, The determination of the multidimensional scoring of the multiple segmented paths to be scored includes: Based on the preset dictionary, the dictionary hit score of the multiple word segmentation paths to be scored is determined; The structural reward score of the multiple segmented word paths to be scored is determined based on the segmentation length of the multiple segmented word paths to be scored. The grammatical fluency score of the multiple segmented word paths to be scored is determined based on the part-of-speech tags of the multiple segmented word paths to be scored. Based on a preset language model, the statistical probability scores of the multiple segmented word paths to be scored are determined; The first weight of the dictionary hit score, the second weight of the structure reward score, the third weight of the grammatical fluency score, and the fourth weight of the statistical probability score are determined. The multidimensional score for each segmented word path to be scored is determined based on the dictionary hit score, the first weight, the structural reward score, the second weight, the grammatical fluency score, the third weight, the statistical probability score, and the fourth weight.

6. The word segmentation and annotation method according to claim 5, characterized in that, The determination of the first weight of the dictionary hit score, the second weight of the structure reward score, the third weight of the grammatical fluency score, and the fourth weight of the statistical probability score includes: The first weight, the second weight, the third weight, and the fourth weight are obtained by using a pre-trained multilayer perceptron and a supervised learning test set.

7. The word segmentation and annotation method according to claim 6, characterized in that, The process of obtaining the first weight, the second weight, the third weight, and the fourth weight using a pre-trained multilayer perceptron and a supervised learning test set includes: The first initial weight, the second weight, the initial third weight, and the fourth weight are obtained through the pre-trained multilayer perceptron and using the supervised learning test set. Determine the text attributes of the text to be segmented; If the text attribute is technical text, increase the first initial weight to obtain the first weight, and decrease the third initial weight to obtain the third weight; or if the text attribute is chat text, increase the third initial weight to obtain the third weight, and decrease the first initial weight to obtain the first weight.

8. A word segmentation and annotation device, characterized in that, include: The acquisition module is used to acquire the text to be segmented. The first word segmentation module is used to perform word segmentation on the text to be segmented based on a preset vocabulary to obtain the first word segmentation path; The second word segmentation module is used to perform rule-based word segmentation on the text to be segmented to obtain a second word segmentation path; The third word segmentation module is used to perform a statistical word segmentation strategy on the text to be segmented to obtain the third word segmentation path; The path generation module is used to generate multiple candidate paths based on a preset dictionary and the text to be segmented; The first determining module is used to determine multiple word segmentation paths to be scored among the first word segmentation path, the second word segmentation path, and the third word segmentation path based on the multiple candidate paths; The second determining module is used to determine the multidimensional scores of the multiple word segmentation paths to be scored; The third determining module is used to determine the target word segmentation path among the multiple word segmentation paths to be scored based on the multidimensional score.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the word segmentation and annotation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the word segmentation and annotation method as described in any one of claims 1-7.

Images