Word segmentation model training method, system, device, storage medium and word segmentation method

Abstract

The application relates to an artificial intelligence technology and discloses a word segmentation model training method and word segmentation method suitable for professional term word segmentation in the medical field, a word segmentation model training system, computer equipment and a storage medium. According to the scheme, a term set corresponding to a target field is acquired, professional terms in the term set are matched with a corpus, an anchor point word is obtained, a text segment containing the anchor point word in the corpus is taken as training data, and then the training data is used to train a word segmentation model, so that the word segmentation model is trained. Since the anchor point word itself is a term word strongly related to the field, other term words are easily generated around the anchor point word. Through the matching with the corpus to obtain the anchor point word, potential text content containing professional terms can be screened out to train the word segmentation model, and no labeled data is needed, the algorithm is simple, and the efficiency is high.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a word segmentation model training method, word segmentation method, word segmentation model training system, computer equipment and storage medium. Background Technology

[0002] Word segmentation is the foundation of natural language processing (e.g., machine translation, automatic summarization, automatic classification, full-text retrieval of literature databases, search engines, etc.), and the accuracy of word segmentation directly affects the results of natural language processing.

[0003] With the development of artificial intelligence and digital healthcare technologies, word segmentation algorithms are being applied more and more widely. For example, in intelligent consultation scenarios, the text information input by the user is segmented into words to further perform semantic analysis to understand the user's intent, thereby enabling intelligent consultation based on the user's input. If word segmentation is inaccurate, the text content input by the user cannot be correctly understood, and there may even be a significant misunderstanding of the user's intent, resulting in the inability to execute the intelligent consultation program normally.

[0004] The medical field contains a large number of specialized terms that differ significantly from everyday language. This is especially true in Traditional Chinese Medicine (TCM), which employs numerous specialized terms in its explanations of etiology and pathogenesis, disease classification, and treatment. These terms are often composed of multiple parts of speech, such as "lung dysfunction" and "liver qi invading the stomach," which are both TCM concepts and reflect the underlying pathogenesis of the disease.

[0005] General-purpose word segmentation models cannot understand the semantics of the underlying mechanisms in text. If a general-purpose word segmentation model is reused, a large amount of domain-annotated text is usually required, and techniques such as supervised neural networks are used for word segmentation, which is costly. Summary of the Invention

[0006] To address the problem of word segmentation for specialized terminology in a specific field, this application provides a word segmentation model training method, a word segmentation method, a word segmentation model training system, a computer device, and a storage medium.

[0007] According to one aspect of the embodiments of this application, a word segmentation model training method is disclosed, including:

[0008] Obtain a terminology set corresponding to the target domain, wherein the terminology set includes multiple professional terms of the target domain;

[0009] The multiple technical terms are matched with the corpus, and the technical terms matched in the corpus are used as anchor words; the corpus contains texts using at least some of the technical terms in the target field.

[0010] Obtain text fragments containing the anchor words from the corpus, and use the text fragments as training data;

[0011] The word segmentation model is trained using the training data until it meets the preset requirements.

[0012] In one exemplary embodiment, obtaining text fragments containing the anchor words in the corpus and using the text fragments as training data includes:

[0013] Obtain the sentence containing the anchor word from the corpus, and use the entire sentence containing the anchor word as the anchor sentence;

[0014] A first sliding window is used to extract several first segments from the anchor sentence, and an unsupervised training algorithm is used to obtain the weight of each word in each first segment;

[0015] Multiple different second sliding windows are used to extract the anchor sentence, thereby obtaining multiple second segments containing the anchor word;

[0016] Based on the weight of each character in the first segment, the average weight of all characters in each second segment is obtained;

[0017] The average weight is compared with a first threshold, and the second segment corresponding to the average weight that is greater than or equal to the first threshold is used as training data.

[0018] In one exemplary embodiment, obtaining the weights of each word in each of the first segments using an unsupervised training algorithm includes:

[0019] Based on the TextRank algorithm, the weights of each word in each of the first segments are obtained.

[0020] In one exemplary embodiment, training the word segmentation model using the training data until the word segmentation model meets preset requirements includes:

[0021] Mask some characters in the training data and input the masked data into the word segmentation model;

[0022] Obtain the data generated by the word segmentation model based on the data after the masked part of the character, and obtain the first loss function based on the difference between the generated data and the training data;

[0023] Obtain the second loss function corresponding to the word segmentation model when performing semantic segmentation;

[0024] The total loss function is obtained based on the first loss function and the second loss function, and the word segmentation model is trained until the total loss function meets the preset requirements.

[0025] According to one aspect of the embodiments of this application, a word segmentation method is disclosed, including:

[0026] Receive the target text to be segmented;

[0027] The target text is segmented using the segmentation model trained by the method described above, and the segmentation results are obtained.

[0028] In one exemplary embodiment, the step of segmenting the target text using a word segmentation model trained using the word segmentation model training method described above, and obtaining the word segmentation result, includes:

[0029] The target text is segmented into multiple words using various segmentation methods;

[0030] The scores for each of the aforementioned words are obtained based on the dynamic programming algorithm;

[0031] The word with the highest score is used as the word segmentation result.

[0032] In one exemplary embodiment, obtaining the scores for each of the words based on the dynamic programming algorithm includes:

[0033] Based on the word segmentation model, the first probability corresponding to each of the multiple segmentation methods is obtained;

[0034] The second probability is obtained based on the frequency of the word's occurrence in the terminology set;

[0035] Based on the first probability and the second probability, the total probability is obtained by interpolation.

[0036] Based on the total probability, the score of each word is obtained by solving the recursive algorithm of the dynamic programming algorithm.

[0037] According to one aspect of the embodiments of this application, a word segmentation model training system is disclosed, comprising:

[0038] The acquisition module is used to acquire a terminology set corresponding to the target domain, wherein the terminology set includes multiple professional terms of the target domain;

[0039] A matching module is used to match the multiple technical terms with the corpus, and to use the technical terms matched in the corpus as anchor words; the corpus contains texts using at least some of the technical terms in the target field.

[0040] The processing module is used to acquire text fragments containing the anchor words in the corpus and use the text fragments as training data.

[0041] The training module is used to train the word segmentation model using the training data until the word segmentation model meets the preset requirements.

[0042] According to one aspect of the embodiments of this application, a computer device is disclosed, the computer device including one or more processors and a memory, the memory being used to store one or more programs, which, when executed by the one or more processors, enable the computer device to implement the aforementioned word segmentation model training method.

[0043] According to one aspect of the embodiments of this application, a computer device is disclosed, the computer device including one or more processors and a memory, the memory being used to store one or more programs, which, when executed by the one or more processors, cause the computer device to implement the aforementioned word segmentation method.

[0044] According to one aspect of the embodiments of this application, a computer-readable storage medium is disclosed, the computer-readable storage medium storing computer-readable instructions, which, when executed by a computer's processor, cause the computer to perform the aforementioned word segmentation model training method.

[0045] According to one aspect of the embodiments of this application, a computer-readable storage medium is disclosed, the computer-readable storage medium storing computer-readable instructions, which, when executed by a computer's processor, cause the computer to perform the aforementioned word segmentation method.

[0046] The technical solutions provided by the embodiments of this application have at least the following beneficial effects:

[0047] The word segmentation model training scheme provided in this application obtains a terminology set corresponding to the target domain, matches the specialized terms in the terminology set with a corpus to obtain anchor words, and uses text fragments containing anchor words in the corpus as training data. The word segmentation model is then trained using this training data. Since anchor words are terms strongly related to their respective domains, other terms easily appear around them. By obtaining anchor words through matching with the corpus, potentially specialized terminology-containing text content can be filtered out for training the word segmentation model. This method requires no labeled data, is simple, and highly efficient.

[0048] The word segmentation model training scheme provided in this application uses a random missing character method in text generation tasks to enable the word segmentation model to learn the probability of predicting the next character given an input text segment; while in semantic representation tasks, it brings the semantics of the professional terms in the terminology set closer to their context in the corpus, while also distinguishing them from other professional terms as much as possible in terms of semantics, which is equivalent to implicitly learning the semantic boundary division between professional terms.

[0049] The word segmentation scheme provided in this application combines word segmentation models with dynamic programming. Utilizing the probabilities output by training a large-scale word segmentation model, it can effectively correct the probability results of terminology sets based on simple word frequency statistics, thus obtaining more accurate scores for each word. Furthermore, through a dynamic programming algorithm, the score of each word can be recursively calculated from the end of the text segment to the beginning, effectively handling the segmentation of important but less commonly used professional terms in specific domains. Moreover, the word segmentation model only needs to be trained once before use, and the dynamic programming method is highly efficient. Overall, it reduces costs while improving performance.

[0050] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this application. Attached Figure Description

[0051] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0052] Figure 1 This is a flowchart illustrating a word segmentation model training method according to an exemplary embodiment.

[0053] Figure 2 yes Figure 1 A detailed flowchart of step S103 in the corresponding embodiment.

[0054] Figure 3 This is a schematic diagram of a second sliding window word selection according to an exemplary embodiment.

[0055] Figure 4 yes Figure 1 A detailed flowchart of step S104 in the corresponding embodiment.

[0056] Figure 5 This is a schematic diagram illustrating a word segmentation model according to an exemplary embodiment.

[0057] Figure 6 This is a flowchart illustrating a word segmentation method according to an exemplary embodiment.

[0058] Figure 7 yes Figure 6 A detailed flowchart of step S202 in the corresponding embodiment.

[0059] Figure 8 This is a schematic diagram illustrating text segmentation according to an exemplary embodiment.

[0060] Figure 9 yes Figure 7 A detailed flowchart of step S2022 in the corresponding embodiment.

[0061] Figure 10 This is a block diagram illustrating a word segmentation model training system according to an exemplary embodiment.

[0062] Figure 11 This is a block diagram illustrating a word segmentation model according to an exemplary embodiment.

[0063] Figure 12 This is a computer system architecture block diagram illustrating a computer device for implementing embodiments of the present application, according to an exemplary embodiment.

[0064] The annotations in the attached figures are explained as follows:

[0065] 101. Encoding unit; 102. Decoding unit; 200. Word segmentation model training system; 201. Acquisition module; 202. Matching module; 203. Processing module; 204. Training module; 300. Word segmentation model; 301. Text receiving module; 302. Segmentation module; 303. Score calculation module; 304. Word segmentation module; 400. Computer system; 401. CPU; 402. ROM; 403. Storage section; 404. RAM; 405. Bus; 406. I / O interface; 407. Input section; 408. Output section; 409. Communication section; 410. Driver; 411. Removable medium. Detailed Implementation

[0066] Although this application can be readily embodied in various forms of implementation, only some specific embodiments are shown in the accompanying drawings and will be described in detail in this specification. It is understood that this specification should be regarded as an exemplary illustration of the principles of this application and is not intended to limit the application to what is described herein.

[0067] Furthermore, the terms "comprising," "including," "having," and any variations thereof used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the steps or modules listed, but may optionally include other steps or modules not listed, or may optionally include other steps or modules inherent to such processes, methods, products, or devices.

[0068] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features.

[0069] It should be noted that in the embodiments of this application, the words "exemplary," "illustrated," or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary," "illustrated," or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary," "illustrated," or "for example" is intended to present the relevant concepts in a specific manner.

[0070] The exemplary embodiments will now be described in detail. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the summary of the invention.

[0071] First, let me explain some of the terms used in this application:

[0072] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0073] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information retrieval, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0074] Corpus: A corpus is a structured, representative, and scalable collection of language data collected specifically for one or more application goals. Essentially, a corpus represents the overall language use identified in a study by randomly sampling natural language usage. A corpus typically refers to a large number of language instances that are practically impossible to observe in statistical natural language processing.

[0075] TextRank Algorithm: The TextRank algorithm is a graph-based ranking algorithm for text. Its basic idea originates from Google's PageRank algorithm. By segmenting text into constituent units (words, sentences) and building a graph model, it uses a voting mechanism to rank the important components of the text, achieving keyword extraction and summarization using only the information within a single document.

[0076] Dynamic programming (DP) algorithm: Dynamic programming is generally used to find the optimal solution to optimization problems, which have multiple solutions, each with a value. Dynamic programming involves planning and computationally oriented strategies by decomposing the original problem into several overlapping subproblems. These subproblems are similar in form to the original problem but smaller in scale. When solving problems with dynamic programming, we often refer to a set of values ​​for the variables related to a subproblem as a "state." A "state" corresponds to one or more subproblems, and the "value" in a particular "state" is the solution to the corresponding subproblem. The set of all "states" constitutes the problem's "state space." The size of the "state space" is directly related to the time complexity of solving the problem using dynamic programming.

[0077] A recursive algorithm, in computer science, is a method for solving a problem by repeatedly breaking it down into similar subproblems. Recursive methods can be used to solve many computer science problems, making it a crucial concept in the field. Most programming languages ​​support self-calling of functions, allowing functions to recursively call themselves. Computational theory has proven that recursion can completely replace loops; therefore, many functional programming languages ​​(such as Scheme) commonly use recursion to implement loops.

[0078] This application provides a word segmentation model training method, word segmentation method, word segmentation model training system, computer equipment, and computer-readable storage medium. It enables intelligent word segmentation of text in a specific domain based on artificial intelligence technology, requiring no labeled data, featuring a simple algorithm and high efficiency. The word segmentation model training method and word segmentation method provided in this application can be applied to specialized fields containing a large number of specialized terms, such as the medical field, as well as to general fields.

[0079] The word segmentation model training method, word segmentation method, word segmentation model training system, computer device, and computer-readable storage medium provided in this application are specifically described through the following embodiments. First, the word segmentation model training method in this application embodiment is described.

[0080] This application first provides a word segmentation model training method. The word segmentation model training method includes:

[0081] Obtain the terminology set corresponding to the target field, which includes multiple professional terms in the target field;

[0082] Matching multiple technical terms with corpora, the technical terms matched in the corpus will be used as anchor words; the corpus contains texts using at least some of the technical terms in the target domain;

[0083] Obtain text fragments containing anchor words from the corpus and use these text fragments as training data;

[0084] The word segmentation model is trained using training data until it meets the preset requirements.

[0085] The technical solution provided in this application obtains a terminology set corresponding to the target domain, matches the specialized terms in the terminology set with a corpus to obtain anchor words, and uses text fragments containing anchor words in the corpus as training data. Then, the training data is used to train a word segmentation model. Since anchor words are terms strongly related to their respective domains, other terms easily appear around them. By obtaining anchor words through matching with a corpus, text content potentially containing specialized terms can be filtered out for training the word segmentation model. No labeled data is required, the algorithm is simple, and highly efficient. It is particularly suitable for domains with high specialization and dense terminology.

[0086] The embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0087] See Figure 1 As shown, an exemplary embodiment of this application provides a word segmentation model training method, which includes the following steps S101 to S104.

[0088] S101, Obtain the terminology set corresponding to the target field. The terminology set includes multiple professional terms in the target field.

[0089] In detail, the terminology set includes a list of specialized terms in the target field and the term frequency (TF) for each term. The TF is the frequency with which the term appears in the terminology set. The specialized terms included in the terminology set can be collected in advance or directly integrated from existing specialized terminology databases.

[0090] The target field is not limited to one or more specific fields; for example, it could be the medical field.

[0091] A word segmentation model can be applied to one domain or to multiple domains. Understandably, if a word segmentation model is applied to multiple domains, the word segmentation model training method of this application can be used to obtain the terminology sets corresponding to each domain, so as to further train the word segmentation model for each domain.

[0092] S102 matches multiple technical terms and corpora, and uses the technical terms matched in the corpus as anchor words.

[0093] The corpus contains texts showcasing at least a portion of the specialized terminology used in the target domain. Corpora can be, for example, books, internet encyclopedia data, or app data.

[0094] Specifically, in step S102, string matching technology is used to match multiple technical terms with the corpus. In addition to identifying anchor words, it is also possible to obtain the surrounding sentences of the anchor words in the corpus, and thus obtain relevant information about the anchor words in the corpus, such as which paragraph, sentence, and position of the anchor words in the corpus text.

[0095] S103: Obtain text fragments containing anchor words from the corpus and use these text fragments as training data.

[0096] In one exemplary embodiment, such as Figure 2 As shown, step S103 includes the following steps S1031 to S1035.

[0097] S1031, retrieve the sentence containing the anchor word from the corpus, and use the entire sentence containing the anchor word as the anchor sentence.

[0098] In detail, expand outwards from the anchor word to the left and right, until the beginning and end of the sentence, to obtain the entire sentence containing the anchor word, and use the entire sentence containing the anchor word as the anchor sentence.

[0099] In S1032, a first sliding window is used to intercept a number of first segments from the anchor sentence, and an unsupervised training algorithm is adopted to obtain the weight of each character in each first segment.

[0100] Specifically, the weight of each character in each first segment is obtained based on the TextRank algorithm.

[0101] In more detail, by setting the first sliding window with a size of N, a number of first segments in the anchor sentence are intercepted; then, on a character basis, characters located in the same sliding window (first segment) in the anchor sentence are connected by an edge to form a graph; then, unsupervised training is performed, and the score (weight) of each character is calculated and obtained based on the following mapping relationship:

[0102]

[0103] Herein, V i represents a character, W(V i ) represents the score of character V i , d represents a damping coefficient with a value range of 0-1, representing the probability of pointing from a specific point in the graph to any other arbitrary point, which is generally set to 0.85, In(V i ) represents the set composed of characters pointing to character V i , V j represents a character, w ji represents the weight of the edge from character V j to character V i , w jk represents the weight of the edge from character V k to character V j , Out(V j ) represents the set composed of characters pointed to by character V j , v k represents a character.

[0104] For example, if the anchor point word obtained in step S102 is "body deficiency", then in step S1031, the entire sentence where "body deficiency" is located is obtained, for example: "Long-term illness with body deficiency, failure to control blood, overstrain and excessive sexual indulgence, or long-term illness with body deficiency, leading to insufficiency of qi and yin of the heart, spleen and kidney, and bleeding due to blood failing to circulate along the channels."

[0105] If the size of the first sliding window is set to 3, for the character "deficiency", the nodes connected thereto include "body deficiency prolonged", "deficiency prolonged illness", "illness body deficiency", "deficiency leads to", etc., and the score of each character in the first sliding window is calculated through the above first mapping relationship.

[0106] In this step S1032, the first segment is obtained by intercepting based on the anchor sentence, and then the score of each character is calculated, which can not only prevent the sliding window from going beyond the sentence boundary, but also the semantics of the sentence are generally complete, contributing to better score calculation.

[0107] S1033, multiple different second sliding windows are used to extract anchor sentences respectively, and multiple second segments containing anchor words are obtained.

[0108] Unlike the TextRank algorithm, this application calculates the score W(V) for each character. i After that, no further processing such as setting a scoring threshold is performed to obtain keywords. In this application, the anchor word is used as the center again, and multiple different second sliding windows are used to extract words, obtaining multiple second segments corresponding to each anchor word, which are then filtered in step S1034 to obtain the second segments with larger average weights as training data. An example of second sliding window word extraction is as follows: Figure 3 As shown.

[0109] Understandably, since the length of a word is generally 2 to 6 characters, the size of the second sliding window is generally set to 2 to 6 characters.

[0110] S1034, based on the weights of each character in the first segment, obtain the average weight of all characters in each second segment.

[0111] In detail, because for each word V i The score W(V) has already been calculated in the aforementioned step S1032. i Therefore, based on the score of each character, the sum of the scores of all characters in the second segment can be calculated, and then the average score (i.e., the weighted average) of all characters in the second segment can be calculated.

[0112] Understandably, if there is a word with an unknown score in the second segment, that is, none of the first segments in the aforementioned step S1032 contain the word, the score of the word can be defined as zero.

[0113] S1035, compare the weight mean with the first threshold, and use the second segment corresponding to the weight mean that is greater than or equal to the first threshold as training data.

[0114] The first threshold can be a value set based on experience. The higher the average weight of all words in the second segment, the higher the probability of forming a word. Therefore, the second segment with an average weight less than the first threshold is removed, and only the second segment with an average weight greater than or equal to the first threshold is retained as training data.

[0115] S104: Train the word segmentation model using training data until the word segmentation model meets the preset requirements.

[0116] In detail, the performance of the word segmentation model is measured by a loss function. The smaller the loss function, the higher the accuracy of the word segmentation model; conversely, the larger the loss function, the lower the accuracy of the word segmentation model. In step S104, the word segmentation model is trained using training data until the loss function of the word segmentation model meets the preset requirements.

[0117] More specifically, the preset requirement could be: the loss function is less than a second threshold. If the loss function is greater than or equal to the second threshold, continue training the word segmentation model; if the loss function decreases to less than the second threshold, then terminate the training of the word segmentation model.

[0118] When training a word segmentation model using training data, on the one hand, the model needs to learn how to generate second segments containing anchor words; on the other hand, the model needs to make the semantic representation of the anchor word as close as possible to the semantic representation of the segment it belongs to. Based on these two training objectives, a language generation loss function (i.e., the first loss function) and a semantic loss function (i.e., the second loss function) are designed. The word segmentation model is trained by combining the two loss functions until the loss meets the preset requirements.

[0119] In detail, in order to reduce the difficulty of training the word segmentation model, in one embodiment, instead of having the word segmentation model generate text fragments containing anchor words from scratch, the characters of all second fragments containing anchor words are randomly masked, and then the word segmentation model is used to predict the masked character content in reverse during the decoding process.

[0120] In detail, the semantic loss function is calculated through the following mapping relationship:

[0121]

[0122] Among them, L SS Let h represent the semantic loss function. i Anchor word V i * The semantic vector encoded by the word segmentation model Anchor word V i * The semantic vector of the second segment, This represents the semantic vector of the second segment containing other anchor words; therefore, i ≠ j. Anchor word V i * The dot product between the semantics of the segment and the segment it belongs to represents the similarity. Anchor word V i * The dot product between the semantics of the two segments represents the similarity.

[0123] In the text generation task, this application uses a random missing character method to enable the word segmentation model to learn the probability of predicting the next character given the input text; while in the semantic representation task, it brings the semantics of the professional terms in the terminology set closer to their context in the corpus, while also distinguishing them from other professional terms as much as possible in terms of semantics, which is equivalent to implicitly learning the semantic boundary division between professional terms.

[0124] In one exemplary embodiment, such as Figure 4 As shown, step S104 includes the following steps S1041 to S1044.

[0125] S1041, Mask some characters in the training data, and input the data after masking some characters into the word segmentation model.

[0126] See Figure 5 As shown, Figure 5 This is a schematic diagram illustrating a word segmentation model according to an exemplary embodiment. The word segmentation model includes an encoding unit 101 and a decoding unit 102. For example, for the word "yin deficiency and excessive fire", the characters "xu" and "wang" are masked before being input into the encoding unit 101 for encoding. Then, the decoding unit 102 restores "xu" and "wang" to obtain "yin deficiency and excessive fire". In step S1042, the first loss function can be obtained based on the difference between the data restored by the decoding unit 102 and the original training data.

[0127] S1042, obtain the data generated by the word segmentation model based on the data after masking some characters, and obtain the first loss function based on the difference between the generated data and the training data.

[0128] S1043, obtain the second loss function corresponding to the word segmentation model when performing semantic segmentation.

[0129] S1044, obtain the total loss function based on the first loss function and the second loss function, and train the word segmentation model until the total loss function meets the preset requirements.

[0130] For example, the first loss function is denoted as L MLM The second loss function is expressed as L SS The total loss function is expressed as Loss: Loss = L MLM +α*L SS Here, α represents a coefficient, with a value between 0 and 1. By setting the value of α, the second loss function L is adjusted. SS The magnitude of its influence on the total loss function.

[0131] Understandably, in some embodiments, only one of the first loss function and the second loss function may be considered. For example, steps S1043 and S1044 may not be executed. After step S1042, the first loss function is used as the criterion for determining whether to end the training. The word segmentation model is trained until the first loss function meets the preset requirements.

[0132] Based on the aforementioned training method, the obtained word segmentation model can segment a text X = {x1, ... x2} according to a given input text. i}, predict the next character x i+1 .

[0133] Please refer to the following. Figure 6 This application provides an exemplary embodiment of a word segmentation model and method. This word segmentation method can be applied to the medical field, such as to intelligent consultation systems. When conducting a consultation dialogue in an intelligent consultation system, after obtaining the text content input by the user, a word segmentation model is first used to segment the text content. For example, word segmentation can identify proper nouns and concepts appearing in the text, thereby further performing semantic analysis to understand the user's intent. Based on the understanding of the user's intent, further analysis and processing are performed, such as replying to the user's consultation questions, or further inquiring about the user based on an intelligent consultation template, etc.

[0134] For example, when applying this to a medical information retrieval system to search for patients with specific symptoms or diseases, the system first uses a word segmentation model to segment the input text, obtaining the technical terms representing the specific symptoms or diseases. Then, based on the technical terms obtained from word segmentation and a database storing information about the symptoms or diseases of the patients, a character matching algorithm is executed to retrieve the patients with the specific symptoms or diseases. Accurate word segmentation can greatly improve the accuracy of the search results.

[0135] For example, in medical research scenarios, word segmentation can be used as a preprocessing step. A word segmentation model can be used to segment a large number of research papers to obtain the professional terms in the research papers. Then, the keywords (professional terms) to be studied can be found from the obtained professional terms, which makes it convenient for researchers to summarize, organize and retrieve them.

[0136] like Figure 6 As shown, the word segmentation method includes the following steps S201 to S202.

[0137] S201, Receive the target text to be segmented.

[0138] S202, use the trained word segmentation model to segment the target text and obtain the segmentation results.

[0139] In an exemplary embodiment, as shown in Figure 7 , step S202 includes the following steps S2021 to S2023.

[0140] In S2021, the target text is segmented into a plurality of words by using a plurality of segmentation manners.

[0141] Specifically, for a piece of target text to be segmented that is input to a word segmentation model, a Directed Acyclic Graph of all word segmentation possibilities of the target text may be generated. The directed acyclic graph refers to a directed graph with no cycles (loops) in the graph.

[0142] For example, the target text is: "The occurrence of depression syndrome is related to liver qi stagnation". In step S2021, the target text is segmented into a plurality of words by using a plurality of segmentation manners, as shown in Figure 8 . Wherein, Figure 8 the arrow lines in the figure represent a word segmentation possibility. For example, single-character segmentation is also a segmentation manner, so arrows are connected between each character. A plurality of arrows represent a plurality of segmentation methods. For example, starting from the character "liver", there may be three segmentation manners of "liver qi", "liver qi depression" and "liver qi stagnation".

[0143] In S2022, the score of each word is obtained based on a dynamic programming algorithm.

[0144] In an exemplary embodiment, as shown in Figure 9 , step S2022 includes the following steps S20221 to S20224.

[0145] In S20221, first probabilities corresponding to the plurality of segmentation manners are respectively obtained based on the word segmentation model.

[0146] In S20222, a second probability is obtained based on the frequency of the word appearing in the term set.

[0147] In S20223, a total probability is obtained by using an interpolation method based on the first probability and the second probability.

[0148] In S20224, the score of each word is obtained by solving the recursive algorithm of the dynamic programming algorithm based on the total probability.

[0149] Combining the word segmentation model with dynamic programming-based word segmentation, the first probability output by large-scale word segmentation model training can effectively correct the deviation of the probability (the second probability) result obtained by simple statistics based on word frequency for the term set, so that the scores of each word can be obtained more accurately. Further, through the dynamic programming algorithm, the score of each word can be calculated recursively from the back to the front of the text segment, which realizes effective handling of word segmentation for professional terms that are very important in a specific field but relatively uncommon. Moreover, the word segmentation model only needs to be trained once before use, the dynamic programming method has high operation efficiency, and overall, it can reduce costs and improve performance at the same time.

[0150] S2023, the word with the highest score is taken as the word segmentation result.

[0151] One specific embodiment:

[0152] Assume that the current processing position is, for example Figure 8 "Gan (liver)" as indicated by the arrow in, using a recursive algorithm, the score Score"Gan Qi Yu Jie" for the word segmentation being "Gan Qi Yu Jie (stagnation of liver qi)" is:

[0153] Score"Gan Qi Yu Jie"=W"Gan Qi Yu Jie"+Path_Score"Yu Jie (stagnation)";

[0154] Wherein, W"Gan Qi Yu Jie" represents the probability of the word "Gan Qi Yu Jie", and Path_Score("Yu Jie" represents the score starting with "Yu Jie".

[0155] For Figure 8 the score starting with "Gan (liver)" shown is:

[0156] Path_Score("Gan")=max(Score("Gan Qi", Score("Gan Qi Yu", Score("Gan Qi Yu Jie";

[0157] Wherein, Path_Score is the score, and the word with the maximum score among multiple possibilities is taken as the word segmentation result. For example, if Score"Gan Qi Yu Jie" is the maximum, then "Gan Qi Yu Jie" is taken as one word.

[0158] Wherein, the calculation of the probability W is based on the first probability obtained by the word segmentation model and the second probability obtained from the occurrence frequency of the word in the term set. In detail, for a segmentation w composed of n characters w={w1,…w n}, the probability LM(1,…w of this segmentation can be calculated through the word segmentation model obtained by training n ). At the same time, the second probability P is obtained based on the occurrence frequency of the word obtained from this segmentation in the term set vocab (w). Finally, the interpolation method is used to calculate the total probability W:

[0159] W=λ1LM(wi ,…w n )+(1-λ1)P vocab (w);

[0160] Here, λ1 represents the coefficient, which takes a value of 0-1. By setting the value of λ1, the influence of the first probability and the second probability on the total probability can be adjusted.

[0161] Substituting the total probability W into the mapping relationship Score("Liver Qi Stagnation" = W"Liver Qi Stagnation" + Path_Score("Stagnation"), the score of the word can be obtained.

[0162] Understandably, obtaining the scores of each word based on the dynamic programming algorithm is not limited to using a recursive algorithm; other algorithms can also be used, such as bottom-up analysis method and top-down analysis method.

[0163] Please refer to the following. Figure 10 , Figure 10 This is a block diagram illustrating a word segmentation model training system 200 according to an exemplary embodiment, which can perform... Figures 1 to 2 and Figure 4 All or part of the steps of any of the word segmentation model training methods shown. For example... Figure 10 As shown, the word segmentation model training system 200 includes, but is not limited to: an acquisition module 201, a matching module 202, a processing module 203, and a training module 204.

[0164] The acquisition module 201 is used to acquire the terminology set corresponding to the target field, which includes multiple professional terms in the target field.

[0165] The matching module 202 is used to match multiple technical terms and corpora, and uses the technical terms matched in the corpus as anchor words; the corpus contains texts using at least some of the technical terms in the target field.

[0166] The processing module 203 is used to obtain text fragments containing anchor words from the corpus and use the text fragments as training data.

[0167] The training module 204 is used to train the word segmentation model using training data until the word segmentation model meets the preset requirements.

[0168] In detail, the processing module 203 obtains the sentence containing the anchor word from the corpus and takes the entire sentence containing the anchor word as the anchor sentence; it uses a first sliding window to extract several first segments from the anchor sentence and uses an unsupervised training algorithm to obtain the weight of each character in each first segment; it uses multiple different second sliding windows to extract the anchor sentence respectively to obtain multiple second segments containing the anchor word; based on the weight of each character in the first segment, it obtains the average weight of all characters in each second segment; it compares the average weight with a first threshold and takes the second segment corresponding to the average weight that is greater than or equal to the first threshold as training data.

[0169] In detail, the training module 204 masks some characters in the training data and inputs the masked data into the word segmentation model; it obtains the data generated by the word segmentation model based on the masked data, and obtains a first loss function based on the difference between the generated data and the training data; it obtains a second loss function corresponding to the word segmentation model when performing semantic segmentation; it obtains a total loss function based on the first and second loss functions, and trains the word segmentation model until the total loss function meets the preset requirements.

[0170] For a more detailed explanation of the functions and roles of each module in the above-mentioned word segmentation model training system 200, please refer to the implementation process of the corresponding steps in the above-mentioned word segmentation model training method, which will not be repeated here.

[0171] The word segmentation model training system 200 can be installed on any computer device with data processing capabilities, such as a desktop computer or a laptop computer.

[0172] Please refer to the following. Figure 11 , Figure 11 This is a block diagram illustrating a word segmentation model 300 according to an exemplary embodiment, which can perform... Figure 6 , Figure 7 , Figure 9 All or part of the steps of any of the word segmentation methods shown. For example... Figure 11 As shown, the word segmentation model 300 includes, but is not limited to: a text receiving module 301, a segmentation module 302, a score calculation module 303, and a word segmentation module 304.

[0173] The text receiving module 301 is used to receive the target text to be segmented.

[0174] The segmentation module 302 is used to segment the target text into multiple words using various segmentation methods.

[0175] The score calculation module 303 is used to obtain the score of each word based on the dynamic programming algorithm.

[0176] The word segmentation module 304 is used to output the word with the highest score as the word segmentation result.

[0177] For a more detailed explanation of the functions and roles of each module in the above word segmentation model 300, please refer to the implementation process of the corresponding steps in the above word segmentation method, which will not be repeated here.

[0178] The word segmentation model 300 can be installed on any computer device with data processing capabilities, such as a desktop computer or a laptop computer.

[0179] Please refer to the following. Figure 12 , Figure 12 A schematic diagram of a computer system architecture for implementing a word segmentation model training method or a word segmentation method according to embodiments of this application is shown.

[0180] It should be noted that, Figure 12 The computer system 400 of the computer device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0181] like Figure 12 As shown, the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 402 or programs loaded from storage section 403 into random access memory (RAM). The RAM 404 also stores various programs and data required for system operation. The CPU 401, ROM 402, and RAM 404 are interconnected via a bus 405. An input / output interface 406 (I / O interface) is also connected to the bus 405.

[0182] The following components are connected to the input / output interface 406: an input section 407 including a keyboard, mouse, etc.; an output section 408 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 403 including a hard disk, etc.; and a communication section 409 including a network interface card such as a local area network card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output interface 406 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 403 as needed.

[0183] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit 401, it performs various functions defined in the system of this application.

[0184] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0185] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this application can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or code on a computer-readable storage medium.

[0186] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0187] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, modular division is only a logical functional division, and other division methods may exist in actual implementation. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed.

[0188] It should be understood that this application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A word segmentation method, characterized in that, include: Receive the target text to be segmented; The target text is segmented using a word segmentation model trained using a word segmentation model training method, and the segmentation results are obtained, including: The target text is segmented into multiple words using various segmentation methods; Based on the word segmentation model, the first probability corresponding to each of the multiple segmentation methods is obtained; A second probability is obtained based on the frequency of the word's occurrence in the terminology set; Based on the first probability and the second probability, the total probability is obtained by interpolation. Based on the total probability, the score of each word is obtained by solving a recursive algorithm of dynamic programming. The word with the highest score is used as the word segmentation result; The word segmentation model training method includes: Obtain a term set corresponding to the target domain, the term set including multiple professional terms in the target domain and the word frequency corresponding to each professional term, the word frequency being the frequency of the professional term appearing in the term set; The string matching technique is used to match the multiple technical terms with the corpus, and the technical terms matched in the corpus are used as anchor words; the corpus contains texts of at least some technical terms in the target field. The process involves obtaining text fragments containing the anchor words from the corpus and using these fragments as training data. This includes: obtaining the sentences containing the anchor words from the corpus and using the entire sentence containing the anchor words as the anchor sentence; using a first sliding window to extract several first segments from the anchor sentence; connecting characters within the same first segment in the anchor sentence with an edge to form a graph; and performing unsupervised training based on mapping relationships. The weights of each character in the first segment are calculated, where, Representing characters, Representation word The weight is given by d, where d represents the damping coefficient, and d represents the probability of moving from a specific point in the graph to any other point. Indicates the word to be pointed to A collection of characters. Representing characters, Representation word To the word The weight of the edge. Representation word To the word The weight of the edge. Representation word The set of characters it points to. The anchor sentence is extracted using multiple different second sliding windows to obtain multiple second segments containing the anchor word; based on the weight of each character in the first segment, the average weight of all characters in each second segment is obtained; the average weight is compared with a first threshold, and the second segment corresponding to the average weight that is greater than or equal to the first threshold is used as training data. The word segmentation model is trained using the training data until it meets the preset requirements.

2. The word segmentation method according to claim 1, characterized in that, The step of training a word segmentation model using the training data until the word segmentation model meets preset requirements includes: Mask some characters in the training data and input the masked data into the word segmentation model; Obtain the data generated by the word segmentation model based on the data after the masked part of the character, and obtain the first loss function based on the difference between the generated data and the training data; Obtain the second loss function corresponding to the word segmentation model when performing semantic segmentation; The total loss function is obtained based on the first loss function and the second loss function, and the word segmentation model is trained until the total loss function meets the preset requirements.

3. A computer device, characterized in that, include: One or more processors; A memory for storing one or more programs, which, when executed by one or more processors, cause the computer device to implement the word segmentation method as described in any one of claims 1 to 2.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions that, when executed by a computer's processor, cause the computer to perform the word segmentation method as described in any one of claims 1 to 2.

Images