Word segmentation method and device, computer device and storage medium

Abstract

Embodiments of the application disclose a word segmentation method and device, computer equipment and a storage medium, belonging to the technical field of computers. The method comprises: performing word segmentation on a text to obtain a plurality of first words; performing feature extraction on each first word by using a large language model to obtain the features of each first word; determining the similarity of a plurality of word groups based on the features of the plurality of first words; determining the co-occurrence frequency of each word group; fusing the similarity and the co-occurrence frequency of each word group to obtain the combination probability of each word group; and based on the combination probability of the plurality of word groups, forming a second word from two first words in a word group with the largest combination probability, or forming a second word from two first words in a word group with a combination probability greater than a first threshold. The application can capture the context relationship between characters in the text to obtain more accurate words, ensure the accuracy of the words obtained by word segmentation, and further ensure the accuracy of word segmentation.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a word segmentation method, apparatus, computer device, and storage medium. Background Technology

[0002] In many natural language processing tasks, text segmentation is often involved to utilize the segmented results for model training or other tasks. While some segmentation tools or models are used to segment text, this method often suffers from low accuracy. Summary of the Invention

[0003] This application provides a word segmentation method, apparatus, computer device, and storage medium, which can improve the accuracy of word segmentation. The technical solution is as follows:

[0004] On the one hand, a word segmentation method is provided, the method comprising:

[0005] The text is segmented to obtain multiple first words, each of which contains at least one character;

[0006] By using a large language model, feature extraction is performed on each of the first words to obtain the features of each of the first words;

[0007] Based on the features of the multiple first words, the similarity of multiple word groups is determined. Each word group includes two adjacent first words in the text, and the similarity of the word group indicates the similarity between the two first words in the word group.

[0008] Determine the co-occurrence frequency of each word group, the co-occurrence frequency of which indicates the frequency at which two first words in the word group appear in adjacent forms in a text database, the text database including multiple reference texts;

[0009] The similarity and co-occurrence frequency of each word group are fused to obtain the combination probability of each word group. The combination probability of the word group indicates the probability that two first words in the word group can form a word.

[0010] Based on the combination probability of the multiple word groups, the two first words in the word group with the highest combination probability are combined to form the second word, or the two first words in the word group with a combination probability greater than the first threshold are combined to form the second word.

[0011] In one possible implementation, determining the co-occurrence frequency of each word group includes:

[0012] Based on the two first words in the phrase, the multiple reference texts are traversed to determine the number of times the two first words in the phrase appear in an adjacent form;

[0013] Determine the number of words contained in the plurality of reference texts;

[0014] The ratio of the number of occurrences to the number of words is determined as the co-occurrence frequency of the word group.

[0015] In another possible implementation, after combining the two first words from the word group with the highest combination probability based on the combination probability of the multiple word groups to form the second word, the method further includes:

[0016] The second word is validated based on the words in the vocabulary database, which is used to store words with entity meanings.

[0017] If the second word passes the verification, the second word will be added to the word database.

[0018] In another possible implementation, the validation of the second word based on the vocabulary database includes:

[0019] Detect whether the second word exists in the vocabulary database;

[0020] Detect whether the number of words in the vocabulary database has reached the second threshold;

[0021] If the second word does not exist in the vocabulary database and the number of words in the vocabulary database does not reach the second threshold, it is determined that the verification of the second word has passed;

[0022] If the second word exists in the vocabulary database, or if the number of words in the vocabulary database reaches the second threshold, it is determined that the verification of the second word has failed.

[0023] In another possible implementation, after combining the two first words from the word group with the highest combination probability based on the combination probability of the multiple word groups to form the second word, the method further includes:

[0024] The first word is validated based on the words in the vocabulary database.

[0025] If the first word passes the verification, the first word is added to the word database.

[0026] In another possible implementation, after combining the two first words from the word group with the highest combination probability based on the combination probability of the multiple word groups to form the second word, the method further includes:

[0027] By using a large language model, features of the second word are extracted to obtain the features of the second word;

[0028] Based on the features of the second word and the features of the third word, the similarity between the second word and the third word is determined, wherein the third word is the first word in the text that is adjacent to the second word;

[0029] Determine the co-occurrence frequency of the second word and the third word, wherein the co-occurrence frequency indicates the frequency at which the second word and the third word appear in adjacent forms in the text database;

[0030] The similarity and co-occurrence frequency of the second word and the third word are fused to obtain the combination probability of the second word and the third word. The combination probability of the second word and the third word indicates the possibility that the second word and the third word can form a word.

[0031] If the probability of combining the second word and the third word reaches the first threshold, the second word and the third word are combined to form the fourth word.

[0032] On the other hand, a word segmentation device is provided, the device comprising:

[0033] The word segmentation module is used to segment the text into words, resulting in multiple first words, each of which contains at least one character;

[0034] The feature extraction module is used to extract features from each of the first words using a large language model to obtain the features of each of the first words.

[0035] A determination module is used to determine the similarity of multiple word groups based on the features of the multiple first words, each word group including two adjacent first words in the text, and the similarity of the word group indicates the similarity between the two first words in the word group;

[0036] The determining module is further configured to determine the co-occurrence frequency of each word group, wherein the co-occurrence frequency of the word group indicates the frequency at which two first words in the word group appear in an adjacent form in a text database, and the text database includes multiple reference texts;

[0037] The fusion module is used to fuse the similarity and co-occurrence frequency of each word group to obtain the combination probability of each word group. The combination probability of the word group indicates the probability that two first words in the word group can form a word.

[0038] The combination module is used to combine two first words from the word group with the highest combination probability to form a second word, or to combine two first words from the word group with a combination probability greater than a first threshold to form a second word.

[0039] In one possible implementation, the determining module is configured to, based on two first words in the phrase, traverse the plurality of reference texts to determine the number of times the two first words in the phrase appear in an adjacent form; determine the number of words contained in the plurality of reference texts; and determine the ratio of the number of occurrences to the number of words as the co-occurrence frequency of the phrase.

[0040] In another possible implementation, the device further includes:

[0041] The verification module is used to verify the second word based on the words in the vocabulary database, wherein the vocabulary database is used to store words with entity meanings;

[0042] An add module is used to add the second word to the vocabulary database if the second word passes the validation.

[0043] In another possible implementation, the verification module is used to detect whether the second word exists in the vocabulary database; detect whether the number of words in the vocabulary database reaches a second threshold; if the second word does not exist in the vocabulary database and the number of words in the vocabulary database does not reach the second threshold, determine that the verification of the second word has passed; if the second word exists in the vocabulary database, or if the number of words in the vocabulary database reaches the second threshold, determine that the verification of the second word has failed.

[0044] In another possible implementation, the verification module is further configured to verify the first word based on the words in the vocabulary database;

[0045] The adding module is further configured to add the first word to the word database if the first word passes the verification.

[0046] In another possible implementation, the feature extraction module is further configured to extract features from the second word using a large language model to obtain the features of the second word;

[0047] The determining module is further configured to determine the similarity between the second word and the third word based on the features of the second word and the features of the third word, wherein the third word is the first word in the text that is adjacent to the second word;

[0048] The determining module is further configured to determine the co-occurrence frequency of the second word and the third word, wherein the co-occurrence frequency indicates the frequency at which the second word and the third word appear in the text database in an adjacent manner;

[0049] The fusion module is further configured to fuse the similarity and co-occurrence frequency of the second word and the third word to obtain the combination probability of the second word and the third word, wherein the combination probability of the second word and the third word indicates the possibility that the second word and the third word can form a word.

[0050] The combination module is further configured to combine the second word and the third word into a fourth word when the combination probability of the second word and the third word reaches the first threshold.

[0051] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to perform the operations performed by the word segmentation method as described above.

[0052] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored therein, the at least one computer program being loaded and executed by a processor to perform the operations performed by the word segmentation method as described above.

[0053] In another aspect, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the operations performed by the word segmentation method described above.

[0054] In the solution provided in this application embodiment, when multiple first words are obtained by preliminary word segmentation of the text, the similarity and co-occurrence frequency of each pair of adjacent first words in the text can be determined, and the combination probability of each pair of adjacent first words can be determined. In order to determine the word groups that can form new words based on the combination probability, the two first words in the word group are combined to form a second word. The newly obtained second word has entity meaning. This method captures the contextual relationship between characters in the text to obtain more accurate words, ensures the accuracy of the words obtained by word segmentation, and thus ensures the accuracy of word segmentation. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of the structure of an implementation environment provided in an embodiment of this application;

[0057] Figure 2This is a flowchart of a word segmentation method provided in an embodiment of this application;

[0058] Figure 3 This is a flowchart of another word segmentation method provided in the embodiments of this application;

[0059] Figure 4 This is a schematic diagram of the structure of a word segmentation device provided in an embodiment of this application;

[0060] Figure 5 This is a schematic diagram of another word segmentation device provided in an embodiment of this application;

[0061] Figure 6 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0062] Figure 7 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0064] The terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, without departing from the scope of this application, the term “first” may be referred to as “second,” and similarly, the term “second” may be referred to as “first.”

[0065] As used in this application, the terms "at least one," "multiple," "each," and "any" have the following meanings: at least one includes one, two, or more; multiple includes two or more; each refers to each of the corresponding multiple; and any refers to any one of the multiple. For example, multiple words may include three words, where each refers to each of the three words, and any refers to any one of the three words, which could be the first word, the second word, or the third word.

[0066] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the texts or words involved in this application were obtained with full authorization.

[0067] The word segmentation method provided in the embodiments of the present application can be executed by a computer device. Optionally, the computer device is a terminal or a server. Optionally, the terminal is any type of terminal such as a smart mobile phone, a desktop computer, a smart wearable device, or an image acquisition device. Optionally, the server is one server, a server cluster composed of several servers, or a cloud computing service center.

[0068] In some embodiments, the computer device is provided as a server. Figure 1 is a schematic diagram of an implementation environment provided in the embodiments of the present application. See Figure 1 , the implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network.

[0069] The terminal 101 is configured to provide a text to the server 102, and the server 102 is configured to receive the text provided by the terminal 101, and perform word segmentation on the text according to the method provided in the embodiments of the present application, so as to obtain words contained in the text.

[0070] Optionally, the terminal 101 is installed with an application served by the server 102, the application has a word segmentation function, and the application is any application, for example, the application is a text processing application. The terminal 101 sends the text to the server 102 through the text processing application, so that the server 102 performs word segmentation on the text.

[0071] Figure 2 is a flowchart of a word segmentation method provided in the embodiments of the present application, and the method is executed by a computer device, as Figure 2 shown, the method includes:

[0072] 201. The computer device performs word segmentation on the text to obtain a plurality of first words, where each first word includes at least one character.

[0073] In the embodiments of the present application, the text is any text, and a plurality of first words are obtained by performing word segmentation on the text, where each first word includes at least one character. Considering that among the plurality of first words obtained by segmenting the text, there may be two adjacent first words that can form a new word, and the newly formed word can better reflect the meaning in the text. For example, among the plurality of first words obtained by segmenting the text, a first word is "I", and the first word adjacent to "I" is "们", these two first words can form "we", and "we" can more accurately reflect the meaning in the text. Therefore, when a plurality of first words are obtained, two adjacent first words that can form a new word will also be combined into a new word, so as to ensure the accuracy of word segmentation.

[0074] In this text, multiple first words constitute the content, and the number of characters contained in these multiple first words is equal to the number of characters in the text. Different first words may contain the same or different numbers of characters. For example, if the text is "I want to go home", and the first words are segmented into four words, they would be "I", "want", "go home", and "home"; or, if the first words are segmented into three words, they would be "I", "want", and "go home".

[0075] 202. The computer device extracts features from each first word using a large language model to obtain the features of each first word.

[0076] In this embodiment, the large language model is a deep learning model trained using a large amount of text data. The large language model is used to obtain the features of words. Through the large language model, features are extracted from each first word, resulting in the features of each first word. The features of each first word are used to represent the corresponding word, and the features of the first word can be represented in any form; for example, the features of the first word can be represented in the form of a feature vector.

[0077] 203. The computer device determines the similarity of multiple word groups based on the features of multiple first words. Each word group includes two adjacent first words in the text. The similarity of the word group indicates the similarity between the two first words in the word group.

[0078] In the embodiments of this application, when multiple first words are obtained, multiple word groups can be formed by the multiple first words in the order of the multiple first words in the text. Each word group contains two adjacent first words in the text. The similarity between each pair of adjacent first words in the text is determined, taking into account the contextual relationship between characters in the text, and the correlation between each pair of adjacent first words in the text is determined, reflecting whether each pair of adjacent first words can form a new word.

[0079] The number of multiple first words is one greater than the number of multiple word groups. For example, if the number of multiple first words is 5, the first first word and the second first word form a word group, the second first word and the third first word form a word group, the third first word and the fourth first word form a word group, and the fourth first word and the fifth first word form a word group, resulting in a total of 4 word groups.

[0080] 204. The computer device determines the co-occurrence frequency of each phrase. The co-occurrence frequency of a phrase indicates the frequency at which two first words in the phrase appear in an adjacent manner in a text database, which includes multiple reference texts.

[0081] In this embodiment, the text database contains multiple reference texts, which is equivalent to a corpus. For each phrase, the frequency of two first words appearing in the text database in an adjacent manner can be determined through the text database. This frequency reflects the probability that the two first words in the phrase can form a new word; the higher the frequency, the higher the probability that the two first words in the phrase can form a new word.

[0082] 205. The computer device fuses the similarity and co-occurrence frequency of each word group to obtain the combination probability of each word group. The combination probability of a word group indicates the probability that two first words in the word group can form a word.

[0083] In the embodiments of this application, the similarity and co-occurrence frequency of word groups can both reflect the possibility that two first words in a word group can form a new word. Therefore, the similarity and co-occurrence frequency of the same word group are fused to obtain the combination probability of the word group, so as to ensure that the combination probability can accurately reflect the possibility that two first words in a word group can form a word.

[0084] 206. The computer device, based on the combination probability of multiple word groups, combines two first words from the word group with the highest combination probability to form a second word, or combines two first words from the word group with a combination probability greater than a first threshold to form a second word.

[0085] In this embodiment of the application, for multiple word groups, the greater the combination probability of any word group, the greater the possibility that the two first words in the word group can form a new word. Therefore, the two first words in the word group with the highest combination probability in the text are combined to form a new word, that is, to form a second word, so as to ensure that the new word obtained by combination has entity meaning and ensure the accuracy of the new word.

[0086] In this embodiment of the application, the first threshold is any value. If the combination probability of any word group is greater than the first threshold, it means that the two first words in the word group can form a new word and the new word has an entity meaning. Therefore, the two first words in the word group with a combination probability greater than the first threshold are combined to form a new word to ensure the accuracy of the new word.

[0087] For example, the text is "I want to go home". The word segmentation results in 5 first words, 4 of which are "I", "want", "go home" and "home". Through the solution provided in the embodiments of this application, it can be determined that "go home" and "home" can form a new word "go home". The final word segmentation result of the text is "I", "want", "go home", which means the word segmentation is more accurate.

[0088] In the solution provided in this application embodiment, when multiple first words are obtained by preliminary word segmentation of the text, the similarity and co-occurrence frequency of each pair of adjacent first words in the text can be determined, and the combination probability of each pair of adjacent first words can be determined. In order to determine the word groups that can form new words based on the combination probability, the two first words in the word group are combined to form a second word. The newly obtained second word has entity meaning. This method captures the contextual relationship between characters in the text to obtain more accurate words, ensures the accuracy of the words obtained by word segmentation, and thus ensures the accuracy of word segmentation.

[0089] In the above Figure 2 Based on the embodiments shown, the embodiments of this application can determine the co-occurrence frequency of two first words based on the number of times two first words appear in the text database and the number of words in the text database. Furthermore, the words obtained from word segmentation can be stored in the vocabulary database to enrich the vocabulary stored in the vocabulary database. For details of the process, please refer to the embodiments below.

[0090] Figure 3 This is a flowchart of a word segmentation method provided in an embodiment of this application. The method is executed by a computer device, such as... Figure 3 As shown, the method includes:

[0091] 301. A computer device segments text to obtain multiple first words, each of which contains at least one character.

[0092] In one possible implementation, step 301 includes: segmenting the text using a word segmentation model to obtain multiple first words.

[0093] The word segmentation model can be any network model capable of segmenting text to obtain the vocabulary contained in the text. In this embodiment, the text is segmented using a word segmentation model to ensure that the resulting multiple first words are as accurate as possible.

[0094] 302. The computer device extracts features from each first word using a large language model to obtain the features of each first word.

[0095] 303. The computer device determines the similarity of multiple word groups based on the features of multiple first words. Each word group includes two adjacent first words in the text. The similarity of the word group indicates the similarity between the two first words in the word group.

[0096] In one possible implementation, the similarity of each phrase can be determined using the following function:

[0097]

[0098] Wherein V1 and V2 are used to represent the features of two first vocabularies in any word group, V1 and V2 are represented in the form of vectors, sim(V1, V2) is used to represent the similarity of the word group, θ is used to represent the angle between V1 and V2, V1·V2 is used to represent the dot product of V1 and V2, ||V1|| is used to represent the Euclidean length of V1, and ||V2|| is used to represent the Euclidean length of V2.

[0099] In the embodiments of the present application, cosine similarity is adopted to determine the similarity between the two first vocabularies in each word group, so as to ensure the accuracy of the determined similarity.

[0100] 304. The computer device traverses a plurality of reference texts in a text database based on the two first vocabularies in any word group, and determines the number of times the two first vocabularies in the word group appear in an adjacent form.

[0101] In the embodiments of the present application, for any word group, a plurality of reference texts are traversed according to the order of the two first vocabularies in the word group in the text, so as to determine the number of times the two first vocabularies appear in an adjacent form. This number can reflect the degree of association between the two first vocabularies in the word group, so as to reflect the possibility that the two first vocabularies can form a single vocabulary.

[0102] In a possible implementation manner, step 304 includes: composing the two first vocabularies in the word group into a candidate vocabulary according to the order of the two first vocabularies in the word group in the text, traversing the plurality of reference texts based on the candidate vocabulary, and determining the occurrence count of the candidate vocabulary in the word group, where the occurrence count of the candidate vocabulary is exactly the number of times the two first vocabularies in the word group appear in an adjacent form.

[0103] In the embodiments of the present application, the manner of pre-combining the two first vocabularies in the word group into a candidate vocabulary is adopted to traverse the plurality of reference texts, so that the occurrence count of the candidate vocabulary can be determined according to the order of the two first vocabularies in the word group in the text, ensuring that the determined count can reflect the possibility that the two first vocabularies in the word group can form a single vocabulary, and ensuring the accuracy of the determined count.

[0104] For example, if the two first vocabularies in any word group are respectively "work" (the Chinese characters "gōng" and "zuò"), which form the vocabulary "work" in the order of the two first vocabularies in the text, the occurrence count of the vocabulary "work" in the plurality of reference texts is determined by traversing the plurality of reference texts based on the vocabulary "work".

[0105] 305. The computer device determines the number of vocabularies contained in the plurality of reference texts.

[0106] In the embodiments of the present application, the number of vocabularies indicates the total number of vocabularies contained in the plurality of reference texts.

[0107] In one possible implementation, step 305 includes: segmenting each reference text to obtain the vocabulary included in each reference text, determining the number of words included in each reference text, and determining the number of words included in multiple reference texts.

[0108] 306. The computer device determines the co-occurrence frequency of a phrase as the ratio of the number of occurrences to the number of words. The co-occurrence frequency of a phrase indicates the frequency at which two first words in a phrase appear in an adjacent form in a text database, which includes multiple reference texts.

[0109] This application embodiment is illustrated by taking the confirmation of the co-occurrence frequency of any word group as an example. In another embodiment, the co-occurrence frequency of each word group can be determined according to the above steps 304-306.

[0110] In one possible implementation, the co-occurrence frequency of phrases can be determined using the following function:

[0111] freq(token1, token2)=count(token1, token2) / N

[0112] In this context, token1 and token2 represent two first words in the same phrase, freq(token1, token2) represents the co-occurrence frequency of first words token1 and first words token2, count(token1, token2) represents the number of times first words token1 and first words token2 appear in an adjacent form, and N represents the number of words contained in multiple reference texts in the text database.

[0113] It should be noted that this application determines the co-occurrence frequency of two first words based on the number of times they appear in the text database and the number of words in the text database. In another embodiment, it is not necessary to perform the above steps 304-306, but other methods are used to determine the co-occurrence frequency of each word group.

[0114] 307. The computer device fuses the similarity and co-occurrence frequency of each word group to obtain the combination probability of each word group. The combination probability of a word group indicates the probability that two first words in the word group can form a word.

[0115] In one possible implementation, the following function is used to determine the combination probability of any phrase:

[0116] score(token1, token2)=α*sim(V1, V2)+β*freq(token1, token2)

[0117] In this context, token1 and token2 represent two first words in the same word group, score(token1, token2) represents the combination probability of first word token1 and first word token2, V1 represents the feature of first word token1, V2 represents the feature of first word token2, sim(V1, V2) represents the similarity between first word token1 and second word token2, freq(token1, token2) represents the co-occurrence frequency of first word token1 and second word token2, α is used as the weight of similarity, β is used as the weight of co-occurrence frequency, and α and β are used to adjust the importance of similarity and co-occurrence frequency in the combination probability.

[0118] 308. The computer device, based on the probabilities of multiple word groups, combines two first words from the word group with the highest probability to form a second word, or combines two first words from the word group with a probability greater than a first threshold to form a second word.

[0119] 309. The computer device verifies the second word based on the words in the vocabulary database, which is used to store words with entity meanings.

[0120] In this embodiment of the application, the vocabulary database is used to store words with entity meanings. Based on the words in the vocabulary database, the second word is verified so that if the verification of the second word is passed, the second word is stored in the vocabulary database to ensure the accuracy of the words stored in the vocabulary database.

[0121] In one possible implementation, step 309 includes: detecting whether a second word exists in the vocabulary database; detecting whether the number of words in the vocabulary database reaches a second threshold; determining that the verification of the second word has passed if the second word does not exist in the vocabulary database and the number of words in the vocabulary database has not reached the second threshold; and determining that the verification of the second word has failed if the second word exists in the vocabulary database or if the number of words in the vocabulary database reaches the second threshold.

[0122] The second threshold can be any value, for example, the second threshold is 30000.

[0123] In this embodiment of the application, it is detected whether a second word exists in the vocabulary database and whether the number of words in the vocabulary database reaches a second threshold, so as to ensure that the words stored in the subsequent vocabulary database are not repeated and the number does not exceed the second threshold, so as to ensure the accuracy of the words stored in the subsequent vocabulary database.

[0124] 310. If the computer device verifies the second word, it adds the second word to the word database.

[0125] In this embodiment of the application, when the second word is obtained, it is verified based on the words in the word database. Only when the second word passes the verification is the second word added to the word database, so as to ensure that the words stored in the word database are not duplicated and the number does not exceed the second threshold, thereby ensuring the accuracy of the words stored in the word database.

[0126] In one possible implementation, the method further includes: validating a first word based on words in a vocabulary database; and adding the first word to the vocabulary database if the first word passes the validation.

[0127] In this embodiment of the application, the vocabulary database is a database constructed to store words with entity meanings. When multiple first words are obtained by word segmentation of the text, second words can also be determined. The multiple first words are also words with entity meanings, and the second words are words with entity meanings in the text. Therefore, the first words are verified. If the verification of the first words passes, the first words are added to the vocabulary database to enrich the vocabulary contained in the vocabulary database and ensure the accuracy of the words in the vocabulary database.

[0128] It should be noted that the process of verifying the first word is the same as the process of verifying the second word, and will not be repeated here.

[0129] It should be noted that the embodiments of this application only enrich the vocabulary in the vocabulary database with a single text. In another embodiment, other texts can be obtained and the vocabulary in the vocabulary database can be enriched according to the steps 301-310 described above.

[0130] For example, given the text "I want to go to work", the word segmentation yields multiple first words: "I", "want", "want", "go", "work", and "work". A large language model is used to extract features from each first word, obtaining its characteristics. The similarity and co-occurrence frequency of each pair of adjacent first words are calculated to determine their combination probability. Based on these probabilities, it is determined that the first words "work" and "work" can form a new word, i.e., the combined second word is "work". If both the first and second words pass verification against the words in the vocabulary database, "I", "want", "want", "go", "work", and "work" are added to the vocabulary database to enrich its content. This process continues with the next text to extract words and add them to the database, until the number of words in the database reaches a second threshold, or until the words in the database can no longer be combined to form new words.

[0131] In this embodiment, the vocabulary database is used to train a large language model. The vocabulary in the database can be enriched in the manner described above; for example, the vocabulary database includes words meeting a second threshold. Once the vocabulary database is obtained, it can be used to train large language models in various scenarios, such as translation and text classification.

[0132] For example, in a translation scenario, the vocabulary database serves as the vocabulary of a large language model, which in turn acts as the translation model. This allows the model to extract features from the input text and, by combining the vocabulary and features, translate the input text word by word into a translated text.

[0133] In the solution provided in this application embodiment, when multiple first words are obtained by preliminary word segmentation of the text, the similarity and co-occurrence frequency of each pair of adjacent first words in the text can be determined, and the combination probability of each pair of adjacent first words can be determined. In order to determine the word groups that can form new words based on the combination probability, the two first words in the word group are combined to form a second word. The newly obtained second word has entity meaning. This method captures the contextual relationship between characters in the text to obtain more accurate words, ensures the accuracy of the words obtained by word segmentation, and thus ensures the accuracy of word segmentation.

[0134] Furthermore, the embodiments of this application employ Context Sensitivity BytePair Encoder (CS-BPE), which can capture the contextual relationships between characters in the text to obtain more accurate vocabulary, thereby enriching the vocabulary database. This allows the vocabulary database to be used for training large language models, thus improving the performance of the large language model.

[0135] It should be noted that the embodiment of this application is illustrated by directly storing the second word in the vocabulary database. In another embodiment, when the second word is obtained, it is also determined whether the second word in the text and the adjacent first word can form a new word. That is, the method also includes the following steps 1-5.

[0136] Step 1: Extract features from the second word using a large language model to obtain the features of the second word.

[0137] Step 2: Based on the features of the second word and the third word, determine the similarity between the second word and the third word. The third word is the first word in the text that is adjacent to the second word.

[0138] Step 3: Determine the co-occurrence frequency of the second and third words. The co-occurrence frequency indicates the frequency at which the second and third words appear in adjacent positions in the text database.

[0139] Step 4: Combine the similarity and co-occurrence frequency of the second and third words to obtain the combination probability of the second and third words. The combination probability of the second and third words indicates the possibility that the second and third words can form a word.

[0140] Step 5: If the probabilities of the second and third words reach the first threshold, combine the second and third words to form the fourth word.

[0141] Steps 1-5 are the same as steps 302-308 above, and will not be repeated here.

[0142] It should be noted that the embodiments of this application determine whether the second word and the adjacent first word can form a new word. In another embodiment, if the second word and the adjacent first word in the text have already formed other second words, then it is still possible to determine whether the two adjacent second words can form a new word in the same way as described above.

[0143] For example, if the text is "Life Square", the first words obtained from text segmentation are "life", "living", "square", and "field". The second words are "life" and "square". If the two second words are adjacent in the text, they can form a new word "Life Square".

[0144] It should be noted that the embodiments of this application only combine the second word and the third word to form a new word. In another embodiment, it is also determined whether the fourth word and the adjacent first word can form a new word. The above process is repeated until the currently obtained word cannot form a new word with the adjacent first word.

[0145] The solution provided in this application can identify low-frequency words or words that have not appeared in the vocabulary database. In the process of word segmentation of text, it not only considers the co-occurrence frequency of two adjacent words in the text, but also the semantics of the text, which can ensure the accuracy of the identified words. When the vocabulary database is applied to model training, it can help the model better understand and process complex language phenomena, and is more adaptable to Chinese language model pre-training, such as polysemy or polysemy.

[0146] It should be noted that the above-mentioned various optional embodiments can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0147] Figure 4 This application provides a word segmentation device, such as... Figure 4 As shown, the device includes:

[0148] The word segmentation module 401 is used to segment the text into words to obtain multiple first words, each of which contains at least one character;

[0149] The feature extraction module 402 is used to extract features from each first word using a large language model to obtain the features of each first word.

[0150] The determination module 403 is used to determine the similarity of multiple word groups based on the features of multiple first words. Each word group includes two adjacent first words in the text, and the similarity of the word group indicates the similarity between the two first words in the word group.

[0151] The determination module 403 is also used to determine the co-occurrence frequency of each phrase, the co-occurrence frequency of which indicates the frequency at which two first words in the phrase appear in an adjacent form in the text database, which includes multiple reference texts;

[0152] The fusion module 404 is used to fuse the similarity and co-occurrence frequency of each word group to obtain the combination probability of each word group. The combination probability of a word group indicates the probability that two first words in the word group can form a word.

[0153] The combination module 405 is used to combine two first words from the word group with the highest combination probability to form a second word, or to combine two first words from the word group with a combination probability greater than a first threshold to form a second word.

[0154] In one possible implementation, the determining module 403 is used to traverse multiple reference texts based on two first words in the phrase, determine the number of times the two first words in the phrase appear in an adjacent form; determine the number of words contained in the multiple reference texts; and determine the ratio of the number of occurrences to the number of words as the co-occurrence frequency of the phrase.

[0155] In another possible implementation, such as Figure 5 As shown, the device also includes:

[0156] The verification module 406 is used to verify the second word based on the words in the vocabulary database, which stores words with entity meanings.

[0157] Add module 407 to add the second word to the vocabulary database if the second word passes the validation.

[0158] In another possible implementation, the verification module 406 is used to detect whether a second word exists in the vocabulary database; detect whether the number of words in the vocabulary database reaches a second threshold; if the second word does not exist in the vocabulary database and the number of words in the vocabulary database does not reach the second threshold, determine that the verification of the second word has passed; if the second word exists in the vocabulary database, or if the number of words in the vocabulary database reaches the second threshold, determine that the verification of the second word has failed.

[0159] In another possible implementation, the verification module 406 is also used to verify the first word based on the words in the vocabulary database;

[0160] Module 407 is also used to add the first word to the vocabulary database if the first word passes the validation.

[0161] In another possible implementation, the feature extraction module 402 is also used to extract features from the second vocabulary through a large language model to obtain the features of the second vocabulary;

[0162] The determination module 403 is also used to determine the similarity between the second word and the third word based on the features of the second word and the features of the third word, wherein the third word is the first word in the text that is adjacent to the second word;

[0163] The determination module 403 is also used to determine the co-occurrence frequency of the second word and the third word, the co-occurrence frequency indicating the frequency at which the second word and the third word appear in the text database in an adjacent form;

[0164] The fusion module 404 is also used to fuse the similarity and co-occurrence frequency of the second word and the third word to obtain the combination probability of the second word and the third word. The combination probability of the second word and the third word indicates the possibility that the second word and the third word can form a word.

[0165] The combination module 405 is also used to combine the second word and the third word into a fourth word when the combination probability of the second word and the third word reaches a first threshold.

[0166] It should be noted that the word segmentation device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the word segmentation device and the word segmentation method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0167] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one computer program, which is loaded and executed by the processor to perform the operations performed by the word segmentation method of the above embodiments.

[0168] Optionally, the computer device is provided as a terminal. Figure 6 This diagram illustrates a structural block diagram of a terminal 600 provided in an exemplary embodiment of this application. The terminal 600 includes a processor 601 and a memory 602.

[0169] Processor 601 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 601 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 601 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 601 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0170] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 602 are used to store at least one computer program, which is executed by the processor 601 to implement the word segmentation method provided in the method embodiments of this application.

[0171] In some embodiments, the terminal 600 may optionally include a peripheral device interface 603 and at least one peripheral device. The processor 601, memory 602, and peripheral device interface 603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 604, a display screen 605, a camera assembly 606, an audio circuit 607, and a power supply 608.

[0172] Peripheral interface 603 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 601 and memory 602. In some embodiments, processor 601, memory 602 and peripheral interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 601, memory 602 and peripheral interface 603 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0173] The radio frequency (RF) circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 604 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 604 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 604 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0174] Display screen 605 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 605 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 601 for processing. In this case, display screen 605 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 605, disposed on the front panel of terminal 600; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 600 or in a folded design; in other embodiments, display screen 605 may be a flexible display screen, disposed on a curved or folded surface of terminal 600. Furthermore, display screen 605 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 605 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0175] The camera assembly 606 is used to acquire images or videos. Optionally, the camera assembly 606 includes a front-facing camera and a rear-facing camera. The front-facing camera is disposed on the front panel of the terminal, and the rear-facing camera is disposed on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash may be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0176] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 601 for processing, or input to the radio frequency circuit 604 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal 600. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 607 may also include a headphone jack.

[0177] Power supply 608 is used to power the various components in terminal 600. Power supply 608 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 608 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0178] Those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on terminal 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0179] Optionally, the computer equipment is provided as a server. Figure 7 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 700 can vary significantly due to differences in configuration or performance. It may include one or more Central Processing Units (CPUs) 701 and one or more memories 702. The memories 702 store at least one computer program, which is loaded and executed by the processor 701 to implement the methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0180] This application also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to perform the operations performed by the word segmentation method of the above embodiments.

[0181] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the operations performed by the word segmentation method of the above embodiments.

[0182] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0183] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the protection scope of the present application.

Claims

1. A word segmentation method, characterized in that, The method includes: The text is segmented to obtain multiple first words, each of which contains at least one character; By using a large language model, features are extracted from each of the first words to obtain the features of each of the first words; Based on the features of the multiple first words, the similarity of multiple word groups is determined. Each word group includes two adjacent first words in the text, and the similarity of the word group indicates the similarity between the two first words in the word group. For any word group, based on the two first words in the word group, traverse multiple reference texts in the text database to determine the number of times the two first words in the word group appear in an adjacent form; determine the number of words contained in the multiple reference texts; determine the ratio of the number of occurrences to the number of words as the co-occurrence frequency of the word group, and the co-occurrence frequency of the word group indicates the frequency of the two first words in the word group appearing in an adjacent form in the text database; The similarity and co-occurrence frequency of each word group are fused to obtain the combination probability of each word group. The combination probability of the word group indicates the probability that two first words in the word group can form a word. Based on the combination probability of the multiple word groups, the two first words in the word group with the highest combination probability are combined into the second word, or the two first words in the word group with a combination probability greater than the first threshold are combined into the second word. For any word group, the following function is used to determine the combination probability of the word group: in, and Used to indicate the first word in the same phrase. Used to represent and The combination probability, Used to represent Features Used to represent Features Used to represent and similarity, Used to represent and co-occurrence frequency, Weights used to represent similarity Weights used to represent co-occurrence frequencies.

2. The method according to claim 1, characterized in that, After combining the two first words from the word group with the highest combination probability based on the combination probability of the multiple word groups to form the second word, the method further includes: The second word is validated based on the words in the vocabulary database, which is used to store words with entity meanings. If the second word passes the verification, the second word will be added to the word database.

3. The method according to claim 2, characterized in that, The validation of the second word based on the vocabulary database includes: Detect whether the second word exists in the vocabulary database; Detect whether the number of words in the vocabulary database has reached the second threshold; If the second word does not exist in the vocabulary database and the number of words in the vocabulary database does not reach the second threshold, it is determined that the verification of the second word has passed; If the second word exists in the vocabulary database, or if the number of words in the vocabulary database reaches the second threshold, it is determined that the verification of the second word has failed.

4. The method according to claim 2, characterized in that, After combining the two first words from the word group with the highest combination probability based on the combination probability of the multiple word groups to form the second word, the method further includes: The first word is validated based on the words in the vocabulary database. If the first word passes the verification, the first word is added to the word database.

5. The method according to claim 1, characterized in that, After combining the two first words from the word group with the highest combination probability based on the combination probability of the multiple word groups to form the second word, the method further includes: By using a large language model, features of the second word are extracted to obtain the features of the second word; Based on the features of the second word and the features of the third word, the similarity between the second word and the third word is determined, wherein the third word is the first word in the text that is adjacent to the second word; Determine the co-occurrence frequency of the second word and the third word, wherein the co-occurrence frequency indicates the frequency at which the second word and the third word appear in adjacent forms in the text database; The similarity and co-occurrence frequency of the second word and the third word are fused to obtain the combination probability of the second word and the third word. The combination probability of the second word and the third word indicates the possibility that the second word and the third word can form a word. If the probability of combining the second word and the third word reaches the first threshold, the second word and the third word are combined to form the fourth word.

6. A word segmentation device, characterized in that, The device includes: The word segmentation module is used to segment the text into words, resulting in multiple first words, each of which contains at least one character; The feature extraction module is used to extract features from each of the first words using a large language model to obtain the features of each of the first words. A determination module is used to determine the similarity of multiple word groups based on the features of the multiple first words, each word group including two adjacent first words in the text, and the similarity of the word group indicates the similarity between the two first words in the word group; The determining module is further configured to, for any word group, based on the two first words in the word group, traverse multiple reference texts in the text database to determine the number of times the two first words in the word group appear in an adjacent form; determine the number of words contained in the multiple reference texts; and determine the ratio of the number of occurrences to the number of words as the co-occurrence frequency of the word group, wherein the co-occurrence frequency of the word group indicates the frequency at which the two first words in the word group appear in an adjacent form in the text database; The fusion module is used to fuse the similarity and co-occurrence frequency of each word group to obtain the combination probability of each word group. The combination probability of the word group indicates the probability that two first words in the word group can form a word. The combination module is used to combine two first words from the word group with the highest combination probability to form a second word, or to combine two first words from the word group with a combination probability greater than a first threshold to form a second word. For any word group, the fusion module uses the following function to determine the combination probability of the word group: , and Used to indicate the first word in the same phrase. Used to represent and The combination probability, Used to represent Features Used to represent Features Used to represent and similarity, Used to represent and co-occurrence frequency, Weights used to represent similarity Weights used to represent co-occurrence frequencies.

7. The apparatus according to claim 6, characterized in that, The determining module is used to determine the number of times the two first words in the phrase appear in adjacent forms by traversing the multiple reference texts based on the two first words in the phrase; Determine the number of words contained in the plurality of reference texts; determine the ratio of the frequency to the number of words as the co-occurrence frequency of the word group.

8. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to perform the operations performed by the word segmentation method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to perform the operations of the word segmentation method as described in any one of claims 1 to 5.

Images