Word recognition method and apparatus, and word recognition model training method and apparatus

Abstract

A word recognition method and apparatus, a word recognition model training method and apparatus, a computer device, a storage medium and a computer program product. The word recognition method comprises: on the basis of the number of times that a target character in a target word segment appears in a first language material set to which the target word segment belongs, and the number of times that a character segment, which contains the target character and is included in the target word segment, appears in the first language material set, obtaining a word internal association degree corresponding to the target word segment; on the basis of the number of times that the target word segment appears in the first language material set and the number of times that a sub-word in the target word segment appears in the first language material set, obtaining a word semantic independence degree corresponding to the target word segment, wherein the sub-word is a sub-segment which is segmented from the target word segment; on the basis of the word internal association degree and the word semantic independence degree which correspond to the target word segment, determining a word segment vector corresponding to the target word segment; and on the basis of the word segment vector, obtaining a word recognition result corresponding to the target word segment.

Description

Word recognition method, word recognition model training method and device

[0001] This application claims priority to Chinese patent application number 202310699819.0, filed on June 13, 2023, entitled “New Word Recognition Method, New Word Recognition Model Training Method and Device”, the contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a word recognition method, a word recognition model training method, an apparatus, a computer device, a storage medium, and a computer program product. Background Art

[0003] With the development of computer technology, natural language processing (NLP) has emerged. NLP is a discipline that integrates linguistics, computer science, and mathematics. Word discovery is a key component of NLP, which involves analyzing large amounts of corpus to discover words.

[0004] Traditional word recognition methods are typically rule-based, where linguists construct specific templates based on morphological principles to mine words. However, using specific templates to mine words has limitations and cannot guarantee accurate word recognition.

[0005] Summary of the Invention

[0006] Based on this, it is necessary to provide a word recognition method, word recognition model training method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of word recognition in response to the above technical problems.

[0007] This application provides a word recognition method. The method includes:

[0008] Obtaining a target word segment and a first corpus to which the target word segment belongs;

[0009] Obtaining an internal word association corresponding to the target word segment based on the number of occurrences of a target character in the target word segment in the first corpus and the number of occurrences of a character segment containing the target character in the target word segment in the first corpus;

[0010] Obtaining word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of subwords in the target word segment in the first corpus; the subwords are subsegments segmented from the target word segment;

[0011] Determining a word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment;

[0012] The target word segment is subjected to word recognition based on the word segment vector to obtain a word recognition result corresponding to the target word segment.

[0013] This application also provides a word recognition device. The device includes:

[0014] An acquisition module, configured to acquire a target word segment and a first corpus to which the target word segment belongs;

[0015] a determination module configured to obtain an internal word association corresponding to the target word segment based on the number of occurrences of a target character in the target word segment in the first corpus and the number of occurrences of a character segment containing the target character in the target word segment in the first corpus; obtain a word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of a subword in the target word segment in the first corpus; the subword being a subsegment segmented from the target word segment; and determine a word segment vector corresponding to the target word segment based at least on the internal word association and the word semantic independence corresponding to the target word segment;

[0016] The recognition module is used to perform word recognition on the target word segment based on the word segment vector to obtain a word recognition result corresponding to the target word segment.

[0017] The present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps described in the above-mentioned word recognition method when executing the computer program.

[0018] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned word recognition method are implemented.

[0019] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned word recognition method when executed by a processor.

[0020] This application provides a method for training a word recognition model. The method includes:

[0021] Obtaining a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs;

[0022] Obtaining an internal word association corresponding to the training word segment based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus;

[0023] Obtaining word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of subwords in the training word segment in the second corpus; the subwords are subsegments segmented from the training word segment;

[0024] Determining a word segment vector corresponding to the training word segment based at least on the word internal association and word semantic independence corresponding to the training word segment;

[0025] Based on the word segment vectors and training word labels corresponding to the training word segments, the initial word recognition model is trained to obtain a target word recognition model.

[0026] This application also provides a word recognition model training device. The device includes:

[0027] A data acquisition module, configured to acquire a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs;

[0028] a vector determination module for obtaining a word internal association corresponding to the training word segment based on the number of occurrences of a target character in the training word segment in the second corpus and the number of occurrences of a character segment containing the target character in the training word segment in the second corpus; obtaining a word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of a subword in the training word segment in the second corpus; the subword being a subsegment segmented from the training word segment; and determining a word segment vector corresponding to the training word segment based at least on the word internal association and the word semantic independence corresponding to the training word segment;

[0029] The model training module is used to train the initial word recognition model based on the word segment vectors and training word labels corresponding to the training word segments to obtain a target word recognition model.

[0030] The present application also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps described in the above-mentioned word recognition model training method when executing the computer program.

[0031] The present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps described in the above-mentioned word recognition model training method when executed by a processor.

[0032] The present application also provides a computer program product, including a computer program, which implements the steps described in the above-mentioned word recognition model training method when executed by a processor.

[0033] The above-mentioned word recognition method, word recognition model training method, apparatus, computer device, storage medium, and computer program product obtain the word internal association degree corresponding to the word segment by counting relevant information about the characters in the word segment. The word internal association degree can reflect the degree of association between the characters in the word segment. The greater the association degree, the more likely the word segment is a word. The word semantic independence degree corresponding to the word segment is obtained by counting relevant information about more fine-grained sub-segments within the word segment. The word semantic independence degree can reflect the semantic completeness of the word segment. The more complete the semantics, the more likely the word segment is a word. The word segment vector corresponding to the word segment is determined based on the word internal association degree and word semantic independence degree corresponding to the word segment. The word segment vector contains information such as the word internal association degree and word semantic independence degree that facilitate word recognition. Based on the word segment vector corresponding to the target word segment, whether the target word segment is a word is identified, which can effectively improve word recognition accuracy. Using the word segment vectors and training labels corresponding to the training word segments as training data to train the word recognition model, the word recognition model can learn the knowledge related to the words in the training data to recognize words. Using the trained word recognition model to perform word recognition can improve the accuracy and efficiency of word recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments described in this specification. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0035] FIG1 is a diagram illustrating an application environment of a word recognition method and a word recognition model training method according to an embodiment;

[0036] FIG2 is a schematic flow chart of a word recognition method according to an embodiment;

[0037] FIG3 is a flow chart of a word recognition interface in one embodiment;

[0038] FIG4 is a flow chart of a word recognition model training method according to an embodiment;

[0039] FIG5 is a schematic diagram of training a word recognition model in one embodiment;

[0040] FIG6 is a flow chart of a word recognition method according to another embodiment;

[0041] FIG7 is a schematic diagram of an application scenario of the method of the present application in one embodiment;

[0042] FIG8 is a block diagram of a word recognition device according to an embodiment;

[0043] FIG9 is a block diagram of a word recognition model training device according to an embodiment;

[0044] FIG10 is a diagram showing the internal structure of a computer device according to one embodiment;

[0045] FIG11 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0047] The word recognition method and word recognition model training method provided in the embodiments of the present application can be applied to the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other servers. Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers or a cloud server.

[0048] Both the terminal and the server can be used independently to execute the word recognition method and word recognition model training method provided in the embodiments of the present application.

[0049] For example, the terminal obtains a target word segment and the first corpus to which the target word segment belongs, obtains the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus, obtains the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of the subword in the target word segment in the first corpus, where the subword is a sub-segment cut out from the target word segment. The terminal determines the word segment vector corresponding to the target word segment based on at least the internal word association and the word semantic independence corresponding to the target word segment, performs word recognition on the target word segment based on the word segment vector corresponding to the target word segment, and obtains a word recognition result corresponding to the target word segment.

[0050] For example, the server obtains a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs. The server obtains the internal word association corresponding to the training word segment based on the number of occurrences of the target character in the training word segment and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus, and obtains the word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of subwords in the training word segment in the second corpus, where the subwords are subsegments cut out from the training word segment. The server determines the word segment vector corresponding to the training word segment based on at least the internal word association and word semantic independence corresponding to the training word segment, and trains the initial word recognition model based on the word segment vector and the training word label corresponding to the training word segment to obtain a target word recognition model.

[0051] The terminal and the server can also be used in collaboration to execute the word recognition method and word recognition model training method provided in the embodiments of the present application.

[0052] For example, the server obtains the target word segment and the first corpus to which the target word segment belongs from the terminal. The server obtains the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus, and obtains the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of the subword in the target word segment in the first corpus, where the subword is a sub-segment cut out from the target word segment. The server determines the word segment vector corresponding to the target word segment based on at least the internal word association and the word semantic independence corresponding to the target word segment, performs word recognition on the target word segment based on the word segment vector corresponding to the target word segment, and obtains the word recognition result corresponding to the target word segment. The server returns the word recognition result corresponding to the target word segment to the terminal.

[0053] For example, the server obtains a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs from the terminal. The server obtains the internal word association corresponding to the training word segment based on the difference between the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus, and obtains the word semantic independence corresponding to the training word segment based on the difference between the number of occurrences of the training word segment in the second corpus and the number of occurrences of the subword in the training word segment in the second corpus, where the subword is a sub-segment cut out from the target word segment. The server determines the word segment vector corresponding to the training word segment based on at least the internal word association and word semantic independence corresponding to the training word segment, and trains the initial word recognition model based on the word segment vector and the training word label corresponding to the training word segment to obtain a target word recognition model. Subsequently, the server can help the terminal perform word recognition based on the target word recognition model. The server can send the target word recognition model to the terminal so that the terminal performs word recognition based on the target word recognition model.

[0054] Standard lexicons refer to standard and commonly used dictionaries. With the development of the internet and the widespread use of various software applications, new words are constantly appearing online. With this constant flow of new words online, timely identification of new words from corpora is crucial. Traditional techniques typically mine words from corpora based on specific templates. However, these templates are limited in scope and have limitations, making accurate word recognition impossible.

[0055] The word recognition method provided in this application obtains a target word fragment and a first corpus to which the target word fragment belongs, calculates various statistical features corresponding to the target word fragment based on the first corpus, for example, the statistical features include word internal association, word semantic independence, and word text independence, determines a word fragment vector corresponding to the target word fragment based on the various statistical features corresponding to the target word fragment, and the word fragment vector contains various statistical features that are helpful for word recognition. Based on the word fragment vector corresponding to the target word fragment, it is identified whether the target word fragment is a word, which can effectively improve the accuracy of word recognition. The word recognition method provided in this application is applicable to word recognition in various scenarios.

[0056] In one embodiment, as shown in FIG2 , a word recognition method is provided. The method is applied to a computer device as an example. The computer device may be a terminal or a server. Referring to FIG2 , the word recognition method includes the following steps:

[0057] Step S202: Obtain a target word segment and a first corpus to which the target word segment belongs.

[0058] The target word segment refers to a word segment to be identified as a word. It is understood that the target word segment includes at least two characters. For example, the target word segment can be a segment including two characters, a segment including three characters, a segment including four characters, a segment including five characters, a segment including six characters, etc.

[0059] The corpus includes multiple corpora, that is, the corpus includes multiple texts. Based on the corpora in the corpus, statistical features corresponding to word segments in the corpus are calculated to identify whether the word segment is a term. The first corpus to which the target word segment belongs includes at least one corpus containing the target word segment. That is, the target word segment appears in at least one corpus in the first corpus to which the target word segment belongs.

[0060] Specifically, the computer device can obtain the target word fragment and the first corpus to which the target word fragment belongs locally or from other devices, calculate at least two statistical features corresponding to the target word fragment based on the corpus in the first corpus, and identify whether the target word fragment is a word based on the at least two statistical features corresponding to the target word fragment.

[0061] In one embodiment, a first corpus may be obtained first, and then the target word segment may be determined from the first corpus. For example, multiple corpora belonging to the same field may be obtained to form the first corpus, and each corpus in the first corpus may be segmented to obtain multiple candidate word segments, and the target word segment may be determined from each candidate word segment. It is understood that if the first corpus is obtained first, each candidate word segment obtained by segmenting each corpus in the first corpus may be used as a target word segment, and all words appearing in the first corpus may be identified, thereby mining a large number of words at one time.

[0062] In one embodiment, the target word fragment may be obtained first, and then the first corpus to which the target word fragment belongs may be obtained. For example, a word fragment of interest is obtained as the target word fragment, the corpus containing the target word fragment is collected, and the first corpus is composed based on the collected corpus. It can be understood that the first corpus may be composed of corpus containing the target word fragment, or may be composed of corpus containing the target word fragment and corpus not containing the target word fragment. In a specific embodiment, the user may input the word fragment of interest as the target word fragment on the relevant interface of the terminal, thereby triggering word recognition for the target word fragment and obtaining the word recognition result corresponding to the target word fragment. For example, referring to Figure 3, on the word recognition interface, the user inputs the word fragment of interest in the input box of the word fragment to be recognized, and the recognition control is triggered after the user completes the input. After the user triggers the recognition control, the terminal may determine the word recognition result locally, or may send a word recognition request carrying the target word fragment to the server, and the server may determine the word recognition result, and obtain the word recognition result from the server. Taking the case where the terminal determines the word recognition result locally as an example, the terminal automatically obtains the first corpus to which the target word segment belongs from the database, calculates at least two statistical features corresponding to the target word segment based on the corpus in the first corpus to identify whether the target word segment is a word, and obtains the word recognition result corresponding to the target word segment. Referring to Figure 3, after the terminal obtains the word recognition result, the word recognition result can be displayed on the interface so that the user can know the word recognition result in a timely manner. It can be understood that in addition to entering the word segment of interest as the target word segment, the user can also upload the first corpus to which the target word segment belongs. If the user triggers word recognition for the target word segment, the word recognition result corresponding to the target word segment input by the user is determined based on the first corpus uploaded by the user.

[0063] In one embodiment, each corpus in the first corpus to which the target word segment belongs belongs to the same field. For example, in the medical field, each corpus in the first corpus to which the target word segment belongs is text related to the medical field. In the financial field, each corpus in the first corpus to which the target word segment belongs is text related to the financial field. In this way, the fact that each corpus in the first corpus to which the target word segment belongs belongs to the same field facilitates the mining of words in specific fields and scenarios.

[0064] Step S204 : obtaining the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus.

[0065] The target character in the target word segment can be any character in the target word segment. For example, if the target word segment is a segment consisting of four characters, each of the four characters in the target word segment can be used as the target character, or at least one character can be selected from the four characters in the target word segment as the target character. It is understood that there can be at least one target character in the target word segment.

[0066] A character segment includes at least two characters. A character segment containing a target character in a target word segment is a character segment composed of the characters contained in the target word segment, and the character segment contains the target character. For example, if the target word segment is "labor fruit" and the target character is "labor," then the character segment containing the target character in the target word segment can be "labor." It is understood that for a target character in a target word segment, there can be at least one character segment containing the target character in the target word segment.

[0067] The intra-word association degree for a word fragment is calculated based on the number of occurrences of the target character in the word fragment within the corpus to which the word fragment belongs, as well as the number of occurrences of the character segment containing the target character within the word fragment within the corpus to which the word fragment belongs. The intra-word association degree for a word fragment reflects the degree of connection, closeness, and stability between the characters within the word fragment. The greater the intra-word association degree for a word fragment, the more stable the characters within the word fragment, and the greater the likelihood that the word fragment will become a word.

[0068] Specifically, the computer device determines a target character and a character segment containing the target character from the target word segment, counts the number of occurrences of the target character in the first corpus, counts the number of occurrences of the character segment containing the target character in the first corpus, and obtains the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus. For example, the number of occurrences of the character segment containing the target character in the target word segment in the first corpus is used as the first number, and the number of occurrences of the target character in the target word segment in the first corpus is used as the second number. The internal word association can be obtained based on the ratio between the first number and the second number; the first number and the second number can be adjusted based on a preset weight, and the internal word association can be obtained based on the difference or ratio between the adjusted first number and the second number; and so on.

[0069] Step S206 : obtaining the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of subwords in the target word segment in the first corpus.

[0070] The subwords in the target word segment are subsegments separated from the target word segment, that is, the subwords in the target word segment refer to the subsegments contained in the target word segment. Segmenting the target word segment can obtain the subwords contained in the target word segment. It is understood that there can be at least one subword in the target word segment. If the target word segment cannot be further split into fine-grained subwords, the word semantic independence can be a preset default value, for example, zero or a negative number.

[0071] The word semantic independence corresponding to a word fragment is calculated based on the number of occurrences of the word fragment in the corpus to which the word fragment belongs, and the number of occurrences of the subwords in the word fragment in the corpus to which the word fragment belongs. The word semantic independence corresponding to a word fragment can reflect the degree of semantic completeness of the word fragment. The greater the word semantic independence corresponding to the word fragment, the more complete the semantics of the word fragment, and the greater the possibility that the word fragment is a word. It can be understood that the closer the number of occurrences of the word fragment in the corpus to which the word fragment belongs and the number of occurrences of the subwords in the word fragment in the corpus to which the word fragment belongs, the greater the possibility that the word fragment is a word, and the closer the semantic completeness of the word fragment and the semantic completeness of the subwords in the word fragment.

[0072] Specifically, the computer device separates fine-grained subwords from the target word segment, counts the number of occurrences of the target word segment in the first corpus, counts the number of occurrences of the subwords in the target word segment in the first corpus, and obtains the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of the subwords in the target word segment in the first corpus. For example, the number of occurrences of the target word segment in the first corpus is used as the first count, and the number of occurrences of the subwords in the target word segment in the first corpus is used as the second count. The word semantic independence can be obtained based on the difference between the first count and the second count; the difference between the first count and the second count is used as the first difference, and the word semantic independence can be obtained based on the ratio of the first count and the first difference; and so on.

[0073] Step S208 : determining a word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment.

[0074] The word fragment vector is a feature vector corresponding to the word fragment, which is used to reflect the feature information of the word fragment.

[0075] Specifically, the computer device may establish a word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment. For example, the word segment vector may be obtained based on the word internal association and word semantic independence; or the word segment vector may be obtained based on the word internal association, word semantic independence, and other statistical features.

[0076] Step S210 , performing word recognition on the target word segment based on the word segment vector to obtain a word recognition result corresponding to the target word segment.

[0077] Among them, word recognition refers to identifying whether a word segment is a word. The standard word library can be a standard dictionary developed by a professional department or professionals, or a commonly used dictionary frequently used by people.

[0078] Specifically, the word segment vector corresponding to the target word segment includes rich information from different perspectives, such as the internal word association and word semantic independence corresponding to the target word segment. The computer device performs word recognition on the target word segment based on the word segment vector corresponding to the target word segment, and can obtain an accurate word recognition result corresponding to the target word segment. For example, the word segment vector corresponding to the target word segment includes the internal word association and word semantic independence corresponding to the target word segment. The weighted sum of the internal word association and word semantic independence is performed to obtain the word probability corresponding to the target word segment. If the word probability corresponding to the target word segment is greater than a preset probability, the word recognition result corresponding to the target word segment is determined to be that the target word segment is a word. If the word probability corresponding to the target word segment is less than or equal to the preset probability, the word recognition result corresponding to the target word segment is determined to be that the target word segment is not a word. For another example, the word fragment vector corresponding to the target word fragment is input into the word recognition model, and the word recognition model outputs the predicted word label corresponding to the target word fragment. If the predicted word label is a label indicating that the word fragment belongs to a word, then the word recognition result corresponding to the target word fragment is determined to be that the target word fragment is a word. If the predicted word label is other labels, then the word recognition result corresponding to the target word fragment is determined to be that the target word fragment is not a word.

[0079] In the above-mentioned word recognition method, a target word segment and a first corpus to which the target word segment belongs are obtained. Based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus, the internal word association corresponding to the target word segment is obtained. Based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of the subword in the target word segment in the first corpus, the word semantic independence corresponding to the target word segment is obtained. The subword is a sub-segment segmented from the target word segment. Based on at least the internal word association and the word semantic independence corresponding to the target word segment, a word segment vector corresponding to the target word segment is determined. Based on the word segment vector, word recognition is performed on the target word segment to obtain a word recognition result corresponding to the target word segment. In this way, the internal word association corresponding to the target word segment is obtained by statistically analyzing the relevant information of the characters in the target word segment. The internal word association can reflect the degree of association between the characters in the word segment. The greater the degree of association, the greater the possibility that the target word segment is a word. The word semantic independence of the target word fragment is determined by statistically analyzing the relevant information of finer-grained sub-segments within the target word fragment. This word semantic independence reflects the semantic completeness of the word fragment; the more complete the semantics, the greater the likelihood that the target word fragment is a word. The word fragment vector corresponding to the target word fragment is determined based on the word internal association and word semantic independence of the target word fragment. The word fragment vector contains information such as word internal association and word semantic independence, which aid in word recognition. Identifying whether the target word fragment is a word based on its word fragment vector can effectively improve word recognition accuracy. Furthermore, the computational complexity of word internal association and word semantic independence is low, which also helps improve word recognition efficiency.

[0080] In one embodiment, obtaining a target word segment and a first corpus to which the target word segment belongs includes:

[0081] A first corpus is obtained; each corpus in the first corpus belongs to the same field; based on a preset segmentation length set, each corpus in the first corpus is segmented to obtain a candidate word segment set that matches each segmentation length in the preset segmentation length set; each candidate word segment in the same candidate word segment set has the same segment length; and a target word segment is determined from each candidate word segment set.

[0082] The preset segmentation length set includes at least one preset segmentation length. The segmentation length is used to determine the segment length of a word segment segmented from the corpus. The segment length corresponding to a word segment refers to the number of characters contained in the word segment. For example, if the preset segmentation length is 3, then the word segment obtained by segmenting the corpus based on the segmentation length is a segment containing 3 characters, and the segment length of the word segment is 3.

[0083] Each corpus in the first corpus is segmented based on a segmentation length to obtain a candidate word fragment set, and the fragment length of each candidate word fragment in the candidate word fragment set is the segmentation length. For example, if the preset segmentation length is 5, each corpus in the first corpus is segmented based on the segmentation length, and the segments obtained by segmentation, each of which includes 5 characters, are combined into a candidate word fragment set.

[0084] Specifically, the computer device may first obtain a first corpus consisting of various corpora in the same field, then segment each corpus in the first corpus, and determine the target word segment from the segmentation results.

[0085] The computer device segments each corpus in the first corpus based on a preset segmentation length set to obtain a set of candidate word fragments that matches each segmentation length in the preset segmentation length set. For any segmentation length in the preset segmentation length set, the corpus in the first corpus is segmented using a sliding window whose window length matches the segmentation length. The sliding window is slid across the corpus character by character, and each slide captures a character string. The captured character string is used as a candidate word fragment, and the candidate word fragments are combined into a set of candidate word fragments.

[0086] The computer device ultimately determines a target word segment from the candidate word segment sets that match each segment length in the preset segment length set. For example, the candidate word segments in each candidate word segment set are respectively used as target word segments; and at least one candidate word segment is selected from each candidate word segment set as the target word segment.

[0087] In the above embodiment, a first corpus consisting of various corpora in the same field is first obtained, and then target word fragments are segmented from the first corpus. The first corpus includes various corpora belonging to the same field. By performing word recognition on the target word fragments segmented from the first corpus, it is helpful to discover new words in a certain field.

[0088] In one embodiment, obtaining the intra-word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus includes:

[0089] Each character in the target word segment is taken as the target character in turn; for any target character, the target character and its adjacent backward characters in the target word segment are combined into a character segment containing the target character in the target word segment; according to the appearance ratio between the number of appearances of the character segment containing the target character in the target word segment in the first corpus and the number of appearances of the target character in the target word segment in the first corpus, the appearance ratio corresponding to each character segment in the target word segment is obtained respectively; the appearance ratio corresponding to each character segment is smoothed to obtain the internal word association corresponding to each character segment; according to the internal word association corresponding to each character segment, the internal word association corresponding to the target word segment is obtained.

[0090] The target character's adjacent backward character in the target word segment refers to the character that is adjacent to the target character and located after the target character in the target word segment. In other words, the target character's adjacent backward character in the target word segment refers to the character that appears immediately after the target character in the target word segment. For example, if the target word segment is "Labor Fruit", and the target character is "Labor", then the target character's adjacent backward character in the target word segment is "Dynamic"; if the target character is "Fruit", then the target character's adjacent backward character in the target word segment is "Solid".

[0091] The target character and its adjacent backward characters in the target word segment are combined to form a character segment containing the target character in the target word segment. That is, the character segment containing the target character in the target word segment includes the target character and its adjacent backward characters in the target word segment. For example, if the target word segment is "Labor Fruit", if the target character is "Labor", then the character segment containing it is "Labor"; if the target character is "Fruit", then the character segment containing it is "Fruit".

[0092] It can be understood that for the last character in the target word segment, if there is no adjacent backward character in the target word segment, then there is no corresponding character segment for the character.

[0093] Data smoothing is performed to make the data smooth and stable for subsequent processing. Smoothing can be performed by compressing the data values, reducing the range of the data. For example, smoothing can be performed using a logarithmic function, a custom formula, or other methods.

[0094] Specifically, when calculating the internal word association corresponding to the target word fragment, the internal word association corresponding to each character fragment in the target word fragment can be calculated, and then the internal word association corresponding to each character fragment can be finally obtained based on the internal word association corresponding to each character fragment.

[0095] The computer device can sequentially use each character in the target word segment as a target character, exhausting the characters in the target word segment and fully utilizing the characters in the target word segment. For any target character, the target character and its adjacent subsequent characters in the target word segment are combined to form a character segment containing the target character in the target word segment, thereby obtaining character segments corresponding to each target character. For any target character, the number of occurrences of the character segment containing the target character in the target word segment in the first corpus is counted, and the number of occurrences of the target character in the target word segment in the first corpus is counted. The ratio between these two occurrences is calculated to obtain the occurrence ratio corresponding to the character segment. The target word segment includes at least two characters, each of which has a corresponding character segment. The occurrence ratio corresponding to each character segment in the target word segment can be calculated. Furthermore, to stabilize the occurrence ratio data for subsequent processing, the occurrence ratio corresponding to each character segment can be smoothed to obtain the intra-word association corresponding to each character segment. Finally, the computer device obtains the intra-word association corresponding to the target word segment based on the intra-word association corresponding to each character segment in the target word segment. For example, the internal word association degree corresponding to each character segment in the target word segment is averaged to obtain the internal word association degree corresponding to the target word segment; the weighted average of the internal word association degree corresponding to each character segment in the target word segment is calculated to obtain the internal word association degree corresponding to the target word segment, and the weight corresponding to the internal word association degree corresponding to each character segment can be set as needed; and so on.

[0096] For example, the target word segment is "labor fruit", the character segment corresponding to the target character "labor" is "labor", the character segment corresponding to the target character "movement" is "movement fruit", and the character segment corresponding to the target character "fruit" is "fruit". Based on the ratio between the number of occurrences of "labor" in the first corpus and the number of occurrences of "labor" in the first corpus, the number of occurrences corresponding to "labor" is obtained. Based on the ratio between the number of occurrences of "movement fruit" in the first corpus and the number of occurrences of "movement" in the first corpus, the number of occurrences corresponding to "movement fruit" is obtained. Based on the ratio between the number of occurrences of "fruit" in the first corpus and the number of occurrences of "fruit", the number of occurrences corresponding to "fruit" is obtained. The number of occurrences corresponding to "labor", "movement fruit", and "fruit" are smoothed respectively to obtain the internal word associations corresponding to "labor", "movement fruit", and "fruit". Based on the internal word associations corresponding to "labor", "movement fruit", and "fruit", the internal word association corresponding to "labor fruit" is obtained.

[0097] In one embodiment, the calculation formula for the word internal relevance T corresponding to the word fragment is:

[0098] Where l represents the length of the word segment, that is, the number of characters that the word segment consists of. i Indicates the i-th character of the word fragment; T i+1 Indicates the i+1th character of the word fragment; p(T i ) represents the number of times the i-th character of a word fragment appears in the corpus to which the word fragment belongs; p(T i |T i+1 ) represents the number of times the combination of the i-th character and the i+1-th character of the word fragment appears in the corpus to which the word fragment belongs. It can be understood that T i is the target character in the word fragment, p(T i ) is the number of times the target character in the word fragment appears in the corpus to which the word fragment belongs. i and T i+1 The character segments that make up the word segments contain the target characters, p(T i |T i+1 ) is the number of times the character segment containing the target character in the word segment appears in the corpus to which the word segment belongs. It is the ratio of the number of occurrences of the corresponding character segments. log represents smoothing. The internal word association degree corresponding to the character fragment. T is the internal word association degree corresponding to the word fragment. The greater the internal word association degree corresponding to the word fragment, the more stable the internal word association and the greater the possibility of the word forming a word. The internal word association degree corresponding to the word fragment can also be called the average internal word association.

[0099] In the above embodiment, each character in the target word segment is taken as the target character in turn, and the target character and the adjacent backward characters of the target character in the target word segment are combined into a character segment containing the target character in the target word segment. The internal word association corresponding to each character segment in the target word segment is calculated respectively, and then the internal word association corresponding to the target word segment is obtained based on the internal word association corresponding to each character segment in the target word segment. This makes the internal word association corresponding to the target word segment integrate the statistical information between each group of adjacent characters in the target word segment, can fully reflect the internal association of the word segment, and has high accuracy.

[0100] In one embodiment, the word semantic independence corresponding to the target word segment is obtained based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of subwords in the target word segment in the first corpus, including:

[0101] The target word fragment is segmented to obtain the subwords contained in the target word fragment; the fused occurrence count is obtained according to the number of occurrences of each subword of the target word fragment in the first corpus; the occurrence count of the target word fragment in the first corpus is adjusted based on the total number of subwords corresponding to the target word fragment to obtain the reference occurrence count; the difference in occurrence count between the fused occurrence count and the reference occurrence count is determined; the semantic independence of the word corresponding to the target word fragment is obtained based on the number of occurrences of the target word fragment in the first corpus and the difference in occurrence count.

[0102] The total number of subwords corresponding to the target word segment refers to the number of subwords contained in the target subsegment. For example, if the target word segment is "labor fruit," "labor fruit" is segmented to obtain "labor" and "fruit." "Labor" and "fruit" are the two subwords contained in "labor fruit," and the total number of subwords corresponding to "labor fruit" is 2.

[0103] Specifically, the target word segment can be composed of fine-grained sub-segments. When calculating the semantic independence of the word corresponding to the target word segment, the fine-grained sub-segments in the target word segment can be exhausted, and all sub-words contained in the target word segment can be fully utilized. The computer device performs word segmentation on the target word segment to obtain the sub-words contained in the target word segment, counts the number of occurrences of each sub-word contained in the target word segment in the first corpus, and obtains the fused number of occurrences based on the number of occurrences of each sub-word contained in the target word segment in the first corpus. For example, the number of occurrences of each sub-word contained in the target word segment in the first corpus can be summed to obtain the fused number of occurrences; the number of occurrences of each sub-word contained in the target word segment in the first corpus can be weighted and summed to obtain the fused number of occurrences, and the weights corresponding to the number of occurrences of each sub-word can be set as needed; and so on. Then, the number of occurrences of the target word segment in the first corpus can be adjusted based on the number of sub-words corresponding to the target word segment to obtain the reference number of occurrences. For example, the number of occurrences of the target word segment in the first corpus is magnified based on the number of subwords corresponding to the target word segment; the reference number of occurrences obtained by adjustment is proportional to the number of subwords used for adjustment; and so on. Further, the difference between the reference number of occurrences and the fused number of occurrences can be calculated to obtain the number of occurrences difference. For example, the difference between the reference number of occurrences and the fused number of occurrences is calculated, and the difference is used as the number of occurrences difference; the difference is adjusted based on a preset value to obtain the number of occurrences difference. Finally, the word semantic independence corresponding to the target word segment can be obtained based on the number of occurrences of the target word segment in the first corpus and the number of occurrences difference. For example, the ratio of the number of occurrences of the target word segment in the first corpus to the number of occurrences difference is used as the word semantic independence; the difference between the number of occurrences of the target word segment in the first corpus and the number of occurrences difference is used as the word semantic independence.

[0104] It is understandable that the target word segment can be segmented using an existing segmentation tool to obtain the subwords contained in the target word segment. For example, the jieba (jieba) segmentation tool can be used to segment the target word segment.

[0105] In one embodiment, the calculation formula for the word semantic independence W corresponding to the word fragment is:

[0106] Where w represents a word fragment; p(w) represents the number of times word fragment w appears in the corpus; N represents that word fragment w can be composed of N fine-grained sub-fragments; w j represents the jth sub-segment among N fine-grained sub-segments; p(w j ) represents the sub-segment w j The number of times the word fragment w appears in the corpus to which it belongs. It can be understood that N is the total number of subwords corresponding to the word fragment, p(w) is the number of times the word fragment appears in the corpus to which the word fragment belongs, and w j That is, the jth subword in the word fragment, p(w j ) is the number of occurrences of the j-th subword in the word fragment in the corpus to which the word fragment belongs, is the number of fusion occurrences, (N-1)p(w) is the number of reference occurrences, This is the difference in the number of occurrences. The greater the word semantic independence W corresponding to a word fragment, the more stable and semantically complete the word fragment, and the greater its probability of being a word. For a given word fragment, the numerator in the calculation formula is fixed. The fewer fine-grained sub-segments the word fragment is broken down into, the smaller the denominator in the calculation formula, and the larger the result, indicating a greater word semantic independence for the word fragment. The more fine-grained sub-segments the word fragment is broken down into, the larger the denominator in the calculation formula, and the lower the word semantic independence for the word fragment, and the lower its probability of being a word. For example, the word fragment W is "the fruits of labor," which can be divided into two more fine-grained sub-segments: "labor" and "fruit." Taken individually, "labor" and "fruit" do not have a specific meaning and are therefore less likely to be a word. However, when combined into "the fruits of labor," they have the specific meaning of "the result obtained through a certain action," making them more likely to be a word. The word semantic independence corresponding to a word fragment can also be called the word semantic independence value.

[0107] In the above embodiment, the target word segment is segmented to obtain the subwords contained in the target word segment. The fused occurrence count is obtained based on the number of occurrences of each subword of the target word segment in the first corpus. The number of occurrences of the target word segment in the first corpus is adjusted based on the total number of subwords corresponding to the target word segment to obtain a reference occurrence count. The difference in the number of occurrences between the fused occurrence count and the reference occurrence count is determined. The word semantic independence corresponding to the target word segment is obtained based on the number of occurrences of the target word segment in the first corpus and the difference in the number of occurrences. In this way, when calculating the word semantic independence, the statistical information of all subwords contained in the reference word segment is referenced, and the combined value of the statistical information of all subwords is compared with the statistical information of the word segment. This allows the word semantic independence to fully reflect the differences between the word segment and the subwords within the word segment, fully reflect the semantic integrity of the word segment, and has a high degree of accuracy.

[0108] In one embodiment, determining a word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment includes:

[0109] Obtain a set of candidate word fragments to which the target word fragment belongs; the fragment lengths of the candidate word fragments in the candidate word fragment set are consistent, and the candidate word fragment set is obtained by segmenting the corpus in the first corpus set; determine at least one reference word fragment corresponding to the target word fragment from the candidate word fragment set; determine a target distance from the edit distances between the target word fragment and the corresponding at least one reference word fragment, and determine the word text independence corresponding to the target word fragment based on the target distance; determine a word fragment vector corresponding to the target word fragment based at least on the word internal association, word semantic independence and word text independence corresponding to the target word fragment.

[0110] The edit distance between two word segments is the minimum number of edit operations required to transform one segment into the other. The larger the edit distance between two segments, the more different they are. Permitted edit operations include replacing one character with another, inserting a character, and deleting a character. The edit distance is also called the Levenshtein distance or edit distance.

[0111] The reference word segment corresponding to the target word segment refers to a word segment whose segment length is consistent with the segment length of the target word segment, and the reference word segment corresponding to the target word segment is also consistent with the corpus to which the target word segment belongs. The reference word segment corresponding to the target word segment is determined from the set of candidate word segments to which the target word segment belongs. The set of candidate word segments to which the target word segment belongs includes at least one candidate word segment contained in the first corpus to which the target word segment belongs, whose segment length is consistent with the segment length of the target word segment. It can be understood that the target word segment and the corresponding reference word segment are different word segments.

[0112] The word text independence of a word fragment is calculated based on the edit distance between each word fragment and at least one corresponding reference word fragment. The smaller the edit distance between a word fragment and its corresponding reference word fragment, the more independent the word fragment is. The greater the word text independence of the word fragment, the greater the probability that it is a word.

[0113] Specifically, in addition to performing word recognition based on the internal word association and word semantic independence corresponding to the target word fragment, the word text independence corresponding to the target word fragment can be further introduced to perform word recognition based on the internal word association, word semantic independence and word text independence corresponding to the target word fragment.

[0114] The computer device can obtain a set of candidate word segments to which the target word segment belongs, and determine at least one reference word segment corresponding to the target word segment from the set of candidate word segments to which the target word segment belongs. For example, each candidate word segment in the set of candidate word segments to which the target word segment belongs can be used as a reference word segment corresponding to the target word segment; a candidate word segment that appears more than a preset number of times in the first corpus can be selected from the set of candidate word segments to which the target word segment belongs as a reference word segment corresponding to the target word segment, with candidate word segments that appear more than a preset number of times in the first corpus being more representative; and so on. Furthermore, the computer device determines a target distance from the edit distances between the target word segment and the corresponding at least one reference word segment, and determines the word text independence corresponding to the target word segment based on the target distance. For example, the minimum value from each edit distance is obtained as the target distance, and the target distance is used as the word text independence; the average value of each edit distance is calculated as the target distance, and the target distance is used as the word text independence; and so on.

[0115] Ultimately, the computer device can determine the word fragment vector corresponding to the target word fragment based at least on the internal word association, word semantic independence and word text independence corresponding to the target word fragment, so that the word fragment vector includes information from at least three perspectives, thereby improving the accuracy of word recognition.

[0116] In one embodiment, the calculation formula for the word text independence S corresponding to the word segment is:

[0117] S=min{D word,k =D(word,k),k∈Q}

[0118] Among them, D word,kRepresents the edit distance between word fragment word and word fragment k, word fragment word is the word fragment whose corresponding word text independence needs to be calculated, word fragment k is the reference word fragment corresponding to word fragment word; D(word,k) represents the edit distance calculation formula; Q is a set of word fragments other than word fragment word (for example, word fragment word is a fragment consisting of 3 characters, then Q is a set of word fragments, the word fragments in Q are all fragments consisting of 3 characters, and Q does not include word fragment word); min{} represents taking the minimum value of all D(word,k). It can be understood that word fragment word is the target word fragment, word fragment k is the reference word fragment corresponding to the target word fragment, and the set composed of Q and word fragment word is the candidate word fragment set to which the target word fragment belongs. min{D word,k = D(word,k), k∈Q} is the target distance. Words typically possess a certain degree of novelty and independence. The larger the minimum value of S, the higher the word-text independence of the word fragment and the greater the probability that it is a word. The word-text independence corresponding to a word fragment can also be called the word-text independence value.

[0119] In the above embodiment, the target word segment and the corresponding reference word segment have the same segment length and belong to the same corpus. The target distance is determined from the edit distance between the target word segment and at least one corresponding reference word segment. The word text independence corresponding to the target word segment is determined based on the target distance. This allows the word text independence to reflect the textual differences between the target word segment and the reference word segment, embodying the independence of the target word segment with high accuracy. A word segment vector corresponding to the target word segment is established based on at least the word internal association, word semantic independence, and word text independence corresponding to the target word segment. This provides the word segment vector with richer information, thereby helping to improve the accuracy of word recognition.

[0120] In one embodiment, determining a word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment includes:

[0121] Obtaining supplementary feature sets corresponding to the target word fragments respectively; determining the word fragment vectors corresponding to the target word fragments based at least on the word internal association, word semantic independence and supplementary feature sets corresponding to the target word fragments.

[0122] The supplementary feature set includes at least one of the word external association, part-of-speech value, corpus importance, and segment length corresponding to the target word segment.

[0123] Perform character type statistics on the adjacent forward characters corresponding to a word segment in the corpus to which the word segment belongs, and obtain the number of distinct adjacent forward characters corresponding to the word segment in the corpus to which the word segment belongs. The number of distinct adjacent forward characters corresponding to a word segment in its corpus refers to the number of character types of the single characters that are directly adjacent to the word segment and are in front of the word segment in the corpus to which the word segment belongs. That is, the number of distinct adjacent forward characters corresponding to a word segment in its corpus is used to represent how many different characters the characters on the left side of the word segment appear in the corpus to which the word segment belongs. Perform character type statistics on the adjacent backward characters corresponding to a word segment in the corpus to which the word segment belongs, and obtain the number of distinct adjacent backward characters corresponding to the word segment in the corpus to which the word segment belongs. The number of distinct adjacent backward characters corresponding to a word segment in its corpus refers to the number of character types of the single characters that are directly adjacent to the word segment and are behind the word segment in the corpus to which the word segment belongs. That is, the number of distinct adjacent backward characters corresponding to a word segment in its corpus is used to represent how many different characters the characters on the right side of the word segment appear in the corpus to which the word segment belongs. It can be understood that the adjacent forward characters corresponding to a word segment in the corpus to which the word segment belongs refer to the characters that are adjacent to the word segment and are in front of the word segment in the corpus to which the word segment belongs. The adjacent backward characters corresponding to a word segment in the corpus to which the word segment belongs refer to the characters that are adjacent to the word segment and are behind the word segment in the corpus to which the word segment belongs. For example, if the target word segment is "weather", and a corpus in the first corpus that includes the target word segment is "Today the weather is sunny", then "day" is an adjacent forward character corresponding to the target word segment in the first corpus, and "sunny" is an adjacent backward character corresponding to the target word segment in the first corpus.

[0124] The external word association degree corresponding to a word segment is calculated based on the number of distinct adjacent forward characters and the number of distinct adjacent backward characters corresponding to the word segment in the corpus to which the word segment belongs. For example, the sum of the number of distinct adjacent forward characters and the number of distinct adjacent backward characters is obtained as the external word association degree; the weighted sum of the number of distinct adjacent forward characters and the number of distinct adjacent backward characters is obtained as the external word association degree, and the weights corresponding to the number of distinct adjacent forward characters and the number of distinct adjacent backward characters can be set as needed; and so on. Further illustrate with an example, if the target word segment is "peace", there are 50 different characters that appear on the left side of "peace" in the first corpus, and 30 different characters that appear on the right side of "peace" in the first corpus, then the external word association degree of the target word segment "peace" is 80. The greater the external word association degree corresponding to a word segment, the richer the surrounding characters of the word segment, and the greater the possibility that the word segment forms a word.

[0125] The part-of-speech value corresponding to a word segment is obtained by counting the parts of speech exhibited by the word segment in the corresponding corpus. The part-of-speech value is used to indicate how many parts of speech the target word segment has in the first corpus. For example, if the target word segment exhibits both noun and adjective parts of speech in the first corpus, the part-of-speech value corresponding to the target word segment is 2. Generally speaking, the lower the part-of-speech value corresponding to a word segment, the more stable the word segment is, and the greater the probability that the word segment is a word. In one embodiment, the part-of-speech value corresponding to the word segment can be counted using a tool with a part-of-speech analysis function.

[0126] The corpus importance corresponding to a word fragment is calculated based on the first frequency and second frequency corresponding to the word fragment in the corpus to which the word fragment belongs. The first frequency refers to the frequency of occurrence of the word fragment in the corpus containing the word fragment, and the corpus containing the word fragment is in the corpus to which the word fragment belongs. The second frequency refers to the frequency of occurrence of the corpus containing the word fragment in the corpus to which the word fragment belongs. For example, the external relevance of the word is obtained by multiplying the first frequency and the second frequency. In one embodiment, the corpus importance is a TF-IDF (term frequency–inverse document frequency) value.

[0127] Specifically, in addition to performing word recognition based on the internal word association and word semantic independence corresponding to the target word fragment, a supplementary feature set corresponding to the target word fragment can be further introduced to perform word recognition based on the internal word association, word semantic independence and supplementary feature set corresponding to the target word fragment.

[0128] The computer device obtains the supplementary feature sets corresponding to the target word fragments respectively, and determines the word fragment vector corresponding to the target word fragment based at least on the internal word association, word semantic independence, and supplementary feature sets corresponding to the target word fragments, so that the word fragment vector includes information from multiple perspectives, thereby improving the accuracy of word recognition.

[0129] In the above embodiment, a word fragment vector corresponding to the target word fragment is established based at least on the internal word association, word semantic independence and supplementary feature set corresponding to the target word fragment, so that the word fragment vector has richer information, which helps to improve the accuracy of word recognition.

[0130] In one embodiment, a word segment vector corresponding to the target word segment is established based on at least the word internal association, word semantic independence, word text independence and a supplementary feature set corresponding to the target word segment.

[0131] In one embodiment, determining a word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment includes:

[0132] At least the internal word association and the word semantic independence corresponding to the target word fragment are normalized, and the normalized internal word association and word semantic independence corresponding to the target word fragment are concatenated to obtain a word fragment vector corresponding to the target word fragment.

[0133] Normalization is used to map the data values ​​to a preset range for subsequent data processing. For example, the value of the internal association of words can be mapped to a range between 0 and 1, and the value of the semantic independence of words can be mapped to a range between 0 and 1.

[0134] Specifically, when determining the word fragment vector corresponding to the target word fragment based at least on the internal word association and word semantic independence corresponding to the target word fragment, the internal word association corresponding to the target word fragment can be normalized first, and the word semantic independence corresponding to the target word fragment can be normalized, and then the normalized internal word association and word semantic independence can be spliced ​​to obtain the word fragment vector corresponding to the target word fragment.

[0135] It can be understood that the statistical features used to establish the word fragment vector corresponding to the word fragment include the internal correlation of the word, the semantic independence of the word, the text independence of the word and at least two statistical features in the supplementary feature set. When determining the word fragment vector corresponding to the word fragment based on the statistical features corresponding to the word fragment, the various statistical features can be normalized first, and then the various statistical features after normalization can be spliced ​​to obtain the word fragment vector corresponding to the word fragment.

[0136] In the above embodiment, normalizing various statistical features and then concatenating them to obtain word segment vectors can effectively avoid the weight of a single statistical feature being too large or too small, thereby affecting the accuracy of subsequent word recognition.

[0137] In one embodiment, word recognition is performed on the target word segment based on the word segment vector to obtain a word recognition result corresponding to the target word segment, including:

[0138] The word fragment vector is input into the target word recognition model, and the predicted word label corresponding to the target word fragment is output. The word recognition result corresponding to the target word fragment is determined based on the predicted word label corresponding to the target word fragment; the predicted word label includes any one of a first label for indicating that the word fragment is a non-word, a second label for indicating that the word fragment is a word that has been included in the standard vocabulary, and a third label for indicating that the word fragment is a word that has not been included in the standard vocabulary.

[0139] Among them, the word recognition model is a machine learning model used for word recognition. The input data of the word recognition model is the word segment vector corresponding to the word segment, and the output data is the predicted word label corresponding to the word segment. The word recognition model can output three types of predicted word labels, namely, a first label used to indicate that the word segment is a non-word, a second label used to indicate that the word segment is a word that has been included in the standard vocabulary, and a third label used to indicate that the word segment is a word that has not been included in the standard vocabulary. If the predicted word label corresponding to the word segment is the first label, it means that the word segment is not a word. If the predicted word label corresponding to the word segment is the second label, it means that the word segment is a word that has been included in the standard vocabulary. If the predicted word label corresponding to the word segment is the third label, it means that the word segment is a word that has not been included in the standard vocabulary. The target word recognition model refers to a word recognition model that has completed training. The word segment vector corresponding to the word segment is input into the target word recognition model. The target word recognition model processes the input data and selects one label from the three labels based on the data processing results.

[0140] Specifically, the computer device can perform word recognition with the help of a machine learning model. The computer device can obtain a target word recognition model, input the word segment vector corresponding to the target word segment into the target word recognition model, the target word recognition model processes the input data to obtain a predicted word label corresponding to the target word segment, and the target word recognition model outputs the predicted word label corresponding to the target word segment. The predicted word label corresponding to the target word segment includes any one of a first label for indicating that the word segment is a non-word, a second label for indicating that the word segment is a word that has been included in the standard vocabulary, and a third label for indicating that the word segment is a word that has not been included in the standard vocabulary. The predicted word label corresponding to the target word segment can be used as the word recognition result corresponding to the target word segment.

[0141] It can be understood that the machine learning algorithm used to establish the word recognition model can be selected according to actual needs. For example, a word recognition model is established based on the LR (Logistic Regression) algorithm, a word recognition model is established based on the deepFM algorithm, and a word recognition model is established based on the XGboost (Extreme Gradient Boosting) algorithm.

[0142] In the above embodiment, the word segment vector corresponding to the target word segment is input into the target word recognition model for word recognition. The target word recognition model has a strong learning ability and can accurately recognize words based on the knowledge it has learned, effectively ensuring the accuracy of word recognition. In addition, inputting the word segment vector corresponding to the target word segment into the target word recognition model for word recognition can also ensure the efficiency of word recognition.

[0143] In one embodiment, the word recognition method further includes:

[0144] Based on the first target word segment and the second target word segment corresponding to the first corpus, a domain dictionary corresponding to the domain to which the first corpus belongs is determined.

[0145] Among them, the predicted word label corresponding to the first target word segment is a label used to indicate that the word segment is a word that has been included in the standard vocabulary, and the predicted word label corresponding to the second target word segment is a label used to indicate that the word segment is a word that has not been included in the standard vocabulary.

[0146] Specifically, a plurality of corpora belonging to the same field are obtained to form a first corpus, and words are identified from the first corpus to establish a domain dictionary. A computer device segments the corpus in the first corpus to obtain a plurality of candidate word segments, each of which is used as a target word segment. Word recognition is performed on the target word segment by establishing a word segment vector corresponding to the target word segment, and word recognition results corresponding to each target word segment corresponding to the first corpus are obtained. From these, first target word segments whose word recognition results are words that are not included in the standard lexicon and second target word segments whose word recognition results are words that are included in the standard lexicon are obtained to form a domain dictionary corresponding to the field to which the first corpus belongs.

[0147] In one embodiment, the computer device may build a knowledge graph corresponding to the domain to which the first corpus belongs based on the domain dictionary. For example, during the construction of the knowledge graph, entities need to be identified. Entity recognition can be performed based on the terms in the domain dictionary to identify richer and more comprehensive entities, thereby improving the quality of the knowledge graph.

[0148] In the above embodiment, the words recognized by the word recognition method of the present application are applied to the construction of a domain dictionary, which can enrich the content of the domain dictionary.

[0149] In one embodiment, as shown in FIG4 , a word recognition model training method is provided. The method is applied to a computer device as an example. The computer device can be a terminal or a server. Referring to FIG4 , the word recognition model training method includes the following steps:

[0150] Step S402: Obtain a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs.

[0151] A training word segment refers to a word segment used in training data for generating a model. A training word segment is a word segment for which a word recognition result is known. The training word tag corresponding to the training word segment is used to indicate the correct word recognition result corresponding to the training word segment. The second corpus to which the training word segment belongs includes at least one corpus containing the training word segment. That is, the training word segment appears in at least one corpus in the second corpus to which the training word segment belongs.

[0152] Specifically, the computer device can obtain relevant data of training word segments to establish training data for the word recognition model, train the word recognition model based on the training data, and obtain a machine learning model for word recognition.

[0153] Step S404 : obtaining the internal word association corresponding to the training word segment based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus.

[0154] Step S406 : Obtain the word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of subwords in the training word segment in the second corpus.

[0155] Step S408 : determining a word segment vector corresponding to the training word segment based at least on the word internal association and word semantic independence corresponding to the training word segment.

[0156] It can be understood that the calculation process of the internal word association corresponding to the training word fragment can refer to the calculation process of the internal word association corresponding to the aforementioned target word fragment, the calculation process of the word semantic independence corresponding to the training word fragment can refer to the calculation process of the word semantic independence corresponding to the aforementioned target word fragment, and the determination process of the word fragment vector corresponding to the training word fragment can refer to the establishment process of the word fragment vector corresponding to the aforementioned target word fragment, which will not be repeated here.

[0157] Step S410 : training the initial word recognition model based on the word segment vectors and training word labels corresponding to the training word segments to obtain a target word recognition model.

[0158] The initial word recognition model refers to the word recognition model to be trained, and the target word recognition model refers to the word recognition model that has been trained.

[0159] Specifically, referring to the process of establishing the word segment vector corresponding to the target word segment, a word segment vector corresponding to the training word segment is established based on the training word segment and the second corpus to which the training word segment belongs. The word segment vector and the training word label corresponding to the training word segment are used as training data for the model. During model training, the word segment vector corresponding to the training word segment is used as input data for the model, and the training word label corresponding to the training word segment is used as expected output data for the model. The computer device trains the initial word recognition model based on the word segment vector and the training word label corresponding to the training word segment to obtain the target word recognition model.

[0160] For example, referring to FIG5 , the statistical features corresponding to the training word segments include part-of-speech values, word external association, word internal association, word TF-IDF value, word length, word semantic independence, and word text independence. Each of the statistical features corresponding to the training word segments is normalized, and the normalized statistical features corresponding to the training word segments form a word segment vector corresponding to the training word segment. The word segment vector corresponding to the training word segment is input into the word recognition model to be trained. The word recognition model processes the input data and outputs a predicted word label corresponding to the training word segment. A model loss is generated based on the difference between the predicted word label corresponding to the training word segment and the training word label. The model loss is back-propagated to adjust the parameters of the word recognition model. Through iterative training, the model parameters are continuously adjusted until the training end condition is met, thereby obtaining a trained word recognition model. The training end condition can be at least one condition indicating that the model training has been completed, such as the model loss being less than a preset loss value, the number of model iterations being greater than a preset number of iterations, or the like.

[0161] It can be understood that the training data of the model may include word segment vectors and training word labels of multiple training word segments. The model is trained with a large amount of training data so that the model can learn relevant knowledge about the word recognition task.

[0162] In one embodiment, different word recognition models can be trained for different fields. For example, for the medical field, a second corpus consisting of medical texts is obtained, the corpus in the second corpus is segmented to obtain multiple training word segments, word segment vectors corresponding to the training word segments are established based on the second corpus, and a word recognition model for word recognition in the medical field is trained based on the word segment vectors and training word labels corresponding to a large number of training word segments. For the gaming field, a second corpus consisting of gaming texts is obtained, the corpus in the second corpus is segmented to obtain multiple training word segments, word segment vectors corresponding to the training word segments are established based on the second corpus, and a word recognition model for word recognition in the gaming field is trained based on the word segment vectors and training word labels corresponding to a large number of training word segments.

[0163] In one embodiment, the first corpus is segmented based on a preset first segmentation length set to determine target word segments, and the second corpus is segmented based on a preset second segmentation length set to determine training word segments. In order to ensure the efficiency of model training, the second segmentation length set has a segmentation length restriction. During the training process of the model, the model has gradually learned the relevant knowledge of the word recognition task. During the application of the model, the first segmentation length set may not have a segmentation length restriction, and the model can perform word recognition on word segments of any segment length, thereby ensuring the versatility of the method of the present application.

[0164] The above-mentioned word recognition model training method obtains a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs. Based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus, the internal word association corresponding to the training word segment is obtained. Based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of the subwords in the training word segment in the second corpus, the word semantic independence corresponding to the training word segment is obtained. Based on at least the internal word association and the word semantic independence corresponding to the training word segment, a word segment vector corresponding to the training word segment is determined. Based on the word segment vector and the training word label corresponding to the training word segment, an initial word recognition model is trained to obtain a target word recognition model. In this way, the internal word association corresponding to the training word segment is obtained by statistically analyzing the relevant information of the characters in the training word segment. The internal word association can reflect the degree of association between the characters in the word segment. The greater the degree of association, the greater the likelihood that the training word segment is a word. By collecting statistics on the relevant information of finer-grained sub-segments in the training word segment, the word semantic independence corresponding to the training word segment is obtained. Word semantic independence can reflect the semantic completeness of the word segment. The more complete the semantics, the greater the likelihood that the training word segment is a word. The word segment vector corresponding to the training word segment is determined based on the internal word association and word semantic independence corresponding to the training word segment. The word segment vector contains information such as internal word association and word semantic independence that helps with word recognition. The word segment vector and training label corresponding to the training word segment are used as training data to train the word recognition model. This allows the word recognition model to learn the knowledge related to the word in the training data to recognize the word. Using the trained word recognition model to perform word recognition can improve the accuracy and efficiency of word recognition.

[0165] In a specific embodiment, the method of the present application can be applied to word recognition for a specific field. As shown in FIG6 , the word recognition method of the present application includes the following steps:

[0166] 1. Domain Corpus Acquisition

[0167] The computer device obtains multiple corpora belonging to the same field to form a domain corpus set A, and the domain corpus set A is used for model training.

[0168] 2. Domain Corpus Segmentation

[0169] The computer device segments the domain corpus A using n-grams to obtain n-words, which are the training word segments. The value of n is generally less than or equal to 4. For example, the domain corpus A is segmented using 2-grams, 3-grams, and 4-grams, respectively, to obtain segments consisting of 2 characters, 3 characters, and 4 characters. It is understood that segments consisting of 1 character are less likely to be words, so it is not necessary to obtain segments consisting of 1 character.

[0170] 3. Feature construction

[0171] Referring to Table 1, for any training word segment, the computer device calculates various statistical features corresponding to the training word segment, including part-of-speech value, word external association, word internal association, word TF-IDF value, word length value (i.e., segment length), word semantic independence, and word text independence.

[0172] Table 1

[0173] The part-of-speech value refers to the number of parts of speech that a word fragment has in the domain corpus A. Generally, the lower the part-of-speech value of a word fragment, the more stable the word fragment is and the greater the possibility that the word fragment will become a word.

[0174] The external relevance of a word refers to the number of different expressions that appear on the left and right sides of a word fragment in the domain corpus A. The greater the external relevance of a word fragment, the richer the surrounding words of the word fragment, and the greater the possibility that the word fragment will become a word.

[0175] The calculation formula for word internal correlation is: The greater the internal correlation of a word fragment, the more stable the internal correlation is, and the greater the possibility that the word fragment will become a word.

[0176] The word TF-IDF value is used to measure the importance of a word fragment in the domain corpus A. Generally, words have more important meanings. The larger the word TF-IDF value of a word fragment, the more likely it is that the word fragment is a word.

[0177] The word length value indicates the number of characters in the word. Because word fragments of different lengths have different distributions of internal word association, generally, the longer the word fragment, the greater the internal word association. Including the word length value feature can fully leverage the positive impact of internal word association on the model. Including the word length value feature adds more prior knowledge, allowing the model to learn more useful information and understand what kind of internal word association a word of a certain length should have. Word fragments that meet this pattern are more likely to be real words, avoiding model misjudgments. Regarding word semantic independence, the greater the word semantic independence of a word fragment, the more stable the word fragment, the more complete the semantics, and the greater the likelihood that the word fragment is a word.

[0178] The calculation formula of word semantic independence is: The greater the semantic independence of a word fragment, the more stable the word fragment, the more complete the semantics, and the greater the possibility that it is a word.

[0179] The calculation formula of word text independence value is S=min{D word,k =D(word,k),k∈Q}. Words usually have a certain degree of novelty and independence. The larger the word text independence value of a word fragment, the higher the text independence of the word fragment, and the greater the probability that the word fragment is a word.

[0180] 4. Normalization

[0181] The computer equipment normalizes the various features mentioned above to avoid the weight of a single feature being too large or too small, which affects the recognition accuracy of the model.

[0182] 5. Model training

[0183] As shown in Table 1, the "lable" column is the training word label corresponding to the segment of n characters. "0" represents that the training word segment is not a word, "1" represents that the training word segment is a word that has been included in the standard vocabulary, and "2" represents that the training word segment is a word that is not included in the standard vocabulary. The computer device reads the normalized data from the second to eighth columns of Table 1 and converts it into an array format as the feature value array X. It then reads the value of the ninth column of Table 1 and converts it into an array format as the label array Y. The data in the label array Y corresponds one-to-one with the data in the feature value array X. The corresponding data corresponds to the data of the same training word segment. The feature value array X is split into a training set X_train and a test set X_test, and the label array Y is split into a training set Y_train and a test set Y_test. The model is trained based on the training sets X_train and Y_train. After the training, the model is tested based on the test sets X_test and Y_test. The model that passes the test is regarded as the final qualified model.

[0184] 6. Model prediction

[0185] A new corpus from the same domain as the corpus in domain corpus A is selected to form domain corpus B. The corpus in domain corpus B is segmented using n-grams to obtain segments consisting of n characters. Each segment consisting of n characters is used as a word segment to be identified. The statistical features corresponding to the word segments to be identified include part-of-speech value, word external association, word internal association, word TF-IDF value, word length value (i.e., segment length), word semantic independence, and word text independence. The feature value corresponding to each word segment to be identified is calculated and input into the trained model to obtain the predicted word label for each word segment to be identified. During model prediction, the value of n can be greater than 4, so that the model can recognize words of different lengths.

[0186] The word recognition method of the present application can accurately identify whether a word segment is a word with high accuracy. Furthermore, there is no limit on the length of the word segment during model prediction, making it widely applicable. Furthermore, the calculation method for various statistical features is not complex, and using the model for word recognition helps ensure the efficiency of word recognition.

[0187] In a specific embodiment, as shown in FIG7 , the word recognition method of the present application can be applied to construct a domain dictionary for a specific field. For example, by performing word recognition using the word recognition method of the present application, words of different lengths in the medical field can be identified, and the identified new words can be added to the existing domain dictionary in the medical field to enrich the content of the domain dictionary. For example, by performing word recognition using the word recognition method of the present application, words of different lengths in the financial field can be identified, and the identified words can be combined to form a domain dictionary for word segmentation of financial field corpus.

[0188] Furthermore, as shown in FIG7 , the domain dictionary constructed by the word recognition method of the present application can be applied to construct a knowledge graph in a specific field, and the constructed knowledge graph can be further applied to build various types of answering dialogue robots in a specific field. For example, in the construction process of various types of answering dialogue robots in the financial field (such as bank sales robots, bank consulting robots, and bank operation robots), it is necessary to build a complete knowledge graph to improve the service quality of the answering dialogue robot. The knowledge graph determines the comprehensiveness of the knowledge reserve of the answering dialogue robot. In the construction process of the knowledge graph, large and comprehensive entity information is required. The domain dictionary in the financial field constructed by the word recognition method of the present application can be used to assist the entity information in the financial field. For example, the knowledge graph can be a question-answer pair consisting of questions and answers. The establishment of questions and answers involves words. In the knowledge graph, it can be multiple questions corresponding to one answer, or it can be one question corresponding to one answer. The more complete the relevant knowledge graph is, the more the robot knows, and the better it can serve customers' questions or inquiries.

[0189] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0190] Based on the same inventive concept, the present application also provides a word recognition device for implementing the above-mentioned word recognition method. The implementation solution provided by this device is similar to the implementation solution described in the above-mentioned method. Therefore, the specific limitations of one or more word recognition device embodiments provided below can be found in the above-mentioned limitations of the word recognition method, and will not be repeated here.

[0191] In one embodiment, as shown in FIG8 , a word recognition device is provided, including: an acquisition module 802 , a determination module 804 , and a recognition module 806 , wherein:

[0192] The acquisition module 802 is configured to acquire a target word segment and a first corpus to which the target word segment belongs.

[0193] Determination module 804 is used to obtain the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus; obtain the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of subwords in the target word segment in the first corpus; subwords are sub-segments cut out from the target word segment; and determine the word segment vector corresponding to the target word segment based on at least the internal word association and word semantic independence corresponding to the target word segment.

[0194] The recognition module 806 is configured to perform word recognition on the target word segment based on the word segment vector to obtain a word recognition result corresponding to the target word segment.

[0195] In one embodiment, the acquisition module 802 is further configured to:

[0196] A first corpus is obtained; each corpus in the first corpus belongs to the same field; based on a preset segmentation length set, each corpus in the first corpus is segmented to obtain a candidate word segment set that matches each segmentation length in the preset segmentation length set; each candidate word segment in the same candidate word segment set has the same segment length; and a target word segment is determined from each candidate word segment set.

[0197] In one embodiment, the determination module 804 is further configured to:

[0198] Each character in the target word segment is taken as the target character in turn; for any target character, the target character and its adjacent backward characters in the target word segment are combined into a character segment containing the target character in the target word segment; the adjacent backward characters are characters that are adjacent to the target character and located after the target character in the target word segment; according to the appearance ratio between the number of occurrences of the character segment containing the target character in the target word segment in the first corpus and the number of occurrences of the target character in the target word segment in the first corpus, the appearance ratio corresponding to each character segment in the target word segment is obtained respectively; the appearance ratio corresponding to each character segment is smoothed to obtain the internal word association corresponding to each character segment; according to the internal word association corresponding to each character segment, the internal word association corresponding to the target word segment is obtained.

[0199] In one embodiment, the determination module 804 is further configured to:

[0200] The target word fragment is segmented to obtain the subwords contained in the target word fragment; the fused occurrence count is obtained according to the number of occurrences of each subword of the target word fragment in the first corpus; the occurrence count of the target word fragment in the first corpus is adjusted based on the total number of subwords corresponding to the target word fragment to obtain the reference occurrence count; the difference in occurrence count between the fused occurrence count and the reference occurrence count is determined; the semantic independence of the word corresponding to the target word fragment is obtained based on the number of occurrences of the target word fragment in the first corpus and the difference in occurrence count.

[0201] In one embodiment, the determination module 804 is further configured to:

[0202] Obtain a set of candidate word fragments to which the target word fragment belongs; the fragment lengths of the candidate word fragments in the candidate word fragment set are consistent, and the candidate word fragment set is obtained by segmenting the corpus in the first corpus set; determine at least one reference word fragment corresponding to the target word fragment from the candidate word fragment set; determine a target distance from the edit distances between the target word fragment and the corresponding at least one reference word fragment, and determine the word text independence corresponding to the target word fragment based on the target distance; determine a word fragment vector corresponding to the target word fragment based at least on the word internal association, word semantic independence and word text independence corresponding to the target word fragment.

[0203] In one embodiment, the determination module 804 is further configured to:

[0204] Obtain the supplementary feature sets corresponding to the target word fragments respectively; the supplementary feature sets include at least one of the word external association degree, part-of-speech value, corpus importance, and fragment length corresponding to the target word fragment; the word external association degree is obtained based on the number of adjacent forward deduplicated characters and the number of adjacent backward deduplicated characters corresponding to the target word fragment in the first corpus, the number of adjacent forward deduplicated characters is obtained by performing character type statistics on the adjacent forward characters corresponding to the target word fragment in the first corpus, the adjacent forward characters are characters adjacent to the target word fragment in the first corpus and located before the target word fragment, and the number of adjacent backward deduplicated characters is obtained by performing character type statistics on the adjacent backward characters corresponding to the target word fragment in the first corpus. The adjacent backward characters are the characters adjacent to the target word fragment and located after the target word fragment in the first corpus; the part-of-speech value is obtained by counting the part of speech reflected by the target word fragment in the first corpus; the corpus importance is obtained according to the first frequency and second frequency corresponding to the target word fragment in the first corpus, the first frequency is the frequency of occurrence of the target word fragment in the corpus containing the target word fragment, and the corpus containing the target word fragment is in the first corpus, and the second frequency is the frequency of occurrence of the corpus containing the target word fragment in the first corpus; the word fragment vector corresponding to the target word fragment is determined based on at least the internal correlation degree, word semantic independence and supplementary feature set corresponding to the target word fragment.

[0205] In one embodiment, the identification module 806 is further configured to:

[0206] The word fragment vector is input into the target word recognition model, and the predicted word label corresponding to the target word fragment is output. The word recognition result corresponding to the target word fragment is determined based on the predicted word label corresponding to the target word fragment; the predicted word label includes any one of a first label for indicating that the word fragment is a non-word, a second label for indicating that the word fragment is a word that has been included in the standard vocabulary, and a third label for indicating that the word fragment is a word that has not been included in the standard vocabulary.

[0207] In one embodiment, the word recognition device is further configured to:

[0208] Based on the first target word segment and the second target word segment corresponding to the first corpus, the domain dictionary corresponding to the domain to which the first corpus belongs is determined; the predicted word label corresponding to the first target word segment is a label used to indicate that the word segment is a word that has been included in the standard vocabulary, and the predicted word label corresponding to the second target word segment is a label used to indicate that the word segment is a word that has not been included in the standard vocabulary.

[0209] The above-mentioned word recognition device obtains the internal word association corresponding to the target word segment by counting the relevant information of the characters in the target word segment. The internal word association can reflect the degree of association between the characters in the word segment. The greater the degree of association, the more likely the target word segment is a word. The word semantic independence corresponding to the target word segment is obtained by counting the relevant information of more fine-grained sub-segments in the target word segment. The word semantic independence can reflect the semantic completeness of the word segment. The more complete the semantics, the more likely the target word segment is a word. Based on the internal word association and word semantic independence corresponding to the target word segment, a word segment vector corresponding to the target word segment is established. The word segment vector contains information such as the internal word association and word semantic independence that are helpful for word recognition. Identifying whether the target word segment is a word based on the word segment vector corresponding to the target word segment can effectively improve word recognition accuracy.

[0210] Based on the same inventive concept, the present application also provides a word recognition model training device for implementing the above-mentioned word recognition model training method. The implementation solution provided by this device is similar to the implementation solution described in the above-mentioned method. Therefore, the specific limitations of one or more word recognition model training device embodiments provided below can be found in the above-mentioned limitations of the word recognition model training method and will not be repeated here.

[0211] In one embodiment, as shown in FIG9 , a word recognition model training device is provided, comprising: a data acquisition module 902 , a vector determination module 904 , and a model training module 906 , wherein:

[0212] A data acquisition module 902 is configured to acquire a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs;

[0213] Vector determination module 904 is configured to obtain an intra-word association corresponding to the training word segment based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of a character segment containing the target character in the training word segment in the second corpus; obtain a word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of a subword in the training word segment in the second corpus; a subword is a subsegment segmented from the training word segment; and determine a word segment vector corresponding to the training word segment based on at least the intra-word association and the word semantic independence corresponding to the training word segment.

[0214] The model training module 906 is used to train the initial word recognition model based on the word segment vectors and training word labels corresponding to the training word segments to obtain a target word recognition model.

[0215] The above-mentioned word recognition model training device obtains the internal word correlation corresponding to the training word segment by counting the relevant information of the characters in the training word segment. The internal word correlation can reflect the degree of correlation between the characters in the word segment. The greater the correlation, the more likely the training word segment is a word. The word semantic independence corresponding to the training word segment is obtained by counting the relevant information of the finer-grained sub-segments in the training word segment. The word semantic independence can reflect the semantic completeness of the word segment. The more complete the semantics, the more likely the training word segment is a word. Based on the internal word correlation and word semantic independence corresponding to the training word segment, a word segment vector corresponding to the training word segment is established. The word segment vector contains information such as the internal word correlation and word semantic independence that are helpful for word recognition. The word segment vector corresponding to the training word segment and the training label are used as training data to train the word recognition model, which can enable the word recognition model to learn the knowledge related to the word in the training data to recognize the word. The accuracy and efficiency of word recognition can be improved by performing word recognition through the trained word recognition model.

[0216] Each module in the above-mentioned word recognition device and word recognition model training device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor of the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0217] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in Figure 10. The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as a corpus and a word recognition model. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a word recognition method and a word recognition model training method are implemented.

[0218] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in Figure 11. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a word recognition method and a word recognition model training method are implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0219] Those skilled in the art will understand that the structures shown in Figures 10 and 11 are merely block diagrams of partial structures related to the solution of the present application, and do not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figures, or combine certain components, or have a different component arrangement.

[0220] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0221] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0222] In one embodiment, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0223] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0224] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0225] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0226] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A word recognition method, comprising: Acquire a target word segment and a first corpus to which the target word segment belongs; Based on the number of occurrences of a target character in the target word segment in the first corpus and the number of occurrences of a character segment containing the target character in the target word segment in the first corpus, obtaining an internal word association corresponding to the target word segment; Based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of a subword in the target word segment in the first corpus, obtaining the word semantic independence corresponding to the target word segment; the subword is a subsegment segmented from the target word segment; Determining a word fragment vector corresponding to the target word fragment based at least on the word internal association and word semantic independence corresponding to the target word fragment; The target word segment is subjected to word recognition based on the word segment vector to obtain a word recognition result corresponding to the target word segment.

2. The method according to claim 1, wherein: The step of obtaining a target word segment and a first corpus to which the target word segment belongs includes: Acquire a first corpus; each corpus in the first corpus belongs to the same field; Based on a preset segmentation length set, segment each corpus in the first corpus set to obtain a candidate word segment set that matches each segmentation length in the preset segmentation length set; each candidate word segment in the same candidate word segment set has the same segment length; Determine the target word segment from each set of candidate word segments.

3. The method according to claim 1 or 2, wherein: The obtaining the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus includes: Taking each character in the target word segment as a target character in turn; For any target character, the target character and the target character's adjacent backward character in the target word segment are combined into a character segment containing the target character in the target word segment; the adjacent backward character is a character adjacent to the target character in the target word segment and located after the target character; Obtaining the occurrence ratios of each character segment in the target word segment according to the occurrence ratios of the character segment containing the target character in the target word segment in the first corpus and the occurrence ratios of the target character in the target word segment in the first corpus; Smoothing the occurrence ratios corresponding to the character segments to obtain the internal correlations of the words corresponding to the character segments; The internal word association degree corresponding to the target word segment is obtained according to the internal word association degrees corresponding to the respective character segments.

4. The method according to any one of claims 1 to 3, wherein: The obtaining the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of subwords in the target word segment in the first corpus includes: Segmenting the target word segment to obtain subwords contained in the target word segment; Obtaining a fused occurrence count according to the occurrence counts of each subword of the target word segment in the first corpus; Based on the total number of subwords corresponding to the target word segment, adjusting the number of occurrences of the target word segment in the first corpus to obtain a reference number of occurrences; determining a difference in occurrence count between the fused occurrence count and the reference occurrence count; Based on the number of occurrences of the target word segment in the first corpus and the difference in the number of occurrences, the word semantic independence corresponding to the target word segment is obtained.

5. The method according to any one of claims 1 to 4, wherein: The determining the word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment includes: Acquire a candidate word fragment set to which the target word fragment belongs; each candidate word fragment in the candidate word fragment set has the same fragment length, and the candidate word fragment set is obtained by segmenting the corpus in the first corpus set; Determine at least one reference word segment corresponding to the target word segment from the candidate word segment set; Determine a target distance from the edit distances between the target word segment and at least one corresponding reference word segment, and determine the word text independence corresponding to the target word segment based on the target distance; The word fragment vector corresponding to the target word fragment is determined based at least on the word internal association, word semantic independence and word text independence corresponding to the target word fragment.

6. The method according to any one of claims 1 to 4, wherein: The determining the word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment includes: Obtain supplementary feature sets corresponding to the target word fragments respectively; the supplementary feature sets include at least one of the word external association degree, part-of-speech value, corpus importance, and fragment length corresponding to the target word fragment; the word external association degree is obtained according to the number of adjacent forward deduplicated characters and the number of adjacent backward deduplicated characters corresponding to the target word fragment in the first corpus, the number of adjacent forward deduplicated characters is obtained by performing character type statistics on adjacent forward characters corresponding to the target word fragment in the first corpus, the adjacent forward characters are characters adjacent to the target word fragment in the first corpus and located before the target word fragment, and the number of adjacent backward deduplicated characters is a number of characters corresponding to the target word fragment in the first corpus. The word importance is obtained by performing character type statistics on the corresponding adjacent backward characters in the first corpus, wherein the adjacent backward characters are characters in the first corpus that are adjacent to the target word segment and located after the target word segment; the part-of-speech value is obtained by counting the part-of-speech reflected by the target word segment in the first corpus; the corpus importance is obtained according to a first frequency and a second frequency corresponding to the target word segment in the first corpus, wherein the first frequency is the frequency of occurrence of the target word segment in the corpus containing the target word segment, and the corpus containing the target word segment is located in the first corpus, and the second frequency is the frequency of occurrence of the corpus containing the target word segment in the first corpus; The word fragment vector corresponding to the target word fragment is determined based at least on the word internal association, word semantic independence and a supplementary feature set corresponding to the target word fragment.

7. The method according to any one of claims 1 to 6, wherein: The performing word recognition on the target word segment based on the word segment vector to obtain a word recognition result corresponding to the target word segment includes: The word fragment vector is input into a target word recognition model, and a predicted word label corresponding to the target word fragment is output; and a word recognition result corresponding to the target word fragment is determined based on the predicted word label corresponding to the target word fragment; the predicted word label includes any one of a first label for indicating that the word fragment is a non-word, a second label for indicating that the word fragment is a word that has been included in a standard vocabulary, and a third label for indicating that the word fragment is a word that has not been included in the standard vocabulary.

8. The method according to any one of claims 1 to 7, wherein: The method further comprises: Based on the first target word segment and the second target word segment corresponding to the first corpus, a domain dictionary corresponding to the domain to which the first corpus belongs is determined; the predicted word label corresponding to the first target word segment is a label used to indicate that the word segment is a word that has been included in the standard vocabulary, and the predicted word label corresponding to the second target word segment is a label used to indicate that the word segment is a word that has not been included in the standard vocabulary.

9. A word recognition model training method, comprising: Obtaining a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs; Based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus, the word corresponding to the training word segment is obtained. Internal relevance; Based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of a subword in the training word segment in the second corpus, obtaining the word semantic independence corresponding to the training word segment; the subword is a subsegment segmented from the training word segment; Determining a word segment vector corresponding to the training word segment based at least on the word internal association and word semantic independence corresponding to the training word segment; Based on the word segment vectors and training word labels corresponding to the training word segments, the initial word recognition model is trained to obtain a target word recognition model.

10. A word recognition device, comprising: An acquisition module, used for acquiring a target word segment and a first corpus to which the target word segment belongs; A determination module, configured to obtain the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus; Based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of a subword in the target word segment in the first corpus, obtaining the word semantic independence corresponding to the target word segment; the subword is a subsegment segmented from the target word segment; based on at least the word internal association and word semantic independence corresponding to the target word segment, determining the word segment vector corresponding to the target word segment; The recognition module is used to perform word recognition on the target word segment based on the word segment vector to obtain a word recognition result corresponding to the target word segment.

11. A word recognition model training device, comprising: A data acquisition module, used to acquire a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs; a vector determination module, configured to obtain the internal word association corresponding to the training word segment based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus; Based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of subwords in the training word segment in the second corpus, obtaining the word semantic independence corresponding to the training word segment; the subword is a subsegment segmented from the training word segment; based on at least the word internal association and word semantic independence corresponding to the training word segment, determining the word segment vector corresponding to the training word segment; The model training module is used to train the initial word recognition model based on the word segment vectors and training word labels corresponding to the training word segments to obtain a target word recognition model.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

14. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images