New word recognition method, new word recognition model training method and device

Abstract

The present application relates to a new word recognition method, a new word recognition model training method, an apparatus, a computer device, a storage medium, and a computer program product. The method comprises: obtaining the internal word association corresponding to the target word segment based on the number of occurrences of a target character in a target word segment in a first corpus to which the target word segment belongs and the number of occurrences of a character segment containing the target character in the target word segment in the first corpus; obtaining the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of a subword in the target word segment in the first corpus; a subword is a word segmented from the target word segment; determining a word segment vector corresponding to the target word segment based on the word internal association and word semantic independence corresponding to the target word segment; and obtaining a new word recognition result corresponding to the target word segment based on the word segment vector. This method can improve the accuracy of new word recognition.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a new word recognition method, a new word recognition model training method, an apparatus, a computer device, a storage medium, and a computer program product. Background Art

[0002] With the development of computer technology, natural language processing (NLP) has emerged. NLP is a discipline that integrates linguistics, computer science, and mathematics. New word discovery is a crucial step in NLP, where new words are discovered by analyzing large amounts of corpus.

[0003] Traditional techniques for identifying new words are often based on specific rules. Linguistic experts construct specific templates based on morphological principles to identify new words. However, using specific templates to identify new words has limitations and cannot guarantee accurate new word recognition. Summary of the Invention

[0004] Based on this, it is necessary to provide a new word recognition method, new word recognition model training method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of new word recognition in response to the above technical problems.

[0005] This application provides a new word recognition method. The method includes:

[0006] Obtaining a target word segment and a first corpus to which the target word segment belongs;

[0007] Obtaining an internal word association corresponding to the target word segment based on the number of occurrences of a target character in the target word segment in the first corpus and the number of occurrences of a character segment containing the target character in the target word segment in the first corpus;

[0008] Obtaining word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of subwords in the target word segment in the first corpus; the subwords are words segmented from the target word segment;

[0009] Determining a word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment;

[0010] New word recognition is performed on the target word segment based on the word segment vector to obtain a new word recognition result corresponding to the target word segment; a new word refers to a word that is not recorded in the standard vocabulary.

[0011] The present application also provides a new word recognition device. The device includes:

[0012] An acquisition module, configured to acquire a target word segment and a first corpus to which the target word segment belongs;

[0013] a determination module configured to obtain an internal word association corresponding to the target word segment based on the number of occurrences of a target character in the target word segment in the first corpus and the number of occurrences of a character segment containing the target character in the target word segment in the first corpus; obtain a word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of a subword in the target word segment in the first corpus; the subword being a word segmented from the target word segment; and determine a word segment vector corresponding to the target word segment based at least on the internal word association and the word semantic independence corresponding to the target word segment;

[0014] The recognition module is used to perform new word recognition on the target word segment based on the word segment vector to obtain a new word recognition result corresponding to the target word segment; a new word refers to a word that is not recorded in the standard vocabulary.

[0015] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned new word recognition method when executing the computer program.

[0016] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned new word recognition method.

[0017] A computer program product includes a computer program, wherein when the computer program is executed by a processor, the steps of the new word recognition method are implemented.

[0018] This application provides a new word recognition model training method. The method includes:

[0019] Obtaining a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs;

[0020] Obtaining an internal word association corresponding to the training word segment based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus;

[0021] Obtaining word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of subwords in the training word segment in the second corpus; the subwords are words segmented from the training word segment;

[0022] Determining a word segment vector corresponding to the training word segment based at least on the word internal association and word semantic independence corresponding to the training word segment;

[0023] Based on the word segment vectors and training word labels corresponding to the training word segments, the initial new word recognition model is trained to obtain a target new word recognition model; new words refer to words that are not recorded in the standard vocabulary.

[0024] This application also provides a new word recognition model training device. The device includes:

[0025] A data acquisition module, configured to acquire a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs;

[0026] a vector determination module for obtaining an internal word association corresponding to the training word segment based on the number of occurrences of a target character in the training word segment in the second corpus and the number of occurrences of a character segment containing the target character in the training word segment in the second corpus; obtaining a word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of a subword in the training word segment in the second corpus; the subword being a word segmented from the training word segment; and determining a word segment vector corresponding to the training word segment based at least on the internal word association and the word semantic independence corresponding to the training word segment;

[0027] The model training module is used to train the initial new word recognition model based on the word segment vectors and training word labels corresponding to the training word segments to obtain a target new word recognition model; new words refer to words that are not recorded in the standard vocabulary.

[0028] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps described in the above-mentioned new word recognition model training method when executing the computer program.

[0029] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above-mentioned new word recognition model training method.

[0030] A computer program product includes a computer program, which implements the steps of the above-mentioned new word recognition model training method when executed by a processor.

[0031] The above-described new word identification method, new word identification model training method, apparatus, computer device, storage medium, and computer program product obtain the intra-word correlation corresponding to a word segment by collecting relevant information about the characters in the word segment. The intra-word correlation reflects the degree of correlation between the characters in the word segment; the greater the correlation, the more likely the word segment is a new word. The word semantic independence corresponding to the word segment is obtained by collecting relevant information about more fine-grained words in the word segment. The word semantic independence reflects the semantic completeness of the word segment; the more complete the semantics, the more likely the word segment is a new word. Based on the intra-word correlation and word semantic independence corresponding to the word segment, a word segment vector corresponding to the word segment is determined. The word segment vector contains information such as the intra-word correlation and word semantic independence that facilitate new word identification. The word segment vector corresponding to the target word segment is used to identify whether the target word segment is a new word, effectively improving the accuracy of new word identification. Using the word segment vectors and training labels corresponding to the training word segments as training data to train the new word identification model allows the new word identification model to learn knowledge related to new words in the training data to identify new words. Using the trained new word identification model to perform new word identification improves the accuracy and efficiency of new word identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments described in this specification. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0033] Figure 1 A diagram illustrating an application environment of a new word recognition method and a new word recognition model training method in one embodiment;

[0034] Figure 2 1 is a flow chart of a new word recognition method according to an embodiment;

[0035] Figure 3 A schematic diagram of a process flow of a new word recognition interface in one embodiment;

[0036] Figure 4 A schematic diagram of a flow chart of a new word recognition model training method in one embodiment;

[0037] Figure 5 A schematic diagram of training a new word recognition model in one embodiment;

[0038] Figure 6 is a flow chart of a new word recognition method according to another embodiment;

[0039] Figure 7A schematic diagram of an application scenario of the method of the present application in an embodiment;

[0040] Figure 8 is a structural block diagram of a new word recognition device in one embodiment;

[0041] Figure 9 is a structural block diagram of a new word recognition model training device in one embodiment;

[0042] Figure 10 is a diagram of the internal structure of a computer device in one embodiment;

[0043] Figure 11 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0045] The new word recognition method and new word recognition model training method provided in the embodiments of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other servers. Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers or a cloud server.

[0046] Both the terminal and the server can be used independently to execute the new word recognition method and the new word recognition model training method provided in the embodiments of the present application.

[0047] For example, the terminal obtains a target word segment and the first corpus to which the target word segment belongs, obtains the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus, obtains the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of the subwords in the target word segment in the first corpus, where the subwords are words segmented from the target word segment. The terminal determines the word segment vector corresponding to the target word segment based on at least the internal word association and the word semantic independence corresponding to the target word segment, performs new word recognition on the target word segment based on the word segment vector corresponding to the target word segment, and obtains a new word recognition result corresponding to the target word segment.

[0048] For example, the server obtains a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs. The server obtains the internal word association corresponding to the training word segment based on the number of occurrences of the target character in the training word segment and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus, and obtains the word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of subwords in the training word segment in the second corpus, where the subwords are words segmented from the training word segment. The server determines the word segment vector corresponding to the training word segment based on at least the internal word association and word semantic independence corresponding to the training word segment, and trains the initial new word recognition model based on the word segment vector and the training word label corresponding to the training word segment to obtain a target new word recognition model.

[0049] The terminal and the server can also be used in collaboration to execute the new word recognition method and the new word recognition model training method provided in the embodiments of the present application.

[0050] For example, the server obtains the target word segment and the first corpus to which the target word segment belongs from the terminal. The server obtains the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus, and obtains the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of the subwords in the target word segment in the first corpus, where the subwords are words segmented from the target word segment. The server determines the word segment vector corresponding to the target word segment based on at least the internal word association and the word semantic independence corresponding to the target word segment, performs new word recognition on the target word segment based on the word segment vector corresponding to the target word segment, and obtains the new word recognition result corresponding to the target word segment. The server returns the new word recognition result corresponding to the target word segment to the terminal.

[0051] For example, the server obtains a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs from the terminal. The server obtains the internal word association corresponding to the training word segment based on the difference between the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus, and obtains the word semantic independence corresponding to the training word segment based on the difference between the number of occurrences of the training word segment in the second corpus and the number of occurrences of the subword in the training word segment in the second corpus, where the subword is a word segmented from the target word segment. The server determines the word segment vector corresponding to the training word segment based on at least the internal word association and word semantic independence corresponding to the training word segment, and trains the initial new word recognition model based on the word segment vector and the training word label corresponding to the training word segment to obtain a target new word recognition model. Subsequently, the server can help the terminal perform new word recognition based on the target new word recognition model. The server can send the target new word recognition model to the terminal so that the terminal performs new word recognition based on the target new word recognition model.

[0052] New words are words not included in standard lexicons. Standard lexicons refer to standard and commonly used dictionaries. With the development of the internet and the widespread use of various software applications, new words are constantly appearing online. With this constant flow of new words online, timely identification of new words in corpora is crucial. Traditional techniques typically mine new words from corpora based on specific templates. However, these templates are limited in scope and have limitations, making accurate new word recognition impossible.

[0053] The new word recognition method provided in this application obtains a target word fragment and a first corpus to which the target word fragment belongs, calculates various statistical features corresponding to the target word fragment based on the first corpus, for example, the statistical features include word internal association, word semantic independence, and word text independence, determines a word fragment vector corresponding to the target word fragment based on the various statistical features corresponding to the target word fragment, and the word fragment vector contains various statistical features that are helpful for new word recognition. Based on the word fragment vector corresponding to the target word fragment, it is identified whether the target word fragment is a new word, which can effectively improve the accuracy of new word recognition. The new word recognition method provided in this application is applicable to new word recognition in various scenarios.

[0054] In one embodiment, Figure 2 As shown, a new word recognition method is provided, and the method is applied to a computer device as an example. The computer device can be a terminal or a server. Figure 2 ,The new word recognition method includes the following steps:

[0055] Step S202: Obtain a target word segment and a first corpus to which the target word segment belongs.

[0056] The target word segment refers to a word segment to be identified as a new word. It is understood that the target word segment includes at least two characters. For example, the target word segment can be a segment including two characters, a segment including three characters, a segment including four characters, a segment including five characters, a segment including six characters, etc.

[0057] The corpus includes multiple corpora, that is, the corpus includes multiple texts. Based on the corpora in the corpus, statistical features corresponding to word segments in the corpora are calculated to identify whether the word segment is a new word. The first corpus to which the target word segment belongs includes at least one corpus containing the target word segment. That is, the target word segment appears in at least one corpus in the first corpus to which the target word segment belongs.

[0058] Specifically, the computer device can obtain the target word fragment and the first corpus to which the target word fragment belongs locally or from other devices, calculate at least two statistical features corresponding to the target word fragment based on the corpus in the first corpus, and identify whether the target word fragment is a new word based on the at least two statistical features corresponding to the target word fragment.

[0059] In one embodiment, a first corpus may be obtained first, and then the target word segment may be determined from the first corpus. For example, multiple corpora belonging to the same field may be obtained to form the first corpus, and each corpus in the first corpus may be segmented to obtain multiple candidate word segments, and the target word segment may be determined from each candidate word segment. It is understood that if the first corpus is obtained first, each candidate word segment obtained by segmenting each corpus in the first corpus may be used as a target word segment, and all new words appearing in the first corpus may be identified, thereby mining a large number of new words at one time.

[0060] In one embodiment, the target word fragment may be obtained first, and then the first corpus to which the target word fragment belongs may be obtained. For example, a word fragment of interest is obtained as the target word fragment, the corpus containing the target word fragment is collected, and the first corpus is composed based on the collected corpus. It can be understood that the first corpus may be composed of corpus containing the target word fragment, or may be composed of corpus containing the target word fragment and corpus not containing the target word fragment. In a specific embodiment, the user can input the word fragment of interest as the target word fragment on the relevant interface of the terminal, thereby triggering new word recognition for the target word fragment, and obtaining the new word recognition result corresponding to the target word fragment. For example, referring to Figure 3, on the new word recognition interface, the user enters the word fragment of interest in the input box of the word fragment to be recognized, and the recognition control is triggered after the user completes the input. After the user triggers the recognition control, the terminal can determine the new word recognition result locally, or send a new word recognition request carrying the target word fragment to the server, and the server determines the new word recognition result, and obtains the new word recognition result from the server. Taking the terminal determining the new word recognition result locally as an example, the terminal automatically obtains the first corpus to which the target word fragment belongs from the database, calculates at least two statistical features corresponding to the target word fragment based on the corpus in the first corpus to identify whether the target word fragment is a new word, and obtains the new word recognition result corresponding to the target word fragment. Reference Figure 3 After the terminal obtains the new word recognition result, it can display the new word recognition result on the interface so that the user can promptly obtain the new word recognition result. It is understood that in addition to entering the word segment of interest as the target word segment, the user can also upload the first corpus to which the target word segment belongs. If the user triggers new word recognition for the target word segment, the new word recognition result corresponding to the target word segment entered by the user is determined based on the first corpus uploaded by the user.

[0061] In one embodiment, each corpus in the first corpus to which the target word segment belongs belongs to the same field. For example, in the medical field, each corpus in the first corpus to which the target word segment belongs is text related to the medical field. In the financial field, each corpus in the first corpus to which the target word segment belongs is text related to the financial field. In this way, the fact that each corpus in the first corpus to which the target word segment belongs belongs to the same field facilitates the discovery of new words in specific fields and scenarios.

[0062] Step S204 : obtaining the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus.

[0063] The target character in the target word segment can be any character in the target word segment. For example, if the target word segment is a segment consisting of four characters, each of the four characters in the target word segment can be used as the target character, or at least one character can be selected from the four characters in the target word segment as the target character. It is understood that there can be at least one target character in the target word segment.

[0064] A character segment includes at least two characters. A character segment containing a target character in a target word segment is a character segment composed of the characters contained in the target word segment, and the character segment contains the target character. For example, if the target word segment is "labor fruit" and the target character is "labor," then the character segment containing the target character in the target word segment can be "labor." It is understood that for a target character in a target word segment, there can be at least one character segment containing the target character in the target word segment.

[0065] The intra-word association degree for a word fragment is calculated based on the number of occurrences of the target character in the word fragment within the corpus to which the word fragment belongs, as well as the number of occurrences of the character segment containing the target character within the word fragment within the corpus to which the word fragment belongs. The intra-word association degree for a word fragment reflects the degree of connection, closeness, and stability between the characters within the word fragment. The greater the intra-word association degree for a word fragment, the more stable the characters within the word fragment, and the greater the likelihood that the word fragment will become a word.

[0066] Specifically, the computer device determines a target character and a character segment containing the target character from the target word segment, counts the number of occurrences of the target character in the first corpus, counts the number of occurrences of the character segment containing the target character in the first corpus, and obtains the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus. For example, the number of occurrences of the character segment containing the target character in the target word segment in the first corpus is used as the first number, and the number of occurrences of the target character in the target word segment in the first corpus is used as the second number. The internal word association can be obtained based on the ratio between the first number and the second number; the first number and the second number can be adjusted based on a preset weight, and the internal word association can be obtained based on the difference or ratio between the adjusted first number and the second number; and so on.

[0067] Step S206 : obtaining the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of subwords in the target word segment in the first corpus.

[0068] The subwords in the target word segment are words segmented from the target word segment, that is, the subwords in the target word segment refer to the words contained in the target word segment. Segmenting the target word segment can obtain the subwords contained in the target word segment. It is understood that there can be at least one subword in the target word segment. If the target word segment cannot be further split into fine-grained subwords, the word semantic independence can be a preset default value, for example, zero or a negative number.

[0069] The word semantic independence corresponding to a word fragment is calculated based on the number of occurrences of the word fragment in the corpus to which the word fragment belongs, and the number of occurrences of the subwords in the word fragment in the corpus to which the word fragment belongs. The word semantic independence corresponding to a word fragment can reflect the degree of semantic completeness of the word fragment. The greater the word semantic independence corresponding to the word fragment, the more complete the semantics of the word fragment, and the greater the possibility that the word fragment is a new word. It can be understood that the closer the number of occurrences of the word fragment in the corpus to which the word fragment belongs and the number of occurrences of the subwords in the word fragment in the corpus to which the word fragment belongs, the greater the possibility that the word fragment is a word, and the closer the semantic completeness of the word fragment and the semantic completeness of the subwords in the word fragment.

[0070] Specifically, the computer device separates fine-grained subwords from the target word segment, counts the number of occurrences of the target word segment in the first corpus, counts the number of occurrences of the subwords in the target word segment in the first corpus, and obtains the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of the subwords in the target word segment in the first corpus. For example, the number of occurrences of the target word segment in the first corpus is used as the first count, and the number of occurrences of the subwords in the target word segment in the first corpus is used as the second count. The word semantic independence can be obtained based on the difference between the first count and the second count; the difference between the first count and the second count is used as the first difference, and the word semantic independence can be obtained based on the ratio of the first count and the first difference; and so on.

[0071] Step S208 : determining a word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment.

[0072] The word fragment vector is a feature vector corresponding to the word fragment, which is used to reflect the feature information of the word fragment.

[0073] Specifically, the computer device may establish a word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment. For example, the word segment vector may be obtained based on the word internal association and word semantic independence; or the word segment vector may be obtained based on the word internal association, word semantic independence, and other statistical features.

[0074] Step S210 , performing new word recognition on the target word segment based on the word segment vector to obtain a new word recognition result corresponding to the target word segment.

[0075] New word recognition refers to identifying whether a word segment is new. New words are words not included in a standard vocabulary. A standard vocabulary can be a standard dictionary developed by a professional department or professionals, or a commonly used dictionary.

[0076] Specifically, the word segment vector corresponding to the target word segment contains rich information from different perspectives, such as the internal word association and word semantic independence corresponding to the target word segment. The computer device performs new word recognition on the target word segment based on the word segment vector corresponding to the target word segment, and can obtain an accurate new word recognition result corresponding to the target word segment. For example, the word segment vector corresponding to the target word segment includes the internal word association and word semantic independence corresponding to the target word segment. The weighted sum of the internal word association and word semantic independence is performed to obtain the new word probability corresponding to the target word segment. If the new word probability corresponding to the target word segment is greater than a preset probability, the new word recognition result corresponding to the target word segment is determined to be that the target word segment is a new word. If the new word probability corresponding to the target word segment is less than or equal to the preset probability, the new word recognition result corresponding to the target word segment is determined to be that the target word segment is not a new word. For another example, the word fragment vector corresponding to the target word fragment is input into the new word recognition model, and the new word recognition model outputs the predicted word label corresponding to the target word fragment. If the predicted word label is a label indicating that the word fragment belongs to a new word, then the new word recognition result corresponding to the target word fragment is determined to be that the target word fragment is a new word. If the predicted word label is other labels, then the new word recognition result corresponding to the target word fragment is determined to be that the target word fragment is not a new word.

[0077] In the above-mentioned new word identification method, a target word segment and a first corpus to which the target word segment belongs are obtained. Based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus, the internal word association corresponding to the target word segment is obtained. Based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of the subwords in the target word segment in the first corpus, the word semantic independence corresponding to the target word segment is obtained. The subwords are words segmented from the target word segment. Based on at least the internal word association and the word semantic independence corresponding to the target word segment, a word segment vector corresponding to the target word segment is determined. New word identification is performed on the target word segment based on the word segment vector to obtain a new word identification result corresponding to the target word segment. In this way, the internal word association corresponding to the target word segment is obtained by statistically analyzing the relevant information of the characters in the target word segment. The internal word association can reflect the degree of association between the characters in the word segment. The greater the degree of association, the greater the possibility that the target word segment is a new word. By collecting statistics on the finer-grained information about the words in the target word fragment, the word semantic independence corresponding to the target word fragment is obtained. Word semantic independence reflects the semantic completeness of the word fragment; the more complete the semantics, the greater the likelihood that the target word fragment is a new word. The word fragment vector corresponding to the target word fragment is determined based on the word internal association and word semantic independence corresponding to the target word fragment. The word fragment vector contains information such as word internal association and word semantic independence, which can help identify new words. Identifying whether the target word fragment is a new word based on the word fragment vector corresponding to the target word fragment can effectively improve the accuracy of new word recognition. In addition, the computational complexity of word internal association and word semantic independence is not high, which also helps improve the efficiency of new word recognition.

[0078] In one embodiment, obtaining a target word segment and a first corpus to which the target word segment belongs includes:

[0079] A first corpus is obtained; each corpus in the first corpus belongs to the same field; based on a preset segmentation length set, each corpus in the first corpus is segmented to obtain a candidate word segment set that matches each segmentation length in the preset segmentation length set; each candidate word segment in the same candidate word segment set has the same segment length; and a target word segment is determined from each candidate word segment set.

[0080] The preset segmentation length set includes at least one preset segmentation length. The segmentation length is used to determine the segment length of a word segment segmented from the corpus. The segment length corresponding to a word segment refers to the number of characters contained in the word segment. For example, if the preset segmentation length is 3, then the word segment obtained by segmenting the corpus based on the segmentation length is a segment containing 3 characters, and the segment length of the word segment is 3.

[0081] Each corpus in the first corpus is segmented based on a segmentation length to obtain a candidate word fragment set, and the fragment length of each candidate word fragment in the candidate word fragment set is the segmentation length. For example, if the preset segmentation length is 5, each corpus in the first corpus is segmented based on the segmentation length, and the segments obtained by segmentation, each of which includes 5 characters, are combined into a candidate word fragment set.

[0082] Specifically, the computer device may first obtain a first corpus consisting of various corpora in the same field, then segment each corpus in the first corpus, and determine the target word segment from the segmentation results.

[0083] The computer device segments each corpus in the first corpus based on a preset segmentation length set to obtain a set of candidate word fragments that matches each segmentation length in the preset segmentation length set. For any segmentation length in the preset segmentation length set, the corpus in the first corpus is segmented using a sliding window whose window length matches the segmentation length. The sliding window is slid across the corpus character by character, and each slide captures a character string. The captured character string is used as a candidate word fragment, and the candidate word fragments are combined into a set of candidate word fragments.

[0084] The computer device ultimately determines a target word segment from the candidate word segment sets that match each segment length in the preset segment length set. For example, the candidate word segments in each candidate word segment set are respectively used as target word segments; and at least one candidate word segment is selected from each candidate word segment set as the target word segment.

[0085] In the above embodiment, a first corpus consisting of various corpora in the same field is first obtained, and then target word segments are segmented out from the first corpus. The first corpus includes various corpora belonging to the same field. By performing new word recognition on the target word segments segmented out from the first corpus, it is helpful to discover new words in a certain field.

[0086] In one embodiment, obtaining the intra-word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus includes:

[0087] Each character in the target word segment is taken as the target character in turn; for any target character, the target character and its adjacent backward characters in the target word segment are combined into a character segment containing the target character in the target word segment; according to the appearance ratio between the number of appearances of the character segment containing the target character in the target word segment in the first corpus and the number of appearances of the target character in the target word segment in the first corpus, the appearance ratio corresponding to each character segment in the target word segment is obtained respectively; the appearance ratio corresponding to each character segment is smoothed to obtain the internal word association corresponding to each character segment; according to the internal word association corresponding to each character segment, the internal word association corresponding to the target word segment is obtained.

[0088] The target character's adjacent backward character in the target word segment refers to the character that is adjacent to the target character and located after the target character in the target word segment. In other words, the target character's adjacent backward character in the target word segment refers to the character that appears immediately after the target character in the target word segment. For example, if the target word segment is "Labor Fruit", and the target character is "Labor", then the target character's adjacent backward character in the target word segment is "Dynamic"; if the target character is "Fruit", then the target character's adjacent backward character in the target word segment is "Solid".

[0089] The target character and its adjacent backward characters in the target word segment are combined to form a character segment containing the target character in the target word segment. That is, the character segment containing the target character in the target word segment includes the target character and its adjacent backward characters in the target word segment. For example, if the target word segment is "Labor Fruit", if the target character is "Labor", then the character segment containing it is "Labor"; if the target character is "Fruit", then the character segment containing it is "Fruit".

[0090] It can be understood that for the last character in the target word segment, if there is no adjacent backward character in the target word segment, then there is no corresponding character segment for the character.

[0091] Data smoothing is performed to make the data smooth and stable for subsequent processing. Smoothing can be performed by compressing the data values, reducing the range of the data. For example, smoothing can be performed using a logarithmic function, a custom formula, or other methods.

[0092] Specifically, when calculating the internal word association corresponding to the target word fragment, the internal word association corresponding to each character fragment in the target word fragment can be calculated, and then the internal word association corresponding to each character fragment can be finally obtained based on the internal word association corresponding to each character fragment.

[0093] The computer device can sequentially use each character in the target word segment as a target character, exhausting the characters in the target word segment and fully utilizing the characters in the target word segment. For any target character, the target character and its adjacent subsequent characters in the target word segment are combined to form a character segment containing the target character in the target word segment, thereby obtaining character segments corresponding to each target character. For any target character, the number of occurrences of the character segment containing the target character in the target word segment in the first corpus is counted, and the number of occurrences of the target character in the target word segment in the first corpus is counted. The ratio between these two occurrences is calculated to obtain the occurrence ratio corresponding to the character segment. The target word segment includes at least two characters, each of which has a corresponding character segment. The occurrence ratio corresponding to each character segment in the target word segment can be calculated. Furthermore, to stabilize the occurrence ratio data for subsequent processing, the occurrence ratio corresponding to each character segment can be smoothed to obtain the intra-word association corresponding to each character segment. Finally, the computer device obtains the intra-word association corresponding to the target word segment based on the intra-word association corresponding to each character segment in the target word segment. For example, the internal word association degree corresponding to each character segment in the target word segment is averaged to obtain the internal word association degree corresponding to the target word segment; the weighted average of the internal word association degree corresponding to each character segment in the target word segment is calculated to obtain the internal word association degree corresponding to the target word segment, and the weight corresponding to the internal word association degree corresponding to each character segment can be set as needed; and so on.

[0094] For example, the target word segment is "labor fruit", the character segment corresponding to the target character "labor" is "labor", the character segment corresponding to the target character "movement" is "movement fruit", and the character segment corresponding to the target character "fruit" is "fruit". Based on the ratio between the number of occurrences of "labor" in the first corpus and the number of occurrences of "labor" in the first corpus, the number of occurrences corresponding to "labor" is obtained. Based on the ratio between the number of occurrences of "movement fruit" in the first corpus and the number of occurrences of "movement" in the first corpus, the number of occurrences corresponding to "movement fruit" is obtained. Based on the ratio between the number of occurrences of "fruit" in the first corpus and the number of occurrences of "fruit", the number of occurrences corresponding to "fruit" is obtained. The number of occurrences corresponding to "labor", "movement fruit", and "fruit" are smoothed respectively to obtain the internal word associations corresponding to "labor", "movement fruit", and "fruit". Based on the internal word associations corresponding to "labor", "movement fruit", and "fruit", the internal word association corresponding to "labor fruit" is obtained.

[0095] In one embodiment, the calculation formula for the word internal relevance T corresponding to the word fragment is:

[0096]

[0097] Where l represents the length of the word segment, that is, the number of characters that the word segment consists of. i Indicates the i-th character of the word fragment; T i+1 Indicates the i+1th character of the word fragment; p(T i ) represents the number of times the i-th character of a word fragment appears in the corpus to which the word fragment belongs; p(T i |T i+1 ) represents the number of times the combination of the i-th character and the i+1-th character of the word fragment appears in the corpus to which the word fragment belongs. It can be understood that T i is the target character in the word fragment, p(T i ) is the number of times the target character in the word fragment appears in the corpus to which the word fragment belongs. i and T i+1 The character segments that make up the word segments contain the target characters, p(T i |T i+1 ) is the number of times the character segment containing the target character in the word segment appears in the corpus to which the word segment belongs. It is the ratio of the number of occurrences of the corresponding character segments. log represents smoothing. The internal word association degree corresponding to the character fragment. T is the internal word association degree corresponding to the word fragment. The greater the internal word association degree corresponding to the word fragment, the more stable the internal word association and the greater the possibility of the word forming a word. The internal word association degree corresponding to the word fragment can also be called the average internal word association.

[0098] In the above embodiment, each character in the target word segment is taken as the target character in turn, and the target character and the adjacent backward characters of the target character in the target word segment are combined into a character segment containing the target character in the target word segment. The internal word association corresponding to each character segment in the target word segment is calculated respectively, and then the internal word association corresponding to the target word segment is obtained based on the internal word association corresponding to each character segment in the target word segment. This makes the internal word association corresponding to the target word segment integrate the statistical information between each group of adjacent characters in the target word segment, can fully reflect the internal association of the word segment, and has high accuracy.

[0099] In one embodiment, the word semantic independence corresponding to the target word segment is obtained based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of subwords in the target word segment in the first corpus, including:

[0100] The target word fragment is segmented to obtain the subwords contained in the target word fragment; the fused occurrence count is obtained according to the number of occurrences of each subword of the target word fragment in the first corpus; the occurrence count of the target word fragment in the first corpus is adjusted based on the total number of subwords corresponding to the target word fragment to obtain the reference occurrence count; the difference in occurrence count between the fused occurrence count and the reference occurrence count is determined; the semantic independence of the word corresponding to the target word fragment is obtained based on the number of occurrences of the target word fragment in the first corpus and the difference in occurrence count.

[0101] The total number of subwords corresponding to the target word segment refers to the number of subwords contained in the target subsegment. For example, if the target word segment is "labor fruit," "labor fruit" is segmented to obtain "labor" and "fruit." "Labor" and "fruit" are the two subwords contained in "labor fruit," and the total number of subwords corresponding to "labor fruit" is 2.

[0102] Specifically, the target word segment can be composed of fine-grained words. When calculating the semantic independence of the words corresponding to the target word segment, the fine-grained words in the target word segment can be exhausted, and all subwords contained in the target word segment can be fully utilized. The computer device performs word segmentation on the target word segment to obtain the subwords contained in the target word segment, counts the number of occurrences of each subword contained in the target word segment in the first corpus, and obtains the fused number of occurrences based on the number of occurrences of each subword contained in the target word segment in the first corpus. For example, the number of occurrences of each subword contained in the target word segment in the first corpus can be summed to obtain the fused number of occurrences; the number of occurrences of each subword contained in the target word segment in the first corpus can be weighted and summed to obtain the fused number of occurrences, and the weights corresponding to the number of occurrences of each subword can be set as needed; and so on. Then, the number of occurrences of the target word segment in the first corpus can be adjusted based on the number of subwords corresponding to the target word segment to obtain the reference number of occurrences. For example, the number of occurrences of the target word segment in the first corpus is magnified based on the number of subwords corresponding to the target word segment; the reference number of occurrences obtained by adjustment is proportional to the number of subwords used for adjustment; and so on. Further, the difference between the reference number of occurrences and the fused number of occurrences can be calculated to obtain the number of occurrences difference. For example, the difference between the reference number of occurrences and the fused number of occurrences is calculated, and the difference is used as the number of occurrences difference; the difference is adjusted based on a preset value to obtain the number of occurrences difference. Finally, the word semantic independence corresponding to the target word segment can be obtained based on the number of occurrences of the target word segment in the first corpus and the number of occurrences difference. For example, the ratio of the number of occurrences of the target word segment in the first corpus to the number of occurrences difference is used as the word semantic independence; the difference between the number of occurrences of the target word segment in the first corpus and the number of occurrences difference is used as the word semantic independence.

[0103] It is understandable that the target word segment can be segmented using an existing segmentation tool to obtain the subwords contained in the target word segment. For example, the jieba (jieba) segmentation tool can be used to segment the target word segment.

[0104] In one embodiment, the calculation formula for the word semantic independence W corresponding to the word fragment is:

[0105]

[0106] Where w represents a word fragment; p(w) represents the number of times the word fragment w appears in the corpus; N represents that the word fragment w can be composed of N fine-grained words; w j represents the jth word among N fine-grained words; p(w j ) represents the word w j The number of times the word fragment w appears in the corpus to which it belongs. It can be understood that N is the total number of subwords corresponding to the word fragment, p(w) is the number of times the word fragment appears in the corpus to which the word fragment belongs, and w j That is, the jth subword in the word fragment, p(w j ) is the number of occurrences of the j-th subword in the word fragment in the corpus to which the word fragment belongs, is the number of fusion occurrences, (N-1)p(w) is the number of reference occurrences, This is the difference in the number of occurrences. The greater the word semantic independence W corresponding to a word fragment, the more stable the word fragment, the more complete its semantics, and the greater the probability that it is a new word. For a given word fragment, the numerator in the calculation formula is fixed. The fewer fine-grained words the word fragment is broken down into, the smaller the denominator in the calculation formula, the larger the result, and the greater the word semantic independence of the word fragment. The more fine-grained words the word fragment is broken down into, the larger the denominator in the calculation formula, the lower the word semantic independence of the word fragment, and the lower the probability that it is a new word. For example, the word fragment W is "the fruits of labor," which can be divided into two more fine-grained words, "labor" and "fruit." Taken individually, "labor" and "fruit" do not have a specific meaning and are less likely to be a new word. However, when combined into "the fruits of labor," they have the specific meaning of "the result obtained through a certain action," making them more likely to be a new word. The word semantic independence corresponding to a word fragment can also be called the word semantic independence value.

[0107] In the above embodiment, the target word segment is segmented to obtain the subwords contained in the target word segment. The fused occurrence count is obtained based on the number of occurrences of each subword of the target word segment in the first corpus. The number of occurrences of the target word segment in the first corpus is adjusted based on the total number of subwords corresponding to the target word segment to obtain a reference occurrence count. The difference in the number of occurrences between the fused occurrence count and the reference occurrence count is determined. The word semantic independence corresponding to the target word segment is obtained based on the number of occurrences of the target word segment in the first corpus and the difference in the number of occurrences. In this way, when calculating the word semantic independence, the statistical information of all subwords contained in the reference word segment is referenced, and the combined value of the statistical information of all subwords is compared with the statistical information of the word segment. This allows the word semantic independence to fully reflect the differences between the word segment and the subwords within the word segment, fully reflect the semantic integrity of the word segment, and has a high degree of accuracy.

[0108] In one embodiment, determining a word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment includes:

[0109] Obtain a set of candidate word fragments to which the target word fragment belongs; the fragment lengths of the candidate word fragments in the candidate word fragment set are consistent, and the candidate word fragment set is obtained by segmenting the corpus in the first corpus set; determine at least one reference word fragment corresponding to the target word fragment from the candidate word fragment set; determine a target distance from the edit distances between the target word fragment and the corresponding at least one reference word fragment, and determine the word text independence corresponding to the target word fragment based on the target distance; determine a word fragment vector corresponding to the target word fragment based at least on the word internal association, word semantic independence and word text independence corresponding to the target word fragment.

[0110] The edit distance between two word segments is the minimum number of edit operations required to transform one segment into the other. The larger the edit distance between two segments, the more different they are. Permitted edit operations include replacing one character with another, inserting a character, and deleting a character. The edit distance is also called the Levenshtein distance or edit distance.

[0111] The reference word segment corresponding to the target word segment refers to a word segment whose segment length is consistent with the segment length of the target word segment, and the reference word segment corresponding to the target word segment is also consistent with the corpus to which the target word segment belongs. The reference word segment corresponding to the target word segment is determined from the set of candidate word segments to which the target word segment belongs. The set of candidate word segments to which the target word segment belongs includes at least one candidate word segment contained in the first corpus to which the target word segment belongs, whose segment length is consistent with the segment length of the target word segment. It can be understood that the target word segment and the corresponding reference word segment are different word segments.

[0112] The word-text independence of a word fragment is calculated based on the edit distance between each word fragment and at least one corresponding reference word fragment. The smaller the edit distance between a word fragment and its corresponding reference word fragment, the more independent the word fragment is. The greater the word-text independence of the word fragment, the greater the probability that it is a new word.

[0113] Specifically, in addition to identifying new words based on the internal word association and word semantic independence corresponding to the target word fragment, we can further introduce the word text independence corresponding to the target word fragment, and identify new words based on the internal word association, word semantic independence and word text independence corresponding to the target word fragment.

[0114] The computer device can obtain a set of candidate word segments to which the target word segment belongs, and determine at least one reference word segment corresponding to the target word segment from the set of candidate word segments to which the target word segment belongs. For example, each candidate word segment in the set of candidate word segments to which the target word segment belongs can be used as a reference word segment corresponding to the target word segment; a candidate word segment that appears more than a preset number of times in the first corpus can be selected from the set of candidate word segments to which the target word segment belongs as a reference word segment corresponding to the target word segment, with candidate word segments that appear more than a preset number of times in the first corpus being more representative; and so on. Furthermore, the computer device determines a target distance from the edit distances between the target word segment and the corresponding at least one reference word segment, and determines the word text independence corresponding to the target word segment based on the target distance. For example, the minimum value from each edit distance is obtained as the target distance, and the target distance is used as the word text independence; the average value of each edit distance is calculated as the target distance, and the target distance is used as the word text independence; and so on.

[0115] Ultimately, the computer device can determine the word fragment vector corresponding to the target word fragment based at least on the internal word association, word semantic independence and word text independence corresponding to the target word fragment, so that the word fragment vector includes information from at least three perspectives, thereby improving the accuracy of new word recognition.

[0116] In one embodiment, the calculation formula for the word text independence S corresponding to the word segment is:

[0117] S=min{D word,k =D(word,k),k Q}

[0118] Among them, D word,k represents the edit distance between word fragment word and word fragment k, word fragment word is the word fragment whose corresponding word text independence needs to be calculated, and word fragment k is the reference word fragment corresponding to word fragment word; D(word,k) represents the edit distance calculation formula; Q is a set of word fragments other than word fragment word (for example, word fragment word is a fragment consisting of 3 characters, then Q is a set of word fragments, and the word fragments in Q are all fragments consisting of 3 characters, and Q does not include word fragment word); min{} represents taking the minimum value of all D(word,k). It can be understood that word fragment word is the target word fragment, word fragment k is the reference word fragment corresponding to the target word fragment, and the set composed of Q and word fragment word is the candidate word fragment set to which the target word fragment belongs. min{D word,k = D(word,k), k∈Q} is the target distance. New words typically possess a certain degree of novelty and independence. The larger the minimum value of S, the higher the word-text independence of the word fragment and the greater the probability of being a new word. The word-text independence corresponding to a word fragment can also be called the word-text independence value.

[0119] In the above embodiment, the target word segment and the corresponding reference word segment have the same segment length and belong to the same corpus. The target distance is determined from the edit distance between the target word segment and at least one corresponding reference word segment. The word text independence corresponding to the target word segment is determined based on the target distance. This allows the word text independence to reflect the textual differences between the target word segment and the reference word segment, embodying the independence of the target word segment with high accuracy. A word segment vector corresponding to the target word segment is established based on at least the word internal association, word semantic independence, and word text independence corresponding to the target word segment. This provides the word segment vector with richer information, thereby helping to improve the accuracy of new word recognition.

[0120] In one embodiment, determining a word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment includes:

[0121] Obtaining supplementary feature sets corresponding to the target word fragments respectively; determining the word fragment vectors corresponding to the target word fragments based at least on the word internal association, word semantic independence and supplementary feature sets corresponding to the target word fragments.

[0122] The supplementary feature set includes at least one of the word external association, part-of-speech value, corpus importance, and segment length corresponding to the target word segment.

[0123] Perform character type statistics on the adjacent forward characters corresponding to a word segment in the corpus to which the word segment belongs, and obtain the number of adjacent forward deduplicated characters corresponding to the word segment in the corpus to which the word segment belongs. The number of adjacent forward deduplicated characters corresponding to a word segment in its corpus refers to the number of character types of the single characters that are directly adjacent to the word segment and are in front of the word segment in the corpus to which the word segment belongs. That is, the number of adjacent forward deduplicated characters corresponding to a word segment in its corpus is used to represent how many different characters the characters on the left side of the word segment appear in the corpus to which the word segment belongs. Perform character type statistics on the adjacent backward characters corresponding to a word segment in the corpus to which the word segment belongs, and obtain the number of adjacent backward deduplicated characters corresponding to the word segment in the corpus to which the word segment belongs. The number of adjacent backward deduplicated characters corresponding to a word segment in its corpus refers to the number of character types of the single characters that are directly adjacent to the word segment and are behind the word segment in the corpus to which the word segment belongs. That is, the number of adjacent backward deduplicated characters corresponding to a word segment in its corpus is used to represent how many different characters the characters on the right side of the word segment appear in the corpus to which the word segment belongs. It can be understood that the adjacent forward characters corresponding to a word segment in the corpus to which the word segment belongs refer to the characters that are adjacent to the word segment and are before the word segment in the corpus to which the word segment belongs. The adjacent backward characters corresponding to a word segment in the corpus to which the word segment belongs refer to the characters that are adjacent to the word segment and are after the word segment in the corpus to which the word segment belongs. For example, if the target word segment is "weather", and one corpus in the first corpus that includes the target word segment is "It's sunny today", then "day" is an adjacent forward character corresponding to the target word segment in the first corpus, and "sunny" is an adjacent backward character corresponding to the target word segment in the first corpus.

[0124] The external association degree of a word segment is calculated based on the number of adjacent forward deduplicated characters and the number of adjacent backward deduplicated characters corresponding to the word segment in the corpus to which the word segment belongs. For example, the sum of the number of adjacent forward deduplicated characters and the number of adjacent backward deduplicated characters is used to obtain the external association degree of the word; the weighted sum of the number of adjacent forward deduplicated characters and the number of adjacent backward deduplicated characters is used to obtain the external association degree of the word, and the weights corresponding to the number of adjacent forward deduplicated characters and the number of adjacent backward deduplicated characters can be set as needed; and so on. Further, for example, if the target word segment is "peace", and there are 50 different characters that appear on the left side of "peace" in the first corpus, and 30 different characters that appear on the right side of "peace" in the first corpus, then the external association degree of the target word segment "peace" is 80. The greater the external association degree of a word segment, the richer the surrounding characters of the word segment, and the greater the possibility of the word segment forming a word.

[0125] The part-of-speech value corresponding to a word segment is obtained by counting the parts of speech exhibited by the word segment in the corresponding corpus. The part-of-speech value is used to indicate how many parts of speech the target word segment has in the first corpus. For example, if the target word segment exhibits both noun and adjective parts of speech in the first corpus, the part-of-speech value corresponding to the target word segment is 2. Generally speaking, the lower the part-of-speech value corresponding to a word segment, the more stable the word segment is, and the greater the probability that the word segment is a word. In one embodiment, the part-of-speech value corresponding to the word segment can be counted using a tool with a part-of-speech analysis function.

[0126] The corpus importance corresponding to a word fragment is calculated based on the first frequency and second frequency corresponding to the word fragment in the corpus to which the word fragment belongs. The first frequency refers to the frequency of occurrence of the word fragment in the corpus containing the word fragment, and the corpus containing the word fragment is in the corpus to which the word fragment belongs. The second frequency refers to the frequency of occurrence of the corpus containing the word fragment in the corpus to which the word fragment belongs. For example, the external relevance of the word is obtained by multiplying the first frequency and the second frequency. In one embodiment, the corpus importance is a TF-IDF (term frequency–inverse document frequency) value.

[0127] Specifically, in addition to identifying new words based on the internal word association and word semantic independence corresponding to the target word fragment, we can further introduce a supplementary feature set corresponding to the target word fragment, and identify new words based on the internal word association, word semantic independence and supplementary feature set corresponding to the target word fragment.

[0128] The computer device obtains the supplementary feature sets corresponding to the target word fragments respectively, and determines the word fragment vector corresponding to the target word fragment based at least on the internal word association, word semantic independence, and supplementary feature sets corresponding to the target word fragments, so that the word fragment vector includes information from multiple perspectives, thereby improving the accuracy of new word recognition.

[0129] In the above embodiment, a word fragment vector corresponding to the target word fragment is established based at least on the internal word association, word semantic independence and supplementary feature set corresponding to the target word fragment, so that the word fragment vector has richer information, which helps to improve the accuracy of new word recognition.

[0130] In one embodiment, a word segment vector corresponding to the target word segment is established based on at least the word internal association, word semantic independence, word text independence and a supplementary feature set corresponding to the target word segment.

[0131] In one embodiment, determining a word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment includes:

[0132] At least the internal word association and the word semantic independence corresponding to the target word fragment are normalized, and the normalized internal word association and word semantic independence corresponding to the target word fragment are concatenated to obtain a word fragment vector corresponding to the target word fragment.

[0133] Normalization is used to map the data values ​​to a preset range for subsequent data processing. For example, the value of the internal association of words can be mapped to a range between 0 and 1, and the value of the semantic independence of words can be mapped to a range between 0 and 1.

[0134] Specifically, when determining the word fragment vector corresponding to the target word fragment based at least on the internal word association and word semantic independence corresponding to the target word fragment, the internal word association corresponding to the target word fragment can be normalized first, and the word semantic independence corresponding to the target word fragment can be normalized, and then the normalized internal word association and word semantic independence can be spliced ​​to obtain the word fragment vector corresponding to the target word fragment.

[0135] It can be understood that the statistical features used to establish the word fragment vector corresponding to the word fragment include the internal correlation of the word, the semantic independence of the word, the text independence of the word and at least two statistical features in the supplementary feature set. When determining the word fragment vector corresponding to the word fragment based on the statistical features corresponding to the word fragment, the various statistical features can be normalized first, and then the various statistical features after normalization can be spliced ​​to obtain the word fragment vector corresponding to the word fragment.

[0136] In the above embodiment, normalizing various statistical features and then concatenating them to obtain word segment vectors can effectively avoid the weight of a single statistical feature being too large or too small, thereby affecting the accuracy of subsequent new word recognition.

[0137] In one embodiment, new word recognition is performed on the target word segment based on the word segment vector to obtain a new word recognition result corresponding to the target word segment, including:

[0138] The word fragment vector is input into the target new word recognition model, and the predicted word label corresponding to the target word fragment is output. The new word recognition result corresponding to the target word fragment is determined based on the predicted word label corresponding to the target word fragment; the predicted word label includes any one of a first label used to indicate that the word fragment belongs to a non-word, a second label used to indicate that the word fragment belongs to a word but not a new word, and a third label used to indicate that the word fragment belongs to a new word.

[0139] Among them, the new word recognition model is a machine learning model used to recognize new words. The input data of the new word recognition model is the word segment vector corresponding to the word segment, and the output data is the predicted word label corresponding to the word segment. The new word recognition model can output three types of predicted word labels, namely, a first label used to indicate that the word segment is a non-word, a second label used to indicate that the word segment is a word but not a new word, and a third label used to indicate that the word segment is a new word. If the predicted word label corresponding to the word segment is the first label, it means that the word segment is not a word. If the predicted word label corresponding to the word segment is the second label, it means that the word segment is a word but not a new word. If the predicted word label corresponding to the word segment is the third label, it means that the word segment is a new word. The target new word recognition model refers to a new word recognition model that has completed training. The word segment vector corresponding to the word segment is input into the target new word recognition model. The target new word recognition model processes the input data and selects one label from the three labels according to the data processing results.

[0140] Specifically, the computer device can use a machine learning model to perform new word recognition. The computer device can obtain a target new word recognition model, input the word segment vector corresponding to the target word segment into the target new word recognition model, and the target new word recognition model processes the input data to obtain a predicted word label corresponding to the target word segment. The target new word recognition model outputs the predicted word label corresponding to the target word segment. The predicted word label corresponding to the target word segment includes any one of a first label for indicating that the word segment is a non-word, a second label for indicating that the word segment is a word but not a new word, and a third label for indicating that the word segment is a new word. The predicted word label corresponding to the target word segment can be used as the new word recognition result corresponding to the target word segment.

[0141] It can be understood that the machine learning algorithm used to establish the new word recognition model can be selected according to actual needs. For example, a new word recognition model is established based on the LR (Logistic Regression) algorithm, a new word recognition model is established based on the deepFM algorithm, and a new word recognition model is established based on the XGboost (Extreme Gradient Boosting) algorithm.

[0142] In the above embodiment, the word segment vector corresponding to the target word segment is input into the target new word recognition model for new word recognition. The target new word recognition model has a strong learning ability and can accurately recognize new words based on the knowledge it has learned, effectively ensuring the accuracy of new word recognition. In addition, inputting the word segment vector corresponding to the target word segment into the target new word recognition model for new word recognition can also ensure the efficiency of new word recognition.

[0143] In one embodiment, the new word identification method further includes:

[0144] Based on the first target word segment and the second target word segment corresponding to the first corpus, a domain dictionary corresponding to the domain to which the first corpus belongs is determined.

[0145] The predicted word label corresponding to the first target word segment is a label indicating that the word segment belongs to a word but is not a new word, and the predicted word label corresponding to the second target word segment is a label indicating that the word segment belongs to a new word.

[0146] Specifically, a plurality of corpora belonging to the same field are obtained to form a first corpus, and words and new words are identified from the first corpus to establish a domain dictionary. A computer device segments the corpus in the first corpus to obtain a plurality of candidate word segments, each of which is used as a target word segment. New word recognition is performed on the target word segment by establishing a word segment vector corresponding to the target word segment, and new word recognition results corresponding to each target word segment corresponding to the first corpus are obtained. From these, first target word segments whose new word recognition results indicate that the word segment is a new word and second target word segments whose new word recognition results indicate that the word segment is a word but not a new word are obtained to form a domain dictionary corresponding to the domain to which the first corpus belongs.

[0147] In one embodiment, the computer device may build a knowledge graph corresponding to the domain to which the first corpus belongs based on the domain dictionary. For example, during the construction of the knowledge graph, entities need to be identified. Entity recognition can be performed based on the terms in the domain dictionary to identify richer and more comprehensive entities, thereby improving the quality of the knowledge graph.

[0148] In the above embodiment, the words and new words identified by the new word identification method of the present application are applied to the construction of a domain dictionary, which can enrich the content of the domain dictionary.

[0149] In one embodiment, Figure 4 As shown, a new word recognition model training method is provided, and the method is applied to a computer device as an example. The computer device can be a terminal or a server. Figure 4 ,The new word recognition model training method includes the following steps:

[0150] Step S402: Obtain a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs.

[0151] A training word segment refers to a word segment used in training data for generating a model. A training word segment is a word segment for which a new word recognition result is known. The training word tag corresponding to the training word segment is used to indicate the correct new word recognition result corresponding to the training word segment. The second corpus to which the training word segment belongs includes at least one corpus containing the training word segment. That is, the training word segment appears in at least one corpus within the second corpus to which the training word segment belongs.

[0152] Specifically, the computer device can obtain relevant data of training word segments to establish training data for the new word recognition model, train the new word recognition model based on the training data, and obtain a machine learning model for new word recognition.

[0153] Step S404 : obtaining the internal word association corresponding to the training word segment based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus.

[0154] Step S406 : Obtain the word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of subwords in the training word segment in the second corpus.

[0155] Step S408 : determining a word segment vector corresponding to the training word segment based at least on the word internal association and word semantic independence corresponding to the training word segment.

[0156] It can be understood that the calculation process of the internal word association corresponding to the training word fragment can refer to the calculation process of the internal word association corresponding to the aforementioned target word fragment, the calculation process of the word semantic independence corresponding to the training word fragment can refer to the calculation process of the word semantic independence corresponding to the aforementioned target word fragment, and the determination process of the word fragment vector corresponding to the training word fragment can refer to the establishment process of the word fragment vector corresponding to the aforementioned target word fragment, which will not be repeated here.

[0157] Step S410 : Training the initial new word recognition model based on the word segment vectors and training word labels corresponding to the training word segments to obtain a target new word recognition model.

[0158] The initial new word recognition model refers to the new word recognition model to be trained, and the target new word recognition model refers to the trained new word recognition model.

[0159] Specifically, referring to the process of establishing the word segment vector corresponding to the target word segment, a word segment vector corresponding to the training word segment is established based on the training word segment and the second corpus to which the training word segment belongs. The word segment vector and the training word label corresponding to the training word segment are used as training data for the model. During model training, the word segment vector corresponding to the training word segment is used as input data for the model, and the training word label corresponding to the training word segment is used as expected output data for the model. The computer device trains the initial new word recognition model based on the word segment vector and the training word label corresponding to the training word segment to obtain a target new word recognition model.

[0160] For example, reference Figure 5, the statistical features corresponding to the training word fragments include part-of-speech value, word external association, word internal association, word TF-IDF value, word length value, word semantic independence, and word text independence. The various statistical features corresponding to the training word fragments are normalized respectively, and the various statistical features corresponding to the training word fragments after normalization constitute the word fragment vector corresponding to the training word fragment. The word fragment vector corresponding to the training word fragment is input into the new word recognition model to be trained. The new word recognition model processes the input data and outputs the predicted word label corresponding to the training word fragment. The model loss is generated based on the difference between the predicted word label corresponding to the training word fragment and the training word label. The model loss is back-propagated to adjust the parameters of the new word recognition model. Through iterative training, the parameters of the model are continuously adjusted until the training end condition is met, and a trained new word recognition model is obtained. The training end condition can be at least one condition indicating that the model training is completed, such as the model loss is less than the preset loss value, the number of model iterations is greater than the preset number of iterations, etc.

[0161] It can be understood that the training data of the model may include word segment vectors and training word labels of multiple training word segments. The model is trained with a large amount of training data so that the model can learn relevant knowledge about the new word recognition task.

[0162] In one embodiment, different new word recognition models can be trained for different fields. For example, for the medical field, a second corpus consisting of medical texts is obtained, the corpus in the second corpus is segmented to obtain multiple training word segments, word segment vectors corresponding to the training word segments are established based on the second corpus, and a new word recognition model for new word recognition in the medical field is trained based on the word segment vectors and training word labels corresponding to a large number of training word segments. For the game field, a second corpus consisting of game texts is obtained, the corpus in the second corpus is segmented to obtain multiple training word segments, word segment vectors corresponding to the training word segments are established based on the second corpus, and a new word recognition model for new word recognition in the game field is trained based on the word segment vectors and training word labels corresponding to a large number of training word segments.

[0163] In one embodiment, the first corpus is segmented based on a preset first segmentation length set to determine target word segments, and the second corpus is segmented based on a preset second segmentation length set to determine training word segments. In order to ensure the efficiency of model training, the second segmentation length set has a segmentation length restriction. During the training process of the model, the model has gradually learned the relevant knowledge of the new word recognition task. During the application of the model, the first segmentation length set may not have a segmentation length restriction, and the model can perform new word recognition on word segments of any segment length, thereby ensuring the versatility of the method of the present application.

[0164] The above-mentioned new word recognition model training method obtains a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs. Based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus, the internal word association corresponding to the training word segment is obtained. Based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of the subwords in the training word segment in the second corpus, the word semantic independence corresponding to the training word segment is obtained. Based on at least the internal word association and the word semantic independence corresponding to the training word segment, a word segment vector corresponding to the training word segment is determined. Based on the word segment vector and the training word label corresponding to the training word segment, an initial new word recognition model is trained to obtain a target new word recognition model. In this way, the internal word association corresponding to the training word segment is obtained by statistically analyzing the relevant information of the characters in the training word segment. The internal word association can reflect the degree of association between the characters in the word segment. The greater the degree of association, the greater the possibility that the training word segment is a new word. By collecting statistics on the more fine-grained word information in the training word fragments, the word semantic independence of the training word fragments is obtained. Word semantic independence can reflect the semantic completeness of the word fragments. The more complete the semantics, the greater the possibility that the training word fragment is a new word. The word fragment vector corresponding to the training word fragment is determined based on the internal word association and word semantic independence of the training word fragments. The word fragment vector contains information such as internal word association and word semantic independence that helps to identify new words. The word fragment vector and training label corresponding to the training word fragment are used as training data to train the new word recognition model. This allows the new word recognition model to learn knowledge related to new words in the training data to recognize new words. Using the trained new word recognition model to perform new word recognition can improve the accuracy and efficiency of new word recognition.

[0165] In a specific embodiment, the method of the present application can be applied to the recognition of new words in a specific field. Figure 6 As shown, the new word recognition method of the present application includes the following steps:

[0166] 1. Domain Corpus Acquisition

[0167] The computer device obtains multiple corpora belonging to the same field to form a domain corpus set A, and the domain corpus set A is used for model training.

[0168] 2. Domain Corpus Segmentation

[0169] The computer device segments the domain corpus A using n-grams to obtain n-words, which are the training word segments. The value of n is generally less than or equal to 4. For example, the domain corpus A is segmented using 2-grams, 3-grams, and 4-grams, respectively, to obtain segments consisting of 2 characters, 3 characters, and 4 characters. It is understood that segments consisting of 1 character are less likely to be new words, so it is not necessary to obtain segments consisting of 1 character.

[0170] 3. Feature construction

[0171] Referring to Table 1, for any training word segment, the computer device calculates various statistical features corresponding to the training word segment, including part-of-speech value, word external association, word internal association, word TF-IDF value, word length value (i.e., segment length), word semantic independence, and word text independence.

[0172] Table 1

[0173]

[0174] The part-of-speech value refers to the number of parts of speech that a word fragment has in the domain corpus A. Generally, the lower the part-of-speech value of a word fragment, the more stable the word fragment is and the greater the possibility that the word fragment will become a word.

[0175] The external relevance of a word refers to the number of different expressions that appear on the left and right sides of a word fragment in the domain corpus A. The greater the external relevance of a word fragment, the richer the surrounding words of the word fragment, and the greater the possibility that the word fragment will become a word.

[0176] The calculation formula for word internal correlation is: The greater the internal correlation of a word fragment, the more stable the internal correlation is, and the greater the possibility that the word fragment will become a word.

[0177] The word TF-IDF value is used to measure the importance of a word fragment in the domain corpus A. Generally, new words have more important meanings. The larger the word TF-IDF value of a word fragment, the more likely it is a new word.

[0178] The word length value indicates the number of characters in the word. Because word fragments of different lengths have different distributions of internal word association, generally, the longer the word fragment, the greater the internal word association. Including the word length value feature can fully leverage the positive impact of internal word association on the model. Including the word length value feature adds more prior knowledge, allowing the model to learn more useful information and understand what kind of internal word association a word of a certain length should have. Word fragments that meet this pattern are more likely to be real words, avoiding model misjudgments. Regarding word semantic independence, the greater the word semantic independence of a word fragment, the more stable the word fragment, the more complete the semantics, and the greater the likelihood that the word fragment is a new word.

[0179] The calculation formula of word semantic independence is: The greater the semantic independence of a word fragment, the more stable the word fragment, the more complete the semantics, and the greater the possibility that it is a new word.

[0180] The calculation formula of word text independence value is S=min{D word, k = D(word, k), k∈Q}. New words usually have a certain degree of novelty and independence. The larger the word text independence value of a word fragment, the higher the text independence of the word fragment, and the greater the probability that the word fragment is a new word.

[0181] 4. Normalization

[0182] The computer equipment normalizes the various features mentioned above to avoid the weight of a single feature being too large or too small, which affects the recognition accuracy of the model.

[0183] 5. Model training

[0184] As shown in Table 1, the "lable" column is the training word label corresponding to the segment of n characters. "0" represents that the training word segment is not a word, "1" represents that the training word segment is a word but not a new word, and "2" represents that the training word segment is a word and a new word. The computer device reads the normalized data from the second to eighth columns in Table 1, converts them into an array format, and uses them as the feature value array X; then reads the value of the ninth column in Table 1, converts them into an array format, and uses them as the label array Y. The data in the label array Y corresponds one-to-one to the data in the feature value array X, and the corresponding data are the data corresponding to the same training word segment. The feature value array X is split into a training set X_train and a test set X_test, and the label array Y is split into a training set Y_train and a test set Y_test. The model is trained based on the training sets X_train and Y_train. After the training, the model is tested based on the test sets X_test and Y_test. The model that passes the test is regarded as the final qualified model.

[0185] 6. Model prediction

[0186] A new corpus from the same domain as the corpus in domain corpus A is selected to form domain corpus B. The corpus in domain corpus B is segmented using n-grams to obtain n-character segments. Each n-character segment is used as a word segment to be identified. The statistical features corresponding to the word segments to be identified include part-of-speech value, word external association, word internal association, word TF-IDF value, word length (i.e., segment length), word semantic independence, and word text independence. The feature value corresponding to each word segment to be identified is calculated and input into a trained model to obtain a predicted word label for each word segment to be identified. During model prediction, the value of n can be greater than 4, so that the model can recognize words of different lengths or new words.

[0187] The new word recognition method of the present application can accurately identify whether a word fragment is a term or a new word with high accuracy. Furthermore, there is no limit on the length of the word fragment during model prediction, making it widely applicable. Furthermore, the calculation method for various statistical features is not complex, and the use of the model for new word recognition helps ensure the efficiency of new word recognition.

[0188] In a specific embodiment, Figure 7 As shown, the new word recognition method of the present application can be applied to construct a domain dictionary for a specific field. For example, by using the new word recognition method of the present application to perform new word recognition, new words of different lengths in the medical field can be identified, and the identified new words can be added to the existing domain dictionary in the medical field to enrich the content of the domain dictionary. For example, by using the new word recognition method of the present application to perform new word recognition, words and new words of different lengths in the financial field can be identified, and the identified words and new words can be combined to form a domain dictionary for word segmentation of financial field corpus.

[0189] Further, such as Figure 7As shown, the domain dictionary constructed by the new word recognition method of the present application can be applied to construct a knowledge graph in a specific field, and the constructed knowledge graph can be further applied to build various types of answering dialogue robots in a specific field. For example, in the construction process of various types of answering dialogue robots in the financial field (such as bank sales robots, bank consulting robots, and bank operation robots), it is necessary to build a complete knowledge graph to improve the service quality of the answering dialogue robot. The knowledge graph determines the comprehensiveness of the knowledge reserve of the answering dialogue robot. In the construction process of the knowledge graph, large and comprehensive entity information is required. The domain dictionary in the financial field constructed by the new word recognition method of the present application can be used to assist the entity information in the financial field. For example, the knowledge graph can be a question-answer pair consisting of questions and answers. The establishment of questions and answers involves words. In the knowledge graph, it can be multiple questions corresponding to one answer, or it can be one question corresponding to one answer. The more complete the relevant knowledge graph is, the more the robot knows, and the better it can serve customers' questions or inquiries.

[0190] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0191] Based on the same inventive concept, the present application also provides a new word recognition device for implementing the new word recognition method described above. The solution provided by this device is similar to the solution described in the method described above. Therefore, the specific limitations in one or more new word recognition device embodiments provided below can be found in the above-mentioned limitations on the new word recognition method and will not be repeated here.

[0192] In one embodiment, Figure 8 As shown, a new word recognition device is provided, including: an acquisition module 802, a determination module 804 and a recognition module 806, wherein:

[0193] The acquisition module 802 is configured to acquire a target word segment and a first corpus to which the target word segment belongs.

[0194] Determination module 804 is used to obtain the internal word association corresponding to the target word fragment based on the number of occurrences of the target character in the target word fragment in the first corpus and the number of occurrences of the character fragment containing the target character in the target word fragment in the first corpus; obtain the word semantic independence corresponding to the target word fragment based on the number of occurrences of the target word fragment in the first corpus and the number of occurrences of subwords in the target word fragment in the first corpus; subwords are words segmented from the target word fragment; and determine the word fragment vector corresponding to the target word fragment based on at least the internal word association and word semantic independence corresponding to the target word fragment.

[0195] The recognition module 806 is configured to perform new word recognition on the target word segment based on the word segment vector to obtain a new word recognition result corresponding to the target word segment.

[0196] In one embodiment, the acquisition module 802 is further configured to:

[0197] A first corpus is obtained; each corpus in the first corpus belongs to the same field; based on a preset segmentation length set, each corpus in the first corpus is segmented to obtain a candidate word segment set that matches each segmentation length in the preset segmentation length set; each candidate word segment in the same candidate word segment set has the same segment length; and a target word segment is determined from each candidate word segment set.

[0198] In one embodiment, the determination module 804 is further configured to:

[0199] Each character in the target word segment is taken as the target character in turn; for any target character, the target character and its adjacent backward characters in the target word segment are combined into a character segment containing the target character in the target word segment; the adjacent backward characters are characters that are adjacent to the target character and located after the target character in the target word segment; according to the appearance ratio between the number of occurrences of the character segment containing the target character in the target word segment in the first corpus and the number of occurrences of the target character in the target word segment in the first corpus, the appearance ratio corresponding to each character segment in the target word segment is obtained respectively; the appearance ratio corresponding to each character segment is smoothed to obtain the internal word association corresponding to each character segment; according to the internal word association corresponding to each character segment, the internal word association corresponding to the target word segment is obtained.

[0200] In one embodiment, the determination module 804 is further configured to:

[0201] The target word fragment is segmented to obtain the subwords contained in the target word fragment; the fused occurrence count is obtained according to the number of occurrences of each subword of the target word fragment in the first corpus; the occurrence count of the target word fragment in the first corpus is adjusted based on the total number of subwords corresponding to the target word fragment to obtain the reference occurrence count; the difference in occurrence count between the fused occurrence count and the reference occurrence count is determined; the semantic independence of the word corresponding to the target word fragment is obtained based on the number of occurrences of the target word fragment in the first corpus and the difference in occurrence count.

[0202] In one embodiment, the determination module 804 is further configured to:

[0203] Obtain a set of candidate word fragments to which the target word fragment belongs; the fragment lengths of the candidate word fragments in the candidate word fragment set are consistent, and the candidate word fragment set is obtained by segmenting the corpus in the first corpus set; determine at least one reference word fragment corresponding to the target word fragment from the candidate word fragment set; determine a target distance from the edit distances between the target word fragment and the corresponding at least one reference word fragment, and determine the word text independence corresponding to the target word fragment based on the target distance; determine a word fragment vector corresponding to the target word fragment based at least on the word internal association, word semantic independence and word text independence corresponding to the target word fragment.

[0204] In one embodiment, the determination module 804 is further configured to:

[0205] Obtain the supplementary feature sets corresponding to the target word fragments respectively; the supplementary feature sets include at least one of the word external association degree, part-of-speech value, corpus importance, and fragment length corresponding to the target word fragment; the word external association degree is obtained based on the number of adjacent forward deduplicated characters and the number of adjacent backward deduplicated characters corresponding to the target word fragment in the first corpus, the number of adjacent forward deduplicated characters is obtained by performing character type statistics on the adjacent forward characters corresponding to the target word fragment in the first corpus, the adjacent forward characters are characters adjacent to the target word fragment in the first corpus and located before the target word fragment, and the number of adjacent backward deduplicated characters is obtained by performing character type statistics on the adjacent backward characters corresponding to the target word fragment in the first corpus. The adjacent backward characters are the characters adjacent to the target word fragment and located after the target word fragment in the first corpus; the part-of-speech value is obtained by counting the part of speech reflected by the target word fragment in the first corpus; the corpus importance is obtained according to the first frequency and second frequency corresponding to the target word fragment in the first corpus, the first frequency is the frequency of occurrence of the target word fragment in the corpus containing the target word fragment, and the corpus containing the target word fragment is in the first corpus, and the second frequency is the frequency of occurrence of the corpus containing the target word fragment in the first corpus; the word fragment vector corresponding to the target word fragment is determined based on at least the internal correlation degree, word semantic independence and supplementary feature set corresponding to the target word fragment.

[0206] In one embodiment, the identification module 806 is further configured to:

[0207] The word fragment vector is input into the target new word recognition model, and the predicted word label corresponding to the target word fragment is output. The new word recognition result corresponding to the target word fragment is determined based on the predicted word label corresponding to the target word fragment; the predicted word label includes any one of a first label used to indicate that the word fragment belongs to a non-word, a second label used to indicate that the word fragment belongs to a word but not a new word, and a third label used to indicate that the word fragment belongs to a new word.

[0208] In one embodiment, the new word identification device is further configured to:

[0209] Based on the first target word segment and the second target word segment corresponding to the first corpus, the domain dictionary corresponding to the domain to which the first corpus belongs is determined; the predicted word label corresponding to the first target word segment is a label used to indicate that the word segment belongs to a word but is not a new word, and the predicted word label corresponding to the second target word segment is a label used to indicate that the word segment belongs to a new word.

[0210] The above-mentioned new word identification device obtains the internal word association corresponding to the target word segment by counting the relevant information of the characters in the target word segment. The internal word association can reflect the degree of association between the characters in the word segment. The greater the degree of association, the more likely the target word segment is a new word. The word semantic independence corresponding to the target word segment is obtained by counting the relevant information of finer-grained words in the target word segment. The word semantic independence can reflect the semantic completeness of the word segment. The more complete the semantics, the more likely the target word segment is a new word. Based on the internal word association and word semantic independence corresponding to the target word segment, a word segment vector corresponding to the target word segment is established. The word segment vector contains information such as the internal word association and word semantic independence that are helpful for new word identification. Identifying whether the target word segment is a new word based on the word segment vector corresponding to the target word segment can effectively improve the accuracy of new word identification.

[0211] Based on the same inventive concept, the present application also provides a new word recognition model training device for implementing the new word recognition model training method described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more new word recognition model training device embodiments provided below can be found in the above limitations of the new word recognition model training method and will not be repeated here.

[0212] In one embodiment, Figure 9 As shown, a new word recognition model training device is provided, including: a data acquisition module 902, a vector determination module 904 and a model training module 906, wherein:

[0213] A data acquisition module 902 is configured to acquire a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs;

[0214] Vector determination module 904 is configured to obtain an intra-word association corresponding to the training word segment based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of a character segment containing the target character in the training word segment in the second corpus; obtain a word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of subwords in the training word segment in the second corpus; a subword is a word segmented from the training word segment; and determine a word segment vector corresponding to the training word segment based on at least the intra-word association and the word semantic independence corresponding to the training word segment.

[0215] The model training module 906 is used to train the initial new word recognition model based on the word segment vectors and training word labels corresponding to the training word segments to obtain a target new word recognition model.

[0216] The above-mentioned new word recognition model training device obtains the internal word correlation corresponding to the training word segment by counting the relevant information of the characters in the training word segment. The internal word correlation can reflect the degree of correlation between the characters in the word segment. The greater the correlation, the more likely the training word segment is a new word. The word semantic independence corresponding to the training word segment is obtained by counting the relevant information of more fine-grained words in the training word segment. The word semantic independence can reflect the semantic completeness of the word segment. The more complete the semantics, the more likely the training word segment is a new word. Based on the internal word correlation and word semantic independence corresponding to the training word segment, a word segment vector corresponding to the training word segment is established. The word segment vector contains information such as the internal word correlation and word semantic independence that are helpful for new word recognition. The word segment vector corresponding to the training word segment and the training label are used as training data to train the new word recognition model, which can enable the new word recognition model to learn the knowledge related to the new word in the training data to recognize the new word. The accuracy and efficiency of new word recognition can be improved by performing new word recognition through the trained new word recognition model.

[0217] Each module in the new word recognition device and new word recognition model training device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0218] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 10As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as a corpus and a new word recognition model. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a new word recognition method and a new word recognition model training method are implemented.

[0219] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 11 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a new word recognition method and a new word recognition model training method are implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0220] Those skilled in the art will understand that Figure 10 、 11The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0221] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0222] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0223] In one embodiment, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0224] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0225] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0226] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0227] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A new word recognition method, characterized in that: The method comprises: Obtaining a target word segment and a first corpus to which the target word segment belongs; Obtaining an internal word association corresponding to the target word segment based on the number of occurrences of a target character in the target word segment in the first corpus and the number of occurrences of a character segment containing the target character in the target word segment in the first corpus; Obtaining word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of subwords in the target word segment in the first corpus; the subwords are words segmented from the target word segment; Determining a word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment; New word recognition is performed on the target word segment based on the word segment vector to obtain a new word recognition result corresponding to the target word segment; a new word refers to a word that is not recorded in the standard vocabulary.

2. The method according to claim 1, characterized in that The obtaining of the target word segment and the first corpus to which the target word segment belongs includes: Obtain a first corpus; each corpus in the first corpus belongs to the same field; Based on a preset segmentation length set, segmenting each corpus in the first corpus to obtain a set of candidate word segments that respectively match each segmentation length in the preset segmentation length set; each candidate word segment in the same candidate word segment set has the same segment length; Determine the target word segment from each set of candidate word segments.

3. The method according to claim 1, characterized in that Obtaining the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus includes: Taking each character in the target word segment as a target character in turn; For any target character, the target character and its adjacent backward characters in the target word segment are combined to form a character segment containing the target character in the target word segment; the adjacent backward characters are characters that are adjacent to the target character and located after the target character in the target word segment; Obtaining the occurrence ratio of each character segment in the target word segment according to the occurrence ratio of the character segment containing the target character in the target word segment in the first corpus and the occurrence ratio of the target character in the target word segment in the first corpus; Smoothing the occurrence ratios of the character segments to obtain the internal word associations of the character segments; The internal word association degree corresponding to the target word segment is obtained according to the internal word association degrees corresponding to the respective character segments.

4. The method according to claim 1, wherein The obtaining the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of subwords in the target word segment in the first corpus includes: Segmenting the target word segment to obtain subwords contained in the target word segment; Obtaining a fused occurrence count based on the occurrence counts of each subword of the target word segment in the first corpus; Adjusting the number of occurrences of the target word segment in the first corpus based on the total number of subwords corresponding to the target word segment to obtain a reference number of occurrences; determining a difference in occurrence counts between the fused occurrence count and the reference occurrence count; Based on the number of occurrences of the target word segment in the first corpus and the difference in the number of occurrences, the word semantic independence corresponding to the target word segment is obtained.

5. The method according to claim 1, wherein The determining of the word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment includes: Obtaining a set of candidate word segments to which the target word segment belongs; wherein each candidate word segment in the set of candidate word segments has the same segment length, and the set of candidate word segments is obtained by segmenting the corpus in the first corpus; determining at least one reference word segment corresponding to the target word segment from the candidate word segment set; determining a target distance from the edit distances between the target word segments and at least one corresponding reference word segment, and determining the word text independence corresponding to the target word segment based on the target distance; The word segment vector corresponding to the target word segment is determined based at least on the word internal association, word semantic independence and word text independence corresponding to the target word segment.

6. The method according to claim 1, characterized in that The determining of the word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment includes: Obtain a supplementary feature set corresponding to each of the target word fragments; the supplementary feature set includes at least one of the word external association degree, part-of-speech value, corpus importance, and fragment length corresponding to the target word fragment; the word external association degree is obtained based on the number of adjacent forward deduplicated characters and the number of adjacent backward deduplicated characters corresponding to the target word fragment in the first corpus, the number of adjacent forward deduplicated characters is obtained by performing character type statistics on the adjacent forward characters corresponding to the target word fragment in the first corpus, the adjacent forward characters are characters adjacent to the target word fragment in the first corpus and located before the target word fragment, and the number of adjacent backward deduplicated characters is obtained by performing character type statistics on the target word fragment in the first corpus. The method comprises the following steps: performing character type statistics on corresponding adjacent backward characters in the first corpus, wherein the adjacent backward characters are characters in the first corpus that are adjacent to the target word segment and located after the target word segment; the part-of-speech value is obtained by counting the parts of speech reflected by the target word segment in the first corpus; and the corpus importance is obtained according to a first frequency and a second frequency corresponding to the target word segment in the first corpus, wherein the first frequency is the frequency of occurrence of the target word segment in the corpus containing the target word segment, and the corpus containing the target word segment is located in the first corpus, and the second frequency is the frequency of occurrence of the corpus containing the target word segment in the first corpus. The word fragment vector corresponding to the target word fragment is determined based at least on the word internal association, word semantic independence and supplementary feature set corresponding to the target word fragment.

7. The method according to claim 1, characterized in that The performing new word recognition on the target word segment based on the word segment vector to obtain a new word recognition result corresponding to the target word segment includes: The word fragment vector is input into the target new word recognition model, the predicted word label corresponding to the target word fragment is output, and the new word recognition result corresponding to the target word fragment is determined based on the predicted word label corresponding to the target word fragment; the predicted word label includes any one of a first label for indicating that the word fragment belongs to a non-word, a second label for indicating that the word fragment belongs to a word but not a new word, and a third label for indicating that the word fragment belongs to a new word.

8. The method according to claim 1, characterized in that The method further comprises: Based on the first target word segment and the second target word segment corresponding to the first corpus, the domain dictionary corresponding to the domain to which the first corpus belongs is determined; the predicted word label corresponding to the first target word segment is a label used to indicate that the word segment belongs to a word but is not a new word, and the predicted word label corresponding to the second target word segment is a label used to indicate that the word segment belongs to a new word.

9. A new word recognition model training method, characterized in that: The method comprises: Obtaining a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs; Obtaining an internal word association corresponding to the training word segment based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus; Obtaining word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of subwords in the training word segment in the second corpus; the subwords are words segmented from the training word segment; Determining a word segment vector corresponding to the training word segment based at least on the word internal association and word semantic independence corresponding to the training word segment; Based on the word segment vectors and training word labels corresponding to the training word segments, the initial new word recognition model is trained to obtain a target new word recognition model; new words refer to words that are not recorded in the standard vocabulary.

10. A new word recognition device, characterized in that: The device comprises: An acquisition module, configured to acquire a target word segment and a first corpus to which the target word segment belongs; a determination module configured to obtain an internal word association corresponding to the target word segment based on the number of occurrences of a target character in the target word segment in the first corpus and the number of occurrences of a character segment containing the target character in the target word segment in the first corpus; obtain a word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of a subword in the target word segment in the first corpus; the subword being a word segmented from the target word segment; and determine a word segment vector corresponding to the target word segment based at least on the internal word association and the word semantic independence corresponding to the target word segment; The recognition module is used to perform new word recognition on the target word segment based on the word segment vector to obtain a new word recognition result corresponding to the target word segment; a new word refers to a word that is not recorded in the standard vocabulary.

11. A new word recognition model training device, characterized in that: The device comprises: A data acquisition module, configured to acquire a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs; a vector determination module for obtaining a word internal association corresponding to the training word segment based on the number of occurrences of a target character in the training word segment in the second corpus and the number of occurrences of a character segment containing the target character in the training word segment in the second corpus; obtaining a word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of a subword in the training word segment in the second corpus; the subword being a word segmented from the training word segment; and determining a word segment vector corresponding to the training word segment based at least on the word internal association and the word semantic independence corresponding to the training word segment; The model training module is used to train the initial new word recognition model based on the word segment vectors and training word labels corresponding to the training word segments to obtain a target new word recognition model; new words refer to words that are not recorded in the standard vocabulary.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images