New word recognition method and new word recognition model training method and device

Abstract

The invention relates to a new word recognition method, a new word recognition model training method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps of obtaining a word internal association degree corresponding to a target word segment based on the occurrence frequency of a target character in the target word segment in a first corpus set to which the target word segment belongs and the occurrence frequency of a character segment containing the target character in the target word segment in the first corpus set; based on the occurrence frequency of the target word segment in the first corpus set and the occurrence frequency of the sub-words in the target word segment in the first corpus set, obtaining a word semantic independence degree corresponding to the target word segment; the sub-words are words segmented from the target word segment; determining a word segment vector corresponding to the target word segment based on the word internal correlation degree and the word meaning independence degree corresponding to the target word segment; and obtaining a new word recognition result corresponding to the target word segment based on the word segment vector. By adopting the method, the new word recognition accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a new word recognition method, a new word recognition model training method, an apparatus, a computer device, a storage medium and a computer program product. Background Art

[0002] With the development of computer technology, natural language processing technology has emerged. Natural language processing technology is a science that integrates linguistics, computer science, and mathematics. New word discovery is a relatively important part of natural language processing technology. New words are mined by analyzing a large amount of corpus.

[0003] In traditional technology, new word recognition methods are usually based on specific rules, in which language experts construct specific templates based on word formation principles to mine new words. However, mining new words through specific templates has limitations and cannot guarantee the accuracy of new word recognition. Summary of the invention

[0004] Based on this, it is necessary to provide a new word recognition method, a new word recognition model training method, an apparatus, a computer device, a computer-readable storage medium and a computer program product that can improve the accuracy of new word recognition in response to the above technical problems.

[0005] The present application provides a new word recognition method. The method comprises:

[0006] Acquire a target word segment and a first corpus to which the target word segment belongs;

[0007] Based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus, obtaining the internal word association corresponding to the target word segment;

[0008] Based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of subwords in the target word segment in the first corpus, obtaining the word semantic independence corresponding to the target word segment; the subwords are words segmented from the target word segment;

[0009] Determining a word fragment vector corresponding to the target word fragment based at least on the word internal association and word semantic independence corresponding to the target word fragment;

[0010] New word recognition is performed on the target word segment based on the word segment vector to obtain a new word recognition result corresponding to the target word segment; a new word refers to a word that is not recorded in the standard word library.

[0011] The present application also provides a new word recognition device. The device comprises:

[0012] An acquisition module, used for acquiring a target word segment and a first corpus to which the target word segment belongs;

[0013] A determination module, configured to obtain the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus; obtain the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of the subword in the target word segment in the first corpus; the subword is a word segmented from the target word segment; and determine the word segment vector corresponding to the target word segment based at least on the internal word association and the word semantic independence corresponding to the target word segment;

[0014] The recognition module is used to perform new word recognition on the target word segment based on the word segment vector to obtain a new word recognition result corresponding to the target word segment; a new word refers to a word that is not recorded in a standard word library.

[0015] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned new word recognition method when executing the computer program.

[0016] A computer-readable storage medium stores a computer program, which implements the steps of the above-mentioned new word recognition method when executed by a processor.

[0017] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the steps of the new word recognition method are implemented.

[0018] The present application provides a new word recognition model training method. The method comprises:

[0019] Obtaining a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs;

[0020] Based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus, obtaining the internal word association corresponding to the training word segment;

[0021] Based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of subwords in the training word segment in the second corpus, obtaining the word semantic independence corresponding to the training word segment; the subwords are words segmented from the training word segment;

[0022] Determining a word segment vector corresponding to the training word segment based at least on the word internal association and word semantic independence corresponding to the training word segment;

[0023] Based on the word segment vectors and training word labels corresponding to the training word segments, an initial new word recognition model is trained to obtain a target new word recognition model; a new word refers to a word that is not recorded in a standard word library.

[0024] The present application also provides a new word recognition model training device. The device comprises:

[0025] A data acquisition module, used to acquire a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs;

[0026] A vector determination module, configured to obtain the internal word association corresponding to the training word segment based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus; obtain the word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of subwords in the training word segment in the second corpus; the subwords are words segmented from the training word segment; and determine the word segment vector corresponding to the training word segment based at least on the internal word association and the word semantic independence corresponding to the training word segment;

[0027] The model training module is used to train the initial new word recognition model based on the word segment vectors and training word labels corresponding to the training word segments to obtain a target new word recognition model; new words refer to words that are not recorded in the standard vocabulary.

[0028] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps described in the above-mentioned new word recognition model training method when executing the computer program.

[0029] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above-mentioned new word recognition model training method.

[0030] A computer program product includes a computer program, and when the computer program is executed by a processor, the steps described in the above-mentioned new word recognition model training method are implemented.

[0031] The above-mentioned new word recognition method, new word recognition model training method, device, computer equipment, storage medium and computer program product obtain the internal word association corresponding to the word fragment by counting the relevant information of the characters in the word fragment. The internal word association can reflect the degree of association between the characters in the word fragment. The greater the degree of association, the greater the possibility that the word fragment is a new word. The word semantic independence corresponding to the word fragment is obtained by counting the relevant information of finer-grained words in the word fragment. The word semantic independence can reflect the semantic completeness of the word fragment. The more complete the semantics, the greater the possibility that the word fragment is a new word. The word fragment vector corresponding to the word fragment is determined based on the internal word association and word semantic independence corresponding to the word fragment. The word fragment vector contains information such as the internal word association and word semantic independence that are helpful for new word recognition. Whether the target word fragment is a new word is identified based on the word fragment vector corresponding to the target word fragment, which can effectively improve the accuracy of new word recognition. The word fragment vector and training label corresponding to the training word fragment are used as training data to train the new word recognition model, so that the new word recognition model can learn the knowledge related to the new word in the training data to recognize the new word. The new word recognition model that has been trained can improve the accuracy and efficiency of new word recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for describing the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative labor.

[0033] Figure 1 A diagram of an application environment of a new word recognition method and a new word recognition model training method in one embodiment;

[0034] Figure 2 is a schematic diagram of a flow chart of a new word recognition method in one embodiment;

[0035] Figure 3 A schematic diagram of a flow chart of a new word recognition interface in one embodiment;

[0036] Figure 4 A schematic diagram of a process of a new word recognition model training method in one embodiment;

[0037] Figure 5 A schematic diagram of training a new word recognition model in one embodiment;

[0038] Figure 6 is a flow chart of a new word recognition method in another embodiment;

[0039] Figure 7A schematic diagram of an application scenario of the method of the present application in an embodiment;

[0040] Figure 8 is a structural block diagram of a new word recognition device in one embodiment;

[0041] Fig. 9 is a structural block diagram of a new word recognition model training device in one embodiment;

[0042] Fig.10 is an internal structure diagram of a computer device in one embodiment;

[0043] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] The new word recognition method and new word recognition model training method provided in the embodiments of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other servers. Among them, the terminal 102 can be but is not limited to various desktop computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers or a cloud server.

[0046] Both the terminal and the server can be used independently to execute the new word recognition method and the new word recognition model training method provided in the embodiments of the present application.

[0047] For example, the terminal obtains a target word segment and a first corpus to which the target word segment belongs, obtains the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus, obtains the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of the subword in the target word segment in the first corpus, and the subword is a word segmented from the target word segment. The terminal determines the word segment vector corresponding to the target word segment based on at least the internal word association and the word semantic independence corresponding to the target word segment, performs new word recognition on the target word segment based on the word segment vector corresponding to the target word segment, and obtains a new word recognition result corresponding to the target word segment.

[0048] For example, the server obtains a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs. The server obtains the internal word association corresponding to the training word segment based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus, and obtains the word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of subwords in the training word segment in the second corpus, where the subwords are words segmented from the training word segment. The server determines the word segment vector corresponding to the training word segment based on at least the internal word association and word semantic independence corresponding to the training word segment, and trains the initial new word recognition model based on the word segment vector and the training word label corresponding to the training word segment to obtain a target new word recognition model.

[0049] The terminal and the server can also be used together to execute the new word recognition method and the new word recognition model training method provided in the embodiments of the present application.

[0050] For example, the server obtains a target word segment and a first corpus to which the target word segment belongs from a terminal. The server obtains the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus, and obtains the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of the subword in the target word segment in the first corpus, where the subword is a word segmented from the target word segment. The server determines the word segment vector corresponding to the target word segment based on at least the internal word association and the word semantic independence corresponding to the target word segment, performs new word recognition on the target word segment based on the word segment vector corresponding to the target word segment, and obtains the new word recognition result corresponding to the target word segment. The server returns the new word recognition result corresponding to the target word segment to the terminal.

[0051] For example, the server obtains a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs from the terminal. The server obtains the internal word association corresponding to the training word segment based on the difference between the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus, and obtains the word semantic independence corresponding to the training word segment based on the difference between the number of occurrences of the training word segment in the second corpus and the number of occurrences of the subword in the training word segment in the second corpus, where the subword is a word segmented from the target word segment. The server determines the word segment vector corresponding to the training word segment based on at least the internal word association and the word semantic independence corresponding to the training word segment, and trains the initial new word recognition model based on the word segment vector and the training word label corresponding to the training word segment to obtain a target new word recognition model. Subsequently, the server can help the terminal recognize new words based on the target new word recognition model. The server can send the target new word recognition model to the terminal so that the terminal recognizes new words based on the target new word recognition model.

[0052] New words refer to words that are not recorded in the standard word library. The standard word library refers to standard dictionaries and commonly used dictionaries. With the development of the Internet and the widespread use of various application software, new words continue to appear on the Internet. In the face of the continuous emergence of new words on the Internet, it is important to identify new words in a timely manner from the corpus. In traditional technology, new words in the corpus are usually mined based on specific templates. However, the mining scope of the template is limited and has limitations, and the accuracy of new word recognition cannot be guaranteed.

[0053] The new word recognition method provided by the present application obtains a target word fragment and a first corpus to which the target word fragment belongs, calculates various statistical features corresponding to the target word fragment based on the first corpus, for example, the statistical features include word internal association, word semantic independence, and word text independence, determines a word fragment vector corresponding to the target word fragment based on various statistical features corresponding to the target word fragment, the word fragment vector contains various statistical features that are helpful for new word recognition, and identifies whether the target word fragment is a new word based on the word fragment vector corresponding to the target word fragment, which can effectively improve the accuracy of new word recognition. The new word recognition method provided by the present application is applicable to new word recognition in various scenarios.

[0054] In one embodiment, Figure 2 As shown, a new word recognition method is provided, and the method is applied to a computer device as an example, and the computer device can be a terminal or a server. Figure 2 ,The new word recognition method includes the following steps:

[0055] Step S202: Obtain a target word segment and a first corpus to which the target word segment belongs.

[0056] The target word segment refers to a word segment to be identified as a new word. It is understood that the target word segment includes at least two characters. For example, the target word segment can be a segment including 2 characters, a segment including 3 characters, a segment including 4 characters, a segment including 5 characters, a segment including 6 characters, etc.

[0057] The corpus includes multiple corpora, that is, the corpus includes multiple texts. Based on the corpora in the corpus, statistical features corresponding to word segments in the corpus are calculated to identify whether the word segment is a new word. The first corpus to which the target word segment belongs includes at least one corpus containing the target word segment, that is, in the first corpus to which the target word segment belongs, the target word segment appears in at least one corpus.

[0058] Specifically, the computer device can obtain the target word fragment and the first corpus to which the target word fragment belongs locally or from other devices, calculate at least two statistical features corresponding to the target word fragment based on the corpus in the first corpus, and identify whether the target word fragment is a new word based on the at least two statistical features corresponding to the target word fragment.

[0059] In one embodiment, the first corpus may be obtained first, and then the target word segment may be determined from the first corpus. For example, multiple corpora belonging to the same field may be obtained to form the first corpus, each corpus in the first corpus may be segmented to obtain multiple candidate word segments, and the target word segment may be determined from each candidate word segment. It can be understood that if the first corpus is obtained first, each candidate word segment obtained by segmenting each corpus in the first corpus may be used as a target word segment, and all new words appearing in the first corpus may be identified, thereby mining more new words at one time.

[0060] In one embodiment, the target word fragment may be obtained first, and then the first corpus to which the target word fragment belongs may be obtained. For example, a word fragment of interest is obtained as the target word fragment, the corpus containing the target word fragment is collected, and the first corpus is composed based on the collected corpus. It can be understood that the first corpus may be composed of corpus containing the target word fragment, or may be composed of corpus containing the target word fragment and corpus not containing the target word fragment. In a specific embodiment, the user may input the word fragment of interest as the target word fragment on the relevant interface of the terminal, thereby triggering new word recognition for the target word fragment, and obtaining the new word recognition result corresponding to the target word fragment. For example, refer to Figure 3, on the new word recognition interface, the user enters the word fragment of interest in the input box of the word fragment to be recognized, and the recognition control is triggered after the user completes the input. After the user triggers the recognition control, the terminal can determine the new word recognition result locally, or send a new word recognition request carrying the target word fragment to the server, and the server determines the new word recognition result and obtains the new word recognition result from the server. Taking the example of the terminal determining the new word recognition result locally, the terminal automatically obtains the first corpus set to which the corpus constituting the target word fragment belongs from the database, calculates at least two statistical features corresponding to the target word fragment based on the corpus in the first corpus to identify whether the target word fragment is a new word, and obtains the new word recognition result corresponding to the target word fragment. Reference Figure 3 After the terminal obtains the new word recognition result, the new word recognition result can be displayed on the interface so that the user can know the new word recognition result in time. It can be understood that in addition to inputting the word fragment of interest as the target word fragment, the user can also upload the first corpus to which the target word fragment belongs. If the user triggers new word recognition for the target word fragment, the new word recognition result corresponding to the target word fragment input by the user is determined based on the first corpus uploaded by the user.

[0061] In one embodiment, each corpus in the first corpus to which the target word segment belongs belongs to the same field. For example, for the medical field, each corpus in the first corpus to which the target word segment belongs is text related to the medical field. For the financial field, each corpus in the first corpus to which the target word segment belongs is text related to the financial field. In this way, each corpus in the first corpus to which the target word segment belongs belongs to the same field, which helps to mine new words in specific fields and specific scenarios.

[0062] Step S204, obtaining the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus.

[0063] The target character in the target word segment can be any character in the target word segment. For example, if the target word segment is a segment including 4 characters, the 4 characters in the target word segment can be used as target characters respectively, or at least one character can be selected from the 4 characters in the target word segment as the target character. It can be understood that there can be at least one target character in the target word segment.

[0064] The character segment includes at least two characters. The character segment that contains the target character in the target word segment is a character segment composed of the characters contained in the target word segment, and the character segment contains the target character. For example, if the target word segment is "Labor Fruit", and the target character is "Labor", then the character segment that contains the target character in the target word segment may be "Labor". It can be understood that for a certain target character in the target word segment, there may be at least one character segment that contains the target character in the target word segment.

[0065] The internal word association corresponding to a word fragment is calculated based on the number of occurrences of the target character in the word fragment in the corpus to which the word fragment belongs, and the number of occurrences of the character fragment containing the target character in the word fragment in the corpus to which the word fragment belongs. The internal word association corresponding to a word fragment can reflect the degree of association, closeness, and stability between the characters within the word fragment. The greater the internal word association corresponding to a word fragment, the more stable the characters within the word fragment are, and the greater the possibility that the word fragment will become a word.

[0066] Specifically, the computer device determines the target character and the character segment containing the target character from the target word segment, counts the number of occurrences of the target character in the first corpus, counts the number of occurrences of the character segment containing the target character in the first corpus, and obtains the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus. For example, the number of occurrences of the character segment containing the target character in the target word segment in the first corpus is taken as the first number, and the number of occurrences of the target character in the target word segment in the first corpus is taken as the second number, and the internal word association can be obtained based on the ratio between the first number and the second number; the first number and the second number can be adjusted based on a preset weight, and the internal word association can be obtained based on the difference or ratio between the adjusted first number and the second number; and so on.

[0067] Step S206: Obtain the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of the subwords in the target word segment in the first corpus.

[0068] Among them, the subwords in the target word segment are words segmented from the target word segment, that is, the subwords in the target word segment refer to the words contained in the target word segment. The subwords contained in the target word segment can be obtained by segmenting the target word segment. It can be understood that there can be at least one subword in the target word segment. If the target word segment cannot be further split into fine-grained subwords, the word semantic independence can be a preset default value, for example, zero or a negative number.

[0069] The word semantic independence corresponding to a word fragment is calculated based on the number of occurrences of the word fragment in the corpus to which the word fragment belongs, and the number of occurrences of the subwords in the word fragment in the corpus to which the word fragment belongs. The word semantic independence corresponding to a word fragment can reflect the degree of completeness of the semantics of the word fragment. The greater the word semantic independence corresponding to the word fragment, the more complete the semantics of the word fragment, and the greater the possibility that the word fragment is a new word. It can be understood that the closer the number of occurrences of the word fragment in the corpus to which the word fragment belongs and the number of occurrences of the subwords in the word fragment in the corpus to which the word fragment belongs, the greater the possibility that the word fragment is a word, and the closer the semantic completeness of the word fragment and the semantic completeness of the subwords in the word fragment.

[0070] Specifically, the computer device separates fine-grained subwords from the target word fragment, counts the number of occurrences of the target word fragment in the first corpus, counts the number of occurrences of the subwords in the target word fragment in the first corpus, and obtains the word semantic independence corresponding to the target word fragment based on the number of occurrences of the target word fragment in the first corpus and the number of occurrences of the subwords in the target word fragment in the first corpus. For example, the number of occurrences of the target word fragment in the first corpus is taken as the first number, and the number of occurrences of the subwords in the target word fragment in the first corpus is taken as the second number, and the word semantic independence can be obtained based on the difference between the first number and the second number; the difference between the first number and the second number is taken as the first difference, and the word semantic independence can be obtained based on the ratio of the first number and the first difference; and so on.

[0071] Step S208, determining a word fragment vector corresponding to the target word fragment based at least on the word internal association and word semantic independence corresponding to the target word fragment.

[0072] The word fragment vector is a feature vector corresponding to the word fragment, which is used to reflect the feature information of the word fragment.

[0073] Specifically, the computer device may establish a word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment. For example, the word segment vector is obtained based on the word internal association and word semantic independence; the word segment vector is obtained based on the word internal association, word semantic independence and other statistical features.

[0074] Step S210, performing new word recognition on the target word fragment based on the word fragment vector to obtain a new word recognition result corresponding to the target word fragment.

[0075] Among them, new word recognition refers to identifying whether a word segment is a new word. A new word refers to a word that is not recorded in a standard word library. A standard word library can be a standard dictionary developed by a professional department or a professional, and a standard word library can be a commonly used dictionary that people often use.

[0076] Specifically, the word segment vector corresponding to the target word segment includes rich information from different perspectives such as the internal word association and word semantic independence corresponding to the target word segment. The computer device performs new word recognition on the target word segment based on the word segment vector corresponding to the target word segment, and can obtain an accurate new word recognition result corresponding to the target word segment. For example, the word segment vector corresponding to the target word segment includes the internal word association and word semantic independence corresponding to the target word segment, and the weighted sum of the internal word association and word semantic independence is performed to obtain the new word probability corresponding to the target word segment. If the new word probability corresponding to the target word segment is greater than the preset probability, then the new word recognition result corresponding to the target word segment is determined to be that the target word segment is a new word. If the new word probability corresponding to the target word segment is less than or equal to the preset probability, then the new word recognition result corresponding to the target word segment is determined to be that the target word segment is not a new word. For another example, the word fragment vector corresponding to the target word fragment is input into the new word recognition model, and the new word recognition model outputs the predicted word label corresponding to the target word fragment. If the predicted word label is a label indicating that the word fragment belongs to a new word, then the new word recognition result corresponding to the target word fragment is determined to be that the target word fragment is a new word. If the predicted word label is other labels, then the new word recognition result corresponding to the target word fragment is determined to be that the target word fragment is not a new word.

[0077] In the above-mentioned new word recognition method, by obtaining the target word segment and the first corpus to which the target word segment belongs, the internal word association corresponding to the target word segment is obtained based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus, the word semantic independence corresponding to the target word segment is obtained based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of the subword in the target word segment in the first corpus, the subword is a word segmented from the target word segment, and the word segment vector corresponding to the target word segment is determined based on at least the internal word association and the word semantic independence corresponding to the target word segment; the target word segment is subjected to new word recognition based on the word segment vector, and the new word recognition result corresponding to the target word segment is obtained. In this way, the internal word association corresponding to the target word segment is obtained by counting the relevant information of the characters in the target word segment, and the internal word association can reflect the degree of association between the characters in the word segment, and the greater the degree of association, the greater the possibility that the target word segment is a new word. The word semantic independence corresponding to the target word fragment is obtained by counting the relevant information of finer-grained words in the target word fragment. The word semantic independence can reflect the semantic completeness of the word fragment. The more complete the semantics, the greater the possibility that the target word fragment is a new word. The word fragment vector corresponding to the target word fragment is determined based on the word internal association and word semantic independence corresponding to the target word fragment. The word fragment vector contains information such as word internal association and word semantic independence that are helpful for new word recognition. Based on the word fragment vector corresponding to the target word fragment, whether the target word fragment is a new word can be identified, which can effectively improve the accuracy of new word recognition. In addition, the computational complexity of word internal association and word semantic independence is not high, which also helps to improve the efficiency of new word recognition.

[0078] In one embodiment, obtaining a target word segment and a first corpus to which the target word segment belongs includes:

[0079] Acquire a first corpus; each corpus in the first corpus belongs to the same field; segment each corpus in the first corpus based on a preset segmentation length set to obtain a candidate word fragment set that matches each segmentation length in the preset segmentation length set; each candidate word fragment in the same candidate word fragment set has the same fragment length; determine a target word fragment from each candidate word fragment set.

[0080] The preset segmentation length set includes at least one preset segmentation length. The segmentation length is used to determine the segment length of the word segment segmented from the corpus. The segment length corresponding to the word segment refers to the number of characters contained in the word segment. For example, if the preset segmentation length is 3, the word segment obtained by segmenting the corpus based on the segmentation length is a segment including 3 characters, and the segment length of the word segment is 3.

[0081] Each corpus in the first corpus set is segmented based on a segmentation length to obtain a candidate word fragment set, and the fragment length of each candidate word fragment in the candidate word fragment set is the segmentation length. For example, if the preset segmentation length is 5, each corpus in the first corpus set is segmented based on the segmentation length, and the segmented fragments consisting of 5 characters are formed into a candidate word fragment set.

[0082] Specifically, the computer device may first obtain a first corpus consisting of various corpora in the same field, then segment each corpus in the first corpus, and determine the target word segment from the segmentation result.

[0083] The computer device segments each corpus in the first corpus set based on a preset segmentation length set to obtain a candidate word segment set that matches each segmentation length in the preset segmentation length set. For any segmentation length in the preset segmentation length set, the corpus in the first corpus set is segmented using a sliding window whose window length is consistent with the segmentation length, and the sliding window is slid across the corpus character by character, and each sliding will frame a string, and the framed string is used as a candidate word segment, and each candidate word segment is combined into a candidate word segment set.

[0084] The computer device finally determines the target word segment from the candidate word segment set that matches each segmentation length in the preset segmentation length set. For example, the candidate word segments in each candidate word segment set are respectively used as the target word segment; at least one candidate word segment is selected from each candidate word segment set as the target word segment.

[0085] In the above embodiment, a first corpus consisting of various corpora in the same field is first obtained, and then target word fragments are segmented out from the first corpus. The first corpus includes various corpora belonging to the same field. By performing new word recognition on the target word fragments segmented out from the first corpus, it is helpful to discover new words in a certain field.

[0086] In one embodiment, based on the number of occurrences of a target character in the target word segment in the first corpus and the number of occurrences of a character segment containing the target character in the target word segment in the first corpus, obtaining the internal word association corresponding to the target word segment includes:

[0087] Take each character in the target word segment as a target character in turn; for any target character, form a character segment containing the target character in the target word segment by combining the target character and the adjacent backward character of the target character in the target word segment; according to the occurrence ratio between the number of occurrences of the character segment containing the target character in the target word segment and the number of occurrences of the target character in the target word segment in the first corpus, obtain the occurrence ratios corresponding to each character segment in the target word segment respectively; perform smoothing processing on the occurrence ratios corresponding to each character segment to obtain the internal word association degrees corresponding to each character segment respectively; according to the internal word association degrees corresponding to each character segment respectively, obtain the internal word association degree corresponding to the target word segment.

[0088] Among them, the adjacent backward character of the target character in the target word segment refers to the character that is adjacent to the target character and is located after the target character in the target word segment. That is, the adjacent backward character of the target character in the target word segment refers to the character that immediately appears after the target character in the target word segment. For example, if the target word segment is "fruits of labor", and the target character is "lao", then the adjacent backward character of this target character in the target word segment is "dong"; if the target character is "guo", then the adjacent backward character of this target character in the target word segment is "shi".

[0089] Form a character segment containing the target character in the target word segment by combining the target character and the adjacent backward character of the target character in the target word segment. That is, the character segment containing the target character in the target word segment includes the target character and the adjacent backward character of the target character in the target word segment. For example, if the target word segment is "fruits of labor", and the target character is "lao", then the character segment containing this target character is "labor"; if the target character is "guo", then the character segment containing this target character is "fruits".

[0090] It can be understood that for the last character in the target word segment, since there is no adjacent backward character for this character in the target word segment, there is no corresponding character segment for this character.

[0091] The purpose of smoothing the data is to make the data smooth and stable for subsequent data processing. The smoothing processing can specifically be numerical compression of the data to compress the value range of the data. For example, smoothing processing can be performed through a logarithmic function; smoothing processing can be performed through a custom formula; and so on.

[0092] Specifically, when calculating the internal word association degree corresponding to the target word segment, the internal word association degrees corresponding to each character segment in the target word segment can be calculated, and then according to the internal word association degrees corresponding to each character segment respectively, the internal word association degree corresponding to the target word segment can be finally obtained.

[0093] The computer device can use each character in the target word segment as the target character in turn, exhaust the characters in the target word segment, and make full use of the characters in the target word segment. For any target character, the target character and the adjacent backward characters of the target character in the target word segment form a character segment containing the target character in the target word segment, so as to obtain the character segments corresponding to each target character. For any target character, the number of occurrences of the character segment containing the target character in the target word segment in the first corpus is counted, the number of occurrences of the target character in the target word segment in the first corpus is counted, and the ratio between the two occurrences is calculated to obtain the number of occurrences ratio corresponding to the character segment. The target word segment includes at least two characters, and each character has a corresponding character segment. The number of occurrences ratios corresponding to each character segment in the target word segment can be calculated. Further, in order to make the occurrence ratio data stable for subsequent processing, the number of occurrences ratios corresponding to each character segment can be smoothed to obtain the word internal association degree corresponding to each character segment. Finally, the computer device obtains the word internal association degree corresponding to the target word segment according to the word internal association degree corresponding to each character segment in the target word segment. For example, the internal word association corresponding to each character fragment in the target word fragment is averaged to obtain the internal word association corresponding to the target word fragment; the internal word association corresponding to each character fragment in the target word fragment is weighted averaged to obtain the internal word association corresponding to the target word fragment, and the weights corresponding to the internal word associations corresponding to each character fragment can be set as needed; and so on.

[0094] For example, the target word segment is "labor fruit", the character segment corresponding to the target character "labor" is "labor", the character segment corresponding to the target character "movement" is "movement fruit", and the character segment corresponding to the target character "fruit" is "fruit". Based on the ratio between the number of occurrences of "labor" in the first corpus and the number of occurrences of "labor" in the first corpus, the number of occurrences corresponding to "labor" is obtained, based on the ratio between the number of occurrences of "movement fruit" in the first corpus and the number of occurrences of "movement" in the first corpus, the number of occurrences corresponding to "movement fruit" is obtained, based on the ratio between the number of occurrences of "fruit" in the first corpus and the number of occurrences of "fruit" in the first corpus, the number of occurrences corresponding to "fruit" is obtained, the number of occurrences corresponding to "labor", "movement fruit", and "fruit" are smoothed respectively, and the internal word correlations corresponding to "labor", "movement fruit", and "fruit" are obtained. According to the internal word correlations corresponding to "labor", "movement fruit", and "fruit", the internal word correlation corresponding to "labor fruit" is obtained.

[0095] In one embodiment, the calculation formula of the word internal association degree T corresponding to the word fragment is:

[0096]

[0097] Where l represents the length of the word segment, that is, the number of characters that the word segment consists of. i represents the i-th character of the word fragment; T i+1 represents the i+1th character of the word fragment; p(T i ) represents the number of times the i-th character of a word fragment appears in the corpus to which the word fragment belongs; p(T i |T i+1 ) represents the number of times the combination of the i-th character and the i+1-th character of the word fragment appears in the corpus to which the word fragment belongs. It can be understood that T i is the target character in the word fragment, p(T i ) is the number of times the target character in the word fragment appears in the corpus to which the word fragment belongs. i and T i+1 The character segments that make up the word segments contain the target characters, p(T i |T i+1 ) is the number of times the character fragment containing the target character in the word fragment appears in the corpus to which the word fragment belongs. It is the ratio of the number of occurrences of the corresponding character fragments. log represents smoothing. It is the internal correlation degree of the word corresponding to the character fragment. T is the internal correlation degree of the word corresponding to the word fragment. The greater the internal correlation degree of the word corresponding to the word fragment, the more stable the internal correlation of the word, and the greater the possibility of its formation. The internal correlation degree of the word corresponding to the word fragment can also be called the average internal correlation degree of the word.

[0098] In the above embodiment, each character in the target word fragment is taken as the target character in turn, and the target character and the adjacent backward characters of the target character in the target word fragment are combined into a character fragment containing the target character in the target word fragment, and the internal word association corresponding to each character fragment in the target word fragment is calculated respectively, and then the internal word association corresponding to the target word fragment is obtained based on the internal word association corresponding to each character fragment in the target word fragment, so that the internal word association corresponding to the target word fragment is integrated with the statistical information between each group of adjacent characters in the target word fragment, which can fully reflect the internal association of the word fragment and has high accuracy.

[0099] In one embodiment, based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of the subwords in the target word segment in the first corpus, the word semantic independence corresponding to the target word segment is obtained, including:

[0100] Segment the target word fragment to obtain the subwords contained in the target word fragment; obtain the fused occurrence count according to the number of occurrences of each subword of the target word fragment in the first corpus; adjust the number of occurrences of the target word fragment in the first corpus based on the total number of subwords corresponding to the target word fragment to obtain the reference occurrence count; determine the difference in occurrence count between the fused occurrence count and the reference occurrence count; obtain the word semantic independence corresponding to the target word fragment based on the number of occurrences of the target word fragment in the first corpus and the difference in occurrence count.

[0101] The total number of subwords corresponding to the target word segment refers to the number of subwords contained in the target sub-segment. For example, if the target word segment is "labor fruit", "labor fruit" is segmented to obtain "labor" and "fruit". "Labor" and "fruit" are two subwords contained in "labor fruit", and the total number of subwords corresponding to "labor fruit" is 2.

[0102] Specifically, the target word segment may be composed of fine-grained words. When calculating the semantic independence of the words corresponding to the target word segment, the fine-grained words in the target word segment can be exhausted, and all subwords contained in the target word segment can be fully utilized. The computer device performs word segmentation on the target word segment to obtain each subword contained in the target word segment, and counts the number of occurrences of each subword contained in the target word segment in the first corpus, and obtains the fused number of occurrences according to the number of occurrences of each subword contained in the target word segment in the first corpus. For example, the number of occurrences of each subword contained in the target word segment in the first corpus is summed to obtain the fused number of occurrences; the number of occurrences of each subword contained in the target word segment in the first corpus is weighted and summed to obtain the fused number of occurrences, and the weights corresponding to the number of occurrences of each subword can be set as needed; and so on. Then, the number of occurrences of the target word segment in the first corpus can be adjusted based on the number of subwords corresponding to the target word segment to obtain the reference number of occurrences. For example, the number of occurrences of the target word fragment in the first corpus is magnified based on the number of subwords corresponding to the target word fragment; the reference number of occurrences obtained by adjustment is proportional to the number of subwords used for adjustment; and so on. Further, the difference between the reference number of occurrences and the fused number of occurrences can be calculated to obtain the number of occurrences difference. For example, the difference between the reference number of occurrences and the fused number of occurrences is calculated, and the difference is used as the number of occurrences difference; the difference is adjusted based on a preset value to obtain the number of occurrences difference. Finally, the word semantic independence corresponding to the target word fragment can be obtained based on the number of occurrences of the target word fragment in the first corpus and the number of occurrences difference. For example, the ratio of the number of occurrences of the target word fragment in the first corpus to the number of occurrences difference is used as the word semantic independence; the difference between the number of occurrences of the target word fragment in the first corpus and the number of occurrences difference is used as the word semantic independence.

[0103] It is understandable that the target word segment can be segmented using an existing segmentation tool to obtain subwords contained in the target word segment. For example, the jieba (jieba) segmentation tool can be used to segment the target word segment.

[0104] In one embodiment, the calculation formula of the word semantic independence W corresponding to the word fragment is:

[0105]

[0106] Where w represents a word fragment; p(w) represents the number of times the word fragment w appears in the corpus; N represents that the word fragment w can be composed of N fine-grained words; w j represents the jth word among N fine-grained words; p(w j ) represents the word w j The number of times the word fragment w appears in the corpus to which it belongs. It can be understood that N is the total number of subwords corresponding to the word fragment, p(w) is the number of times the word fragment appears in the corpus to which the word fragment belongs, and w j is the jth subword in the word fragment, p(w j ) is the number of occurrences of the jth subword in the word fragment in the corpus to which the word fragment belongs. is the number of fusion occurrences, (N-1)p(w) is the number of reference occurrences, That is, the difference in the number of occurrences. The larger the word semantic independence W corresponding to the word fragment, the more stable the word fragment, the more complete the semantics, and the greater the probability of being a new word. For a certain word fragment, the value of the numerator in the calculation formula is fixed. If the word fragment is split into fewer fine-grained words, the denominator in the calculation formula will be smaller, and the calculation result of the calculation formula will be larger, and the word semantic independence of the word fragment will be greater. If the word fragment is split into more fine-grained words, the denominator in the calculation formula will be larger, the word semantic independence of the word fragment will be smaller, and the probability of being a new word will be smaller. For example, the word fragment w is "labor fruit", which can be divided into two finer-grained words "labor" and "fruit". The two words "labor" and "fruit" do not have a specific meaning when viewed separately, and the probability of being a new word is low. When combined into "labor fruit", it has a specific meaning "the result obtained through a certain behavior", so it is more likely to be a new word. The word semantic independence corresponding to the word fragment can also be called the word semantic independence value.

[0107] In the above embodiment, the target word segment is segmented to obtain each subword contained in the target word segment, and the fused occurrence count is obtained according to the number of occurrences of each subword of the target word segment in the first corpus. The number of occurrences of the target word segment in the first corpus is adjusted based on the total number of subwords corresponding to the target word segment to obtain the reference occurrence count, and the difference in the number of occurrences between the fused occurrence count and the reference occurrence count is determined. Based on the number of occurrences of the target word segment in the first corpus and the difference in the number of occurrences, the word semantic independence corresponding to the target word segment is obtained. In this way, when calculating the word semantic independence, the statistical information of all subwords contained in the reference word segment is compared with the comprehensive value of the statistical information of all subwords and the statistical information of the word segment, so that the word semantic independence can fully reflect the difference between the word segment and the subwords in the word segment, and fully reflect the semantic integrity of the word segment, with high accuracy.

[0108] In one embodiment, determining the word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment includes:

[0109] A candidate word fragment set to which a target word fragment belongs is obtained; the fragment lengths of the candidate word fragments in the candidate word fragment set are consistent, and the candidate word fragment set is obtained by segmenting the corpus in the first corpus set; at least one reference word fragment corresponding to the target word fragment is determined from the candidate word fragment set; a target distance is determined from the edit distances between the target word fragment and the corresponding at least one reference word fragment, and the word text independence corresponding to the target word fragment is determined based on the target distance; a word fragment vector corresponding to the target word fragment is determined based at least on the word internal association, word semantic independence and word text independence corresponding to the target word fragment.

[0110] The edit distance between two word fragments refers to the minimum number of edit operations required to convert one word fragment into another. The larger the edit distance between two word fragments, the more different the two word fragments are. Permitted edit operations include replacing one character with another, inserting a character, and deleting a character. The edit distance can also be called the Levenshtein distance or Edit Distance.

[0111] The reference word segment corresponding to the target word segment refers to a word segment whose segment length is consistent with the segment length of the target word segment, and the reference word segment corresponding to the target word segment is also consistent with the corpus to which the target word segment belongs. The reference word segment corresponding to the target word segment is determined from the candidate word segment set to which the target word segment belongs. The candidate word segment set to which the target word segment belongs includes at least one candidate word segment contained in the first corpus to which the target word segment belongs, whose segment length is consistent with the segment length of the target word segment. It can be understood that the target word segment and the corresponding reference word segment are different word segments.

[0112] The word text independence degree corresponding to a word fragment is calculated based on the edit distance between the word fragment and at least one corresponding reference word fragment. The smaller the edit distance between a word fragment and the corresponding reference word fragment, the more independent the word fragment is. The greater the word text independence degree corresponding to the word fragment, the greater the probability that it is a new word.

[0113] Specifically, in addition to identifying new words based on the internal word association and word semantic independence corresponding to the target word fragment, we can further introduce the word text independence corresponding to the target word fragment, and identify new words based on the internal word association, word semantic independence and word text independence corresponding to the target word fragment.

[0114] The computer device can obtain the set of candidate word segments to which the target word segment belongs, and determine at least one reference word segment corresponding to the target word segment from the set of candidate word segments to which the target word segment belongs. For example, each candidate word segment in the set of candidate word segments to which the target word segment belongs can be used as a reference word segment corresponding to the target word segment; a candidate word segment that appears more than a preset number of times in the first corpus can be selected from the set of candidate word segments to which the target word segment belongs as a reference word segment corresponding to the target word segment, and the candidate word segment that appears more than a preset number of times in the first corpus is more representative; and so on. Furthermore, the computer device determines the target distance from the edit distance between the target word segment and the corresponding at least one reference word segment, and determines the word text independence corresponding to the target word segment based on the target distance. For example, the minimum value is obtained from each edit distance as the target distance, and the target distance is used as the word text independence; the average value of each edit distance is calculated as the target distance, and the target distance is used as the word text independence; and so on.

[0115] Ultimately, the computer device can determine the word fragment vector corresponding to the target word fragment based at least on the internal word association, word semantic independence and word text independence of the target word fragment, so that the word fragment vector includes information from at least three perspectives, thereby improving the accuracy of new word recognition.

[0116] In one embodiment, the calculation formula of the word text independence S corresponding to the word fragment is:

[0117] S=min{D word,k =D(word,k),k Q}

[0118] Among them, D word,k represents the edit distance between word fragment word and word fragment k, word fragment word is the word fragment whose corresponding word text independence needs to be calculated, and word fragment k is the reference word fragment corresponding to word fragment word; D(word, k) represents the edit distance calculation formula; Q is a set of word fragments other than word fragment word (for example, word fragment word is a fragment consisting of 3 characters, then Q is a set of word fragments, and the word fragments in Q are all fragments consisting of 3 characters, and Q does not include word fragment word); min{} represents taking the minimum value of all D(word, k). It can be understood that word fragment word is the target word fragment, word fragment k is the reference word fragment corresponding to the target word fragment, and the set composed of Q and word fragment word is the candidate word fragment set to which the target word fragment belongs. min{D word,k =D(word,k), k∈Q} is the target distance. New words usually have a certain degree of novelty and independence. The larger the minimum value of S, the higher the word text independence of the word fragment, and the greater the probability of being a new word. The word text independence corresponding to the word fragment can also be called the word text independence value.

[0119] In the above embodiment, the target word segment and the corresponding reference word segment have the same segment length and belong to the same corpus. The target distance is determined from the edit distance between the target word segment and at least one corresponding reference word segment. The word text independence corresponding to the target word segment is determined based on the target distance, so that the word text independence can reflect the text difference between the target word segment and the reference word segment, embody the independence of the target word segment, and have high accuracy. At least based on the word internal association, word semantic independence and word text independence corresponding to the target word segment, a word segment vector corresponding to the target word segment is established, so that the word segment vector has richer information, which helps to improve the accuracy of new word recognition.

[0120] In one embodiment, determining the word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment includes:

[0121] Obtaining supplementary feature sets corresponding to the target word fragments respectively; determining the word fragment vectors corresponding to the target word fragments based at least on the word internal association, word semantic independence and supplementary feature sets corresponding to the target word fragments.

[0122] The supplementary feature set includes at least one of the word external association, part-of-speech value, corpus importance, and segment length corresponding to the target word segment.

[0123] Perform a character type statistics on the adjacent forward characters corresponding to a word segment in the corpus to which the word segment belongs, and obtain the number of distinct adjacent forward characters corresponding to the word segment in the corpus to which the word segment belongs. The number of distinct adjacent forward characters corresponding to a word segment in its corpus refers to the number of character types of the single characters that are directly adjacent to the word segment and are in front of the word segment in the corpus to which the word segment belongs. That is, the number of distinct adjacent forward characters corresponding to a word segment in its corpus is used to represent how many different characters the character on the left side of the word segment appears in the corpus to which the word segment belongs. Perform a character type statistics on the adjacent backward characters corresponding to a word segment in the corpus to which the word segment belongs, and obtain the number of distinct adjacent backward characters corresponding to the word segment in the corpus to which the word segment belongs. The number of distinct adjacent backward characters corresponding to a word segment in its corpus refers to the number of character types of the single characters that are directly adjacent to the word segment and are behind the word segment in the corpus to which the word segment belongs. That is, the number of distinct adjacent backward characters corresponding to a word segment in its corpus is used to represent how many different characters the character on the right side of the word segment appears in the corpus to which the word segment belongs. It can be understood that the adjacent forward characters corresponding to a word segment in the corpus to which the word segment belongs refer to the characters that are adjacent to the word segment and are before the word segment in the corpus to which the word segment belongs. The adjacent backward characters corresponding to a word segment in the corpus to which the word segment belongs refer to the characters that are adjacent to the word segment and are after the word segment in the corpus to which the word segment belongs. For example, if the target word segment is "weather", and one corpus in the first corpus that includes the target word segment is "It's sunny today", then "day" is an adjacent forward character corresponding to the target word segment in the first corpus, and "sunny" is an adjacent backward character corresponding to the target word segment in the first corpus.

[0124] The external word association degree corresponding to a word segment is calculated based on the number of distinct adjacent forward characters and the number of distinct adjacent backward characters corresponding to the word segment in the corpus to which the word segment belongs. For example, the sum of the number of distinct adjacent forward characters and the number of distinct adjacent backward characters is obtained as the external word association degree; the weighted sum of the number of distinct adjacent forward characters and the number of distinct adjacent backward characters is obtained as the external word association degree, and the weights corresponding to the number of distinct adjacent forward characters and the number of distinct adjacent backward characters can be set as needed; and so on. Further illustrate with an example, if the target word segment is "peace", there are 50 different characters that appear on the left side of "peace" in the first corpus, and 30 different characters that appear on the right side of "peace" in the first corpus, then the external word association degree of the target word segment "peace" is 80. The greater the external word association degree corresponding to a word segment, the richer the surrounding characters of the word segment, and the greater the possibility of the word segment forming a word.

[0125] The part-of-speech value corresponding to a word fragment is obtained by counting the parts of speech reflected in the word fragment in the corpus to which it belongs. The part-of-speech value is used to indicate how many parts of speech the target word fragment has in the first corpus. For example, if the target word fragment has two parts of speech, noun and adjective, in the first corpus, the part-of-speech value corresponding to the target word fragment is 2. Generally speaking, the lower the part-of-speech value corresponding to a word fragment, the more stable the word fragment is, and the greater the probability that the word fragment becomes a word. In one embodiment, the part-of-speech value corresponding to a word fragment can be counted by a tool with a part-of-speech analysis function.

[0126] The corpus importance corresponding to a word fragment is calculated based on the first frequency and the second frequency corresponding to the word fragment in the corpus to which the word fragment belongs. The first frequency refers to the frequency of occurrence of the word fragment in the corpus containing the word fragment, and the corpus containing the word fragment is in the corpus to which the word fragment belongs. The second frequency refers to the frequency of occurrence of the corpus containing the word fragment in the corpus to which the word fragment belongs. For example, the first frequency and the second frequency are multiplied to obtain the external relevance of the word. In one embodiment, the corpus importance is a TF-IDF (term frequency–inverse document frequency) value.

[0127] Specifically, in addition to identifying new words based on the internal word association and word semantic independence corresponding to the target word fragment, a supplementary feature set corresponding to the target word fragment can be further introduced to identify new words based on the internal word association, word semantic independence and supplementary feature set corresponding to the target word fragment.

[0128] The computer device obtains the supplementary feature sets corresponding to the target word fragments respectively, and determines the word fragment vector corresponding to the target word fragment based at least on the internal word association, word semantic independence and supplementary feature sets corresponding to the target word fragments, so that the word fragment vector includes information from multiple perspectives, thereby improving the accuracy of new word recognition.

[0129] In the above embodiment, a word fragment vector corresponding to the target word fragment is established based at least on the internal word association, word semantic independence and supplementary feature set corresponding to the target word fragment, so that the word fragment vector has richer information, which helps to improve the accuracy of new word recognition.

[0130] In one embodiment, a word fragment vector corresponding to the target word fragment is established based at least on the word internal association, word semantic independence, word text independence and a supplementary feature set corresponding to the target word fragment.

[0131] In one embodiment, determining the word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment includes:

[0132] At least the internal word association and the word semantic independence corresponding to the target word fragment are normalized, and the normalized internal word association and word semantic independence corresponding to the target word fragment are concatenated to obtain a word fragment vector corresponding to the target word fragment.

[0133] The normalization process is used to map the data value to a preset range for subsequent data processing. For example, the value of the internal association of a word can be mapped to between 0 and 1, and the value of the semantic independence of a word can be mapped to between 0 and 1.

[0134] Specifically, when determining the word fragment vector corresponding to the target word fragment based at least on the internal word association and word semantic independence corresponding to the target word fragment, the internal word association corresponding to the target word fragment can be normalized first, and the word semantic independence corresponding to the target word fragment can be normalized, and then the normalized internal word association and word semantic independence can be concatenated to obtain the word fragment vector corresponding to the target word fragment.

[0135] It can be understood that the statistical features used to establish the word fragment vector corresponding to the word fragment include internal word association, word semantic independence, word text independence and at least two statistical features in the supplementary feature set. When determining the word fragment vector corresponding to the word fragment based on the statistical features corresponding to the word fragment, various statistical features can be normalized first, and then the various normalized statistical features can be concatenated to obtain the word fragment vector corresponding to the word fragment.

[0136] In the above embodiment, various statistical features are normalized and then concatenated to obtain word fragment vectors, which can effectively avoid the weight of a single statistical feature being too large or too small, thereby affecting the accuracy of subsequent new word recognition.

[0137] In one embodiment, new word recognition is performed on the target word segment based on the word segment vector to obtain a new word recognition result corresponding to the target word segment, including:

[0138] The word fragment vector is input into the target new word recognition model, the predicted word label corresponding to the target word fragment is output, and the new word recognition result corresponding to the target word fragment is determined based on the predicted word label corresponding to the target word fragment; the predicted word label includes any one of a first label used to indicate that the word fragment is a non-word, a second label used to indicate that the word fragment is a word but not a new word, and a third label used to indicate that the word fragment is a new word.

[0139] Among them, the new word recognition model is a machine learning model for new word recognition. The input data of the new word recognition model is the word segment vector corresponding to the word segment, and the output data is the predicted word label corresponding to the word segment. The new word recognition model can output three types of predicted word labels, namely, a first label used to indicate that the word segment belongs to a non-word, a second label used to indicate that the word segment belongs to a word but not a new word, and a third label used to indicate that the word segment belongs to a new word. If the predicted word label corresponding to the word segment is the first label, it means that the word segment is not a word. If the predicted word label corresponding to the word segment is the second label, it means that the word segment is a word but not a new word. If the predicted word label corresponding to the word segment is the third label, it means that the word segment is a new word. The target new word recognition model refers to a new word recognition model that has completed training. The word segment vector corresponding to the word segment is input into the target new word recognition model. The target new word recognition model processes the input data and selects one label from the three labels according to the data processing results. Output.

[0140] Specifically, the computer device can perform new word recognition with the help of a machine learning model. The computer device can obtain a target new word recognition model, input the word segment vector corresponding to the target word segment into the target new word recognition model, the target new word recognition model processes the input data to obtain a predicted word label corresponding to the target word segment, and the target new word recognition model outputs the predicted word label corresponding to the target word segment. The predicted word label corresponding to the target word segment includes any one of a first label for indicating that the word segment belongs to a non-word, a second label for indicating that the word segment belongs to a word but is not a new word, and a third label for indicating that the word segment belongs to a new word. The predicted word label corresponding to the target word segment can be used as the new word recognition result corresponding to the target word segment.

[0141] It can be understood that the machine learning algorithm used to establish the new word recognition model can be selected according to actual needs. For example, a new word recognition model is established based on the LR (Logistic Regression) algorithm, a new word recognition model is established based on the deepFM algorithm, and a new word recognition model is established based on the XGboost (Extreme Gradient Boosting) algorithm.

[0142] In the above embodiment, the word segment vector corresponding to the target word segment is input into the target new word recognition model for new word recognition. The target new word recognition model has a strong learning ability and can accurately recognize new words based on the knowledge that has been learned, effectively ensuring the accuracy of new word recognition. In addition, the word segment vector corresponding to the target word segment is input into the target new word recognition model for new word recognition, which can also ensure the efficiency of new word recognition.

[0143] In one embodiment, the new word identification method further includes:

[0144] Based on the first target word segment and the second target word segment corresponding to the first corpus, a domain dictionary corresponding to the domain to which the first corpus belongs is determined.

[0145] The predicted word label corresponding to the first target word segment is a label used to indicate that the word segment belongs to a word but is not a new word, and the predicted word label corresponding to the second target word segment is a label used to indicate that the word segment belongs to a new word.

[0146] Specifically, multiple corpora belonging to the same field are obtained to form a first corpus set, and words and new words are identified from the first corpus to establish a domain dictionary. The computer device can obtain multiple candidate word segments by segmenting the corpus in the first corpus, and each candidate word segment is used as a target word segment. The target word segment is identified as a new word by establishing a word segment vector corresponding to the target word segment, and the new word recognition results corresponding to each target word segment corresponding to the first corpus are obtained. From these, each first target word segment whose new word recognition result is that the word segment belongs to a new word and each second target word segment whose new word recognition result is that the word segment belongs to a word but is not a new word are obtained to form a domain dictionary corresponding to the field to which the first corpus belongs.

[0147] In one embodiment, the computer device can establish a knowledge graph corresponding to the field to which the first corpus belongs based on the field dictionary. For example, in the process of building the knowledge graph, entities need to be identified, and entity recognition can be performed based on the words in the field dictionary to identify richer and more comprehensive entities to improve the quality of the knowledge graph.

[0148] In the above embodiment, the words and new words identified by the new word identification method of the present application are applied to the construction of a domain dictionary, which can enrich the content of the domain dictionary.

[0149] In one embodiment, Figure 4 As shown, a new word recognition model training method is provided, and the method is applied to a computer device as an example, and the computer device can be a terminal or a server. Figure 4 ,The new word recognition model training method includes the following steps:

[0150] Step S402: Obtain training word segments, training word tags corresponding to the training word segments, and a second corpus to which the training word segments belong.

[0151] The training word segment refers to a word segment of training data used to generate the model. The training word segment is a word segment for which the new word recognition result is known. The training word tag corresponding to the training word segment is used to indicate the correct new word recognition result corresponding to the training word segment. The second corpus to which the training word segment belongs includes at least one corpus containing the training word segment, that is, in the second corpus to which the training word segment belongs, the training word segment appears in at least one corpus.

[0152] Specifically, the computer device can obtain relevant data of training word fragments to establish training data for the new word recognition model, train the new word recognition model based on the training data, and obtain a machine learning model for new word recognition.

[0153] Step S404, based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus, obtain the internal word association corresponding to the training word segment.

[0154] Step S406: Obtain the word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of the subwords in the training word segment in the second corpus.

[0155] Step S408, determining the word segment vector corresponding to the training word segment based at least on the word internal association and word semantic independence corresponding to the training word segment.

[0156] It can be understood that the calculation process of the internal word association corresponding to the training word fragment can refer to the calculation process of the internal word association corresponding to the aforementioned target word fragment, the calculation process of the word semantic independence corresponding to the training word fragment can refer to the calculation process of the word semantic independence corresponding to the aforementioned target word fragment, and the determination process of the word fragment vector corresponding to the training word fragment can refer to the establishment process of the word fragment vector corresponding to the aforementioned target word fragment, which will not be repeated here.

[0157] Step S410, training the initial new word recognition model based on the word segment vectors corresponding to the training word segments and the training word labels to obtain a target new word recognition model.

[0158] The initial new word recognition model refers to the new word recognition model to be trained, and the target new word recognition model refers to the trained new word recognition model.

[0159] Specifically, referring to the process of establishing the word segment vector corresponding to the target word segment, the word segment vector corresponding to the training word segment is established based on the training word segment and the second corpus to which the training word segment belongs. The word segment vector and the training word label corresponding to the training word segment are used as training data of the model. During model training, the word segment vector corresponding to the training word segment is used as input data of the model, and the training word label corresponding to the training word segment is used as expected output data of the model. The computer device trains the initial new word recognition model based on the word segment vector and the training word label corresponding to the training word segment to obtain the target new word recognition model.

[0160] For example, refer to Figure 5The statistical features corresponding to the training word fragments include part-of-speech value, word external association, word internal association, word TF-IDF value, word length value, word semantic independence, and word text independence. The various statistical features corresponding to the training word fragments are normalized respectively, and the various statistical features corresponding to the training word fragments after normalization constitute the word fragment vector corresponding to the training word fragment. The word fragment vector corresponding to the training word fragment is input into the new word recognition model to be trained. The new word recognition model processes the input data and outputs the predicted word label corresponding to the training word fragment. The model loss is generated based on the difference between the predicted word label corresponding to the training word fragment and the training word label. The model loss is back-propagated to adjust the parameters of the new word recognition model. Through iterative training, the parameters of the model are continuously adjusted until the training end condition is met, and a new word recognition model with training is obtained. The training end condition can be at least one condition indicating that the model training is completed, such as the model loss is less than the preset loss value, the model iteration number is greater than the preset iteration number, etc.

[0161] It can be understood that the training data of the model may include word segment vectors and training word labels of multiple training word segments. The model is trained with a large amount of training data so that the model can learn relevant knowledge about the new word recognition task.

[0162] In one embodiment, different new word recognition models can be trained for different fields. For example, for the medical field, a second corpus consisting of medical texts is obtained, the corpus in the second corpus is segmented to obtain multiple training word segments, word segment vectors corresponding to the training word segments are established based on the second corpus, and a new word recognition model for new word recognition in the medical field is trained based on a large number of word segment vectors and training word labels corresponding to the training word segments. For the game field, a second corpus consisting of game texts is obtained, the corpus in the second corpus is segmented to obtain multiple training word segments, word segment vectors corresponding to the training word segments are established based on the second corpus, and a new word recognition model for new word recognition in the game field is trained based on a large number of word segment vectors and training word labels corresponding to the training word segments.

[0163] In one embodiment, the first corpus is segmented based on a preset first segmentation length set to determine target word segments, and the second corpus is segmented based on a preset second segmentation length set to determine training word segments. In order to ensure the efficiency of model training, the second segmentation length set has a segmentation length restriction. During the training process of the model, the model has gradually learned the relevant knowledge of the new word recognition task. During the application of the model, the first segmentation length set may not have a segmentation length restriction, and the model can perform new word recognition on word segments of any segment length, thereby ensuring the versatility of the method of the present application.

[0164] The above-mentioned new word recognition model training method obtains a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs, obtains the internal word association corresponding to the training word segment based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus, obtains the word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of the subwords in the training word segment in the second corpus, determines the word segment vector corresponding to the training word segment based on at least the internal word association and the word semantic independence corresponding to the training word segment, and trains the initial new word recognition model based on the word segment vector and the training word label corresponding to the training word segment to obtain the target new word recognition model. In this way, the internal word association corresponding to the training word segment is obtained by statistically analyzing the relevant information of the characters in the training word segment. The internal word association can reflect the degree of association between the characters in the word segment. The greater the degree of association, the greater the possibility that the training word segment is a new word. The semantic independence of the words corresponding to the training word fragments is obtained by counting the related information of finer-grained words in the training word fragments. The semantic independence of the words can reflect the semantic completeness of the word fragments. The more complete the semantics, the greater the possibility that the training word fragment is a new word. The word fragment vector corresponding to the training word fragment is determined based on the internal correlation and semantic independence of the words corresponding to the training word fragments. The word fragment vector contains information such as internal correlation and semantic independence of the words that are helpful for new word recognition. The word fragment vector and training label corresponding to the training word fragment are used as training data to train the new word recognition model, which can enable the new word recognition model to learn the knowledge related to the new words in the training data to recognize the new words. The accuracy and efficiency of new word recognition can be improved by using the trained new word recognition model to perform new word recognition.

[0165] In a specific embodiment, the method of the present application can be applied to new word recognition for a specific field. Figure 6 As shown, the new word recognition method of the present application includes the following steps:

[0166] 1. Domain Corpus Acquisition

[0167] The computer device acquires multiple corpora belonging to the same field to form a domain corpus set A, and the domain corpus set A is used for model training.

[0168] 2. Domain Corpus Segmentation

[0169] The computer device uses n-gram to segment the domain corpus A to obtain n-words, which are training word fragments. The value of n is generally less than or equal to 4. For example, 2-gram, 3-gram, and 4-gram are used to segment the domain corpus A to obtain fragments including 2 characters, fragments including 3 characters, and fragments including 4 characters. It can be understood that the probability that a fragment including 1 character is a new word is small, so it is not necessary to obtain a fragment including 1 character.

[0170] 3. Feature construction

[0171] Referring to Table 1, for any training word fragment, the computer device calculates various statistical features corresponding to the training word fragment, including part-of-speech value, word external association, word internal association, word TF-IDF value, word length value (i.e., fragment length), word semantic independence, and word text independence.

[0172] Table 1

[0173]

[0174] The part-of-speech value refers to the number of part-of-speech representations of a word fragment in the domain corpus A. Generally, the lower the part-of-speech value of a word fragment, the more stable the word fragment is, and the greater the possibility that the word fragment becomes a word.

[0175] The external relevance of a word refers to how many different expressions the left and right sides of a word fragment appear in the domain corpus A. The greater the external relevance of a word fragment, the richer the surrounding words of the word fragment, and the greater the possibility that the word fragment becomes a word.

[0176] The calculation formula for the internal correlation of a word is: The greater the internal correlation of a word fragment, the more stable the internal correlation is, and the greater the possibility that the word fragment will become a word.

[0177] The word TF-IDF value is used to measure the importance of a word fragment in the domain corpus A. Generally, new words have more important meanings. The larger the word TF-IDF value of a word fragment, the greater the possibility that the word fragment is a new word.

[0178] The word length value refers to how many characters the word consists of. Because word fragments of different lengths have different distributions of internal word associations, usually, the longer the word fragment is, the greater the internal word association is. Adding the word length value feature can give full play to the positive impact of the internal word association on the model. Adding the word length value feature is to add more prior knowledge, so that the model can learn more useful information and let the model know what length of words should have what kind of internal word association. Word fragments that meet this pattern are more likely to be real words, avoiding model misjudgment. Regarding word semantic independence, the greater the word semantic independence of a word fragment, the more stable the word fragment, the more complete the semantics, and the greater the possibility that the word fragment is a new word.

[0179] The calculation formula of word semantic independence is: The greater the semantic independence of a word fragment, the more stable the word fragment is, the more complete its semantics is, and the greater the possibility that it is a new word.

[0180] The calculation formula of word text independence value is S = min{D word, k=D(word,k),k∈Q}. New words usually have certain novelty and independence. The larger the word text independence value of a word fragment is, the higher the text independence of the word fragment is, and the greater the probability that the word fragment is a new word is.

[0181] 4. Normalization

[0182] The computer equipment normalizes the above-mentioned features to avoid the weight of a single feature being too large or too small, which would affect the recognition accuracy of the model.

[0183] 5. Model training

[0184] As shown in Table 1, the "lable" column is the training word label corresponding to the segment including n characters, "0" represents that the training word segment is not a word, "1" represents that the training word segment is a word, but not a new word, and "2" represents that the training word segment is a word and a new word. The computer device reads the normalized data from the second to the eighth columns in Table 1, converts them into an array format, as the feature value array X; then reads the value of the ninth column of Table 1, converts them into an array format, as the label array Y, the data in the label array Y corresponds to the data in the feature value array X one by one, and the corresponding data is the data corresponding to the same training word segment. Split the feature value array X into a training set X_train and a test set X_test, and split the label array Y into a training set Y_train and a test set Y_test. Perform model training based on the training sets X_train and Y_train. After the training, perform model testing based on the test sets X_test and Y_test, and use the model that passes the test as the final qualified model.

[0185] 6. Model prediction

[0186] Select a batch of new corpora that belong to the same field as the corpora in the domain corpus set A to form the domain corpus set B. Use n-gram to segment the corpus in the domain corpus set B to obtain segments consisting of n characters. Each segment consisting of n characters is used as a word segment to be identified. The statistical features corresponding to the word segments to be identified include part-of-speech value, word external association, word internal association, word TF-IDF value, word length value (i.e., segment length), word semantic independence, and word text independence. Calculate the feature value corresponding to each word segment to be identified, input the feature value into the qualified training model, and obtain the predicted word label corresponding to each word segment to be identified. When predicting the model, the value of n can be greater than 4, so the model can identify words of different lengths or new words.

[0187] By using the new word recognition method of the present application to recognize new words, it is possible to accurately recognize whether a certain word segment is a word or a new word, with high accuracy. Moreover, there is no restriction on the length of the word segment when predicting the model, and the scope of application is relatively wide. Moreover, the calculation method of various statistical features is not complex, and the use of the model for new word recognition at the same time helps to ensure the efficiency of new word recognition.

[0188] In a specific embodiment, Figure 7 As shown, the new word recognition method of the present application can be applied to construct a domain dictionary for a specific field. For example, by using the new word recognition method of the present application to recognize new words, new words of different lengths in the medical field can be recognized, and the recognized new words can be added to the existing domain dictionary in the medical field to enrich the content of the domain dictionary. For example, by using the new word recognition method of the present application to recognize new words, words and new words of different lengths in the financial field can be recognized, and the recognized words and new words can be combined to form a domain dictionary for word segmentation of financial field corpus.

[0189] Further, such as Figure 7As shown, the domain dictionary constructed by the new word recognition method of the present application can be applied to construct a knowledge graph in a specific field, and the constructed knowledge graph can be further applied to build various types of answering dialogue robots in a specific field. For example, in the construction process of various types of answering dialogue robots in the financial field (such as bank sales robots, bank consulting robots, and bank operation robots), it is necessary to build a complete knowledge graph to improve the service quality of the answering dialogue robot. The knowledge graph determines the comprehensiveness of the knowledge reserve of the answering dialogue robot. In the construction process of the knowledge graph, large and comprehensive entity information is required. The domain dictionary in the financial field constructed by the new word recognition method of the present application can be used to assist the entity information in the financial field. For example, the knowledge graph can be a question-answer pair consisting of questions and answers. The establishment of questions and answers involves words. In the knowledge graph, it can be multiple questions corresponding to one answer, or it can be one question corresponding to one answer. The more complete the relevant knowledge graph is, the more the robot knows, and the better it can serve customers' questions or inquiries.

[0190] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0191] Based on the same inventive concept, the embodiment of the present application also provides a new word recognition device for implementing the new word recognition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more new word recognition device embodiments provided below can refer to the limitations of the new word recognition method above, and will not be repeated here.

[0192] In one embodiment, Figure 8 As shown, a new word recognition device is provided, including: an acquisition module 802, a determination module 804 and a recognition module 806, wherein:

[0193] The acquisition module 802 is used to acquire the target word segment and the first corpus to which the target word segment belongs.

[0194] Determination module 804 is used to obtain the internal word association corresponding to the target word fragment based on the number of occurrences of the target character in the target word fragment in the first corpus and the number of occurrences of the character fragment containing the target character in the target word fragment in the first corpus; obtain the word semantic independence corresponding to the target word fragment based on the number of occurrences of the target word fragment in the first corpus and the number of occurrences of subwords in the target word fragment in the first corpus; subwords are words segmented from the target word fragment; and determine the word fragment vector corresponding to the target word fragment based on at least the internal word association and word semantic independence corresponding to the target word fragment.

[0195] The recognition module 806 is used to perform new word recognition on the target word segment based on the word segment vector to obtain a new word recognition result corresponding to the target word segment.

[0196] In one embodiment, the acquisition module 802 is further used to:

[0197] Acquire a first corpus; each corpus in the first corpus belongs to the same field; segment each corpus in the first corpus based on a preset segmentation length set to obtain a candidate word fragment set that matches each segmentation length in the preset segmentation length set; each candidate word fragment in the same candidate word fragment set has the same fragment length; determine a target word fragment from each candidate word fragment set.

[0198] In one embodiment, the determination module 804 is further configured to:

[0199] Each character in the target word segment is taken as the target character in turn; for any target character, the target character and the target character's adjacent backward characters in the target word segment are combined into a character segment containing the target character in the target word segment; the adjacent backward characters are characters that are adjacent to the target character and located after the target character in the target word segment; according to the appearance frequency ratio between the appearance frequency of the character segment containing the target character in the target word segment in the first corpus and the appearance frequency ratio of the target character in the target word segment in the first corpus, the appearance frequency ratio corresponding to each character segment in the target word segment is obtained respectively; the appearance frequency ratio corresponding to each character segment is smoothed to obtain the internal word association corresponding to each character segment; according to the internal word association corresponding to each character segment, the internal word association corresponding to the target word segment is obtained.

[0200] In one embodiment, the determination module 804 is further configured to:

[0201] Segment the target word fragment to obtain the subwords contained in the target word fragment; obtain the fused occurrence count according to the number of occurrences of each subword of the target word fragment in the first corpus; adjust the number of occurrences of the target word fragment in the first corpus based on the total number of subwords corresponding to the target word fragment to obtain the reference occurrence count; determine the difference in occurrence count between the fused occurrence count and the reference occurrence count; obtain the word semantic independence corresponding to the target word fragment based on the number of occurrences of the target word fragment in the first corpus and the difference in occurrence count.

[0202] In one embodiment, the determination module 804 is further configured to:

[0203] A candidate word fragment set to which a target word fragment belongs is obtained; the fragment lengths of the candidate word fragments in the candidate word fragment set are consistent, and the candidate word fragment set is obtained by segmenting the corpus in the first corpus set; at least one reference word fragment corresponding to the target word fragment is determined from the candidate word fragment set; a target distance is determined from the edit distances between the target word fragment and the corresponding at least one reference word fragment, and the word text independence corresponding to the target word fragment is determined based on the target distance; a word fragment vector corresponding to the target word fragment is determined based at least on the word internal association, word semantic independence and word text independence corresponding to the target word fragment.

[0204] In one embodiment, the determination module 804 is further configured to:

[0205] Obtain supplementary feature sets corresponding to the target word fragments respectively; the supplementary feature sets include at least one of the word external association degree, part-of-speech value, corpus importance, and fragment length corresponding to the target word fragment; the word external association degree is obtained according to the number of adjacent forward deduplicated characters and the number of adjacent backward deduplicated characters corresponding to the target word fragment in the first corpus, the number of adjacent forward deduplicated characters is obtained by performing character type statistics on adjacent forward characters corresponding to the target word fragment in the first corpus, the adjacent forward characters are characters adjacent to the target word fragment in the first corpus and located before the target word fragment, and the number of adjacent backward deduplicated characters is obtained by performing character type statistics on adjacent backward characters corresponding to the target word fragment in the first corpus. The adjacent backward characters are the characters adjacent to the target word fragment and located after the target word fragment in the first corpus; the part-of-speech value is obtained by counting the part-of-speech reflected by the target word fragment in the first corpus; the corpus importance is obtained according to the first frequency and the second frequency corresponding to the target word fragment in the first corpus, the first frequency is the frequency of occurrence of the target word fragment in the corpus containing the target word fragment, the corpus containing the target word fragment is in the first corpus, and the second frequency is the frequency of occurrence of the corpus containing the target word fragment in the first corpus; at least based on the internal correlation degree of the word corresponding to the target word fragment, the semantic independence degree of the word and the supplementary feature set, the word fragment vector corresponding to the target word fragment is determined.

[0206] In one embodiment, the identification module 806 is further configured to:

[0207] The word fragment vector is input into the target new word recognition model, the predicted word label corresponding to the target word fragment is output, and the new word recognition result corresponding to the target word fragment is determined based on the predicted word label corresponding to the target word fragment; the predicted word label includes any one of a first label used to indicate that the word fragment is a non-word, a second label used to indicate that the word fragment is a word but not a new word, and a third label used to indicate that the word fragment is a new word.

[0208] In one embodiment, the new word identification device is further used to:

[0209] Based on the first target word segment and the second target word segment corresponding to the first corpus, the domain dictionary corresponding to the domain to which the first corpus belongs is determined; the predicted word label corresponding to the first target word segment is a label used to indicate that the word segment belongs to a word but is not a new word, and the predicted word label corresponding to the second target word segment is a label used to indicate that the word segment belongs to a new word.

[0210] The above-mentioned new word identification device obtains the internal word association corresponding to the target word segment by counting the relevant information of the characters in the target word segment. The internal word association can reflect the degree of association between the characters in the word segment. The greater the degree of association, the greater the possibility that the target word segment is a new word. The semantic independence of the word corresponding to the target word segment is obtained by counting the relevant information of finer-grained words in the target word segment. The semantic independence of the word can reflect the semantic completeness of the word segment. The more complete the semantics, the greater the possibility that the target word segment is a new word. Based on the internal word association and semantic independence corresponding to the target word segment, a word segment vector corresponding to the target word segment is established. The word segment vector contains information such as internal word association and semantic independence that are helpful for new word identification. Identifying whether the target word segment is a new word based on the word segment vector corresponding to the target word segment can effectively improve the accuracy of new word identification.

[0211] Based on the same inventive concept, the embodiment of the present application also provides a new word recognition model training device for implementing the new word recognition model training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more new word recognition model training device embodiments provided below can refer to the limitations of the new word recognition model training method above, and will not be repeated here.

[0212] In one embodiment, Fig. 9 As shown, a new word recognition model training device is provided, including: a data acquisition module 902, a vector determination module 904 and a model training module 906, wherein:

[0213] A data acquisition module 902 is used to acquire a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs;

[0214] The vector determination module 904 is used to obtain the internal word association corresponding to the training word segment based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus; obtain the word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of the subwords in the training word segment in the second corpus; the subwords are words segmented from the training word segment; and determine the word segment vector corresponding to the training word segment based on at least the internal word association and the word semantic independence corresponding to the training word segment;

[0215] The model training module 906 is used to train the initial new word recognition model based on the word segment vectors corresponding to the training word segments and the training word labels to obtain a target new word recognition model.

[0216] The above-mentioned new word recognition model training device obtains the internal word association corresponding to the training word segment by counting the relevant information of the characters in the training word segment. The internal word association can reflect the degree of association between the characters in the word segment. The greater the degree of association, the greater the possibility that the training word segment is a new word. The semantic independence of the word corresponding to the training word segment is obtained by counting the relevant information of finer-grained words in the training word segment. The semantic independence of the word can reflect the semantic completeness of the word segment. The more complete the semantics, the greater the possibility that the training word segment is a new word. Based on the internal word association and semantic independence corresponding to the training word segment, a word segment vector corresponding to the training word segment is established. The word segment vector contains information such as internal word association and semantic independence that are helpful for new word recognition. The word segment vector corresponding to the training word segment and the training label are used as training data to train the new word recognition model, which can enable the new word recognition model to learn the knowledge related to the new word in the training data to recognize the new word. The new word recognition model that has been trained can improve the accuracy and efficiency of new word recognition by performing new word recognition.

[0217] Each module in the above-mentioned new word recognition device and new word recognition model training device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0218] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as corpora and new word recognition models. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a new word recognition method and a new word recognition model training method are implemented.

[0219] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a new word recognition method and a new word recognition model training method are implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0220] Those skilled in the art will understand that Fig.10 , 11The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0221] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0222] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0223] In one embodiment, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0224] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0225] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0226] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0227] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A new word recognition method, It is characterized in that The method comprises: Acquire a target word segment and a first corpus to which the target word segment belongs; Based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus, obtaining the internal word association corresponding to the target word segment; Based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of subwords in the target word segment in the first corpus, obtaining the word semantic independence corresponding to the target word segment; the subwords are words segmented from the target word segment; Determining a word fragment vector corresponding to the target word fragment based at least on the word internal association and word semantic independence corresponding to the target word fragment; New word recognition is performed on the target word segment based on the word segment vector to obtain a new word recognition result corresponding to the target word segment; a new word refers to a word that is not recorded in the standard word library.

2. The method according to claim 1, It is characterized in that The step of obtaining a target word segment and a first corpus to which the target word segment belongs includes: Acquire a first corpus; each corpus in the first corpus belongs to the same field; Based on a preset segmentation length set, segment each corpus in the first corpus set to obtain a candidate word segment set that matches each segmentation length in the preset segmentation length set; each candidate word segment in the same candidate word segment set has the same segment length; Determine the target word segment from each set of candidate word segments.

3. The method according to claim 1, It is characterized in that The obtaining the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus includes: Taking each character in the target word segment as a target character in turn; For any target character, the target character and the target character's adjacent backward character in the target word segment are combined into a character segment containing the target character in the target word segment; the adjacent backward character is a character adjacent to the target character in the target word segment and located after the target character; Obtaining the occurrence ratios of each character segment in the target word segment according to the occurrence ratios of the character segment containing the target character in the target word segment in the first corpus and the occurrence ratios of the target character in the target word segment in the first corpus; Smoothing the occurrence ratios corresponding to the character segments to obtain the internal correlations of the words corresponding to the character segments; The internal word association degree corresponding to the target word segment is obtained according to the internal word association degrees corresponding to the respective character segments.

4. The method according to claim 1, It is characterized in that The obtaining the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of subwords in the target word segment in the first corpus includes: Segmenting the target word segment to obtain subwords contained in the target word segment; Obtaining a fused occurrence count according to the occurrence counts of each subword of the target word segment in the first corpus; Based on the total number of subwords corresponding to the target word segment, adjusting the number of occurrences of the target word segment in the first corpus to obtain a reference number of occurrences; determining a difference in occurrence count between the fused occurrence count and the reference occurrence count; Based on the number of occurrences of the target word segment in the first corpus and the difference in the number of occurrences, the word semantic independence corresponding to the target word segment is obtained.

5. The method according to claim 1, It is characterized in that The determining the word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment includes: Acquire a candidate word fragment set to which the target word fragment belongs; each candidate word fragment in the candidate word fragment set has the same fragment length, and the candidate word fragment set is obtained by segmenting the corpus in the first corpus set; Determine at least one reference word segment corresponding to the target word segment from the candidate word segment set; Determine a target distance from the edit distances between the target word segment and at least one corresponding reference word segment, and determine the word text independence corresponding to the target word segment based on the target distance; The word fragment vector corresponding to the target word fragment is determined based at least on the word internal association, word semantic independence and word text independence corresponding to the target word fragment.

6. The method according to claim 1, It is characterized in that The determining the word segment vector corresponding to the target word segment based at least on the word internal association and word semantic independence corresponding to the target word segment includes: Obtain supplementary feature sets corresponding to the target word fragments respectively; the supplementary feature sets include at least one of the word external association degree, part-of-speech value, corpus importance, and fragment length corresponding to the target word fragment; the word external association degree is obtained according to the number of adjacent forward deduplicated characters and the number of adjacent backward deduplicated characters corresponding to the target word fragment in the first corpus, the number of adjacent forward deduplicated characters is obtained by performing character type statistics on adjacent forward characters corresponding to the target word fragment in the first corpus, the adjacent forward characters are characters adjacent to the target word fragment in the first corpus and located before the target word fragment, and the number of adjacent backward deduplicated characters is a number of characters corresponding to the target word fragment in the first corpus. The word importance is obtained by performing character type statistics on the corresponding adjacent backward characters in the first corpus, wherein the adjacent backward characters are characters in the first corpus that are adjacent to the target word segment and located after the target word segment; the part-of-speech value is obtained by counting the part-of-speech reflected by the target word segment in the first corpus; the corpus importance is obtained according to a first frequency and a second frequency corresponding to the target word segment in the first corpus, wherein the first frequency is the frequency of occurrence of the target word segment in the corpus containing the target word segment, and the corpus containing the target word segment is located in the first corpus, and the second frequency is the frequency of occurrence of the corpus containing the target word segment in the first corpus; The word fragment vector corresponding to the target word fragment is determined based at least on the word internal association, word semantic independence and a supplementary feature set corresponding to the target word fragment.

7. The method according to claim 1, It is characterized in that The performing new word recognition on the target word segment based on the word segment vector to obtain a new word recognition result corresponding to the target word segment includes: The word fragment vector is input into a target new word recognition model, and a predicted word label corresponding to the target word fragment is output; and a new word recognition result corresponding to the target word fragment is determined based on the predicted word label corresponding to the target word fragment; the predicted word label includes any one of a first label for indicating that the word fragment is a non-word, a second label for indicating that the word fragment is a word but not a new word, and a third label for indicating that the word fragment is a new word.

8. The method according to claim 1, It is characterized in that The method further comprises: Based on the first target word segment and the second target word segment corresponding to the first corpus, a domain dictionary corresponding to the domain to which the first corpus belongs is determined; the predicted word label corresponding to the first target word segment is a label used to indicate that the word segment belongs to a word but is not a new word, and the predicted word label corresponding to the second target word segment is a label used to indicate that the word segment belongs to a new word.

9. A new word recognition model training method, It is characterized in that The method comprises: Obtaining a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs; Based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus, obtaining the internal word association corresponding to the training word segment; Based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of subwords in the training word segment in the second corpus, obtaining the word semantic independence corresponding to the training word segment; the subwords are words segmented from the training word segment; Determining a word segment vector corresponding to the training word segment based at least on the word internal association and word semantic independence corresponding to the training word segment; Based on the word segment vectors and training word labels corresponding to the training word segments, an initial new word recognition model is trained to obtain a target new word recognition model; a new word refers to a word that is not recorded in a standard word library.

10. A new word recognition device, It is characterized in that The device comprises: An acquisition module, used for acquiring a target word segment and a first corpus to which the target word segment belongs; A determination module, configured to obtain the internal word association corresponding to the target word segment based on the number of occurrences of the target character in the target word segment in the first corpus and the number of occurrences of the character segment containing the target character in the target word segment in the first corpus; obtain the word semantic independence corresponding to the target word segment based on the number of occurrences of the target word segment in the first corpus and the number of occurrences of the subword in the target word segment in the first corpus; the subword is a word segmented from the target word segment; and determine the word segment vector corresponding to the target word segment based at least on the internal word association and the word semantic independence corresponding to the target word segment; The recognition module is used to perform new word recognition on the target word segment based on the word segment vector to obtain a new word recognition result corresponding to the target word segment; a new word refers to a word that is not recorded in a standard word library.

11. A new word recognition model training device, It is characterized in that The device comprises: A data acquisition module, used to acquire a training word segment, a training word label corresponding to the training word segment, and a second corpus to which the training word segment belongs; A vector determination module, configured to obtain the internal word association corresponding to the training word segment based on the number of occurrences of the target character in the training word segment in the second corpus and the number of occurrences of the character segment containing the target character in the training word segment in the second corpus; obtain the word semantic independence corresponding to the training word segment based on the number of occurrences of the training word segment in the second corpus and the number of occurrences of subwords in the training word segment in the second corpus; the subwords are words segmented from the training word segment; and determine the word segment vector corresponding to the training word segment based at least on the internal word association and the word semantic independence corresponding to the training word segment; The model training module is used to train the initial new word recognition model based on the word segment vectors and training word labels corresponding to the training word segments to obtain a target new word recognition model; new words refer to words that are not recorded in the standard vocabulary.

12. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

14. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images