New word discovery method, apparatus, device, and storage medium

Abstract

Embodiments of the present application disclose a new word discovery method, device and equipment and a storage medium, which can be applied in the fields of artificial intelligence, cloud technology, database and the like. The method comprises the following steps: determining an initial feature sequence corresponding to each character in a to-be-predicted sentence, inputting the initial feature sequence into a language model to obtain a predicted feature sequence corresponding to each character in the to-be-predicted sentence, and determining a new word in the to-be-predicted sentence based on the predicted feature sequence. According to the embodiments of the present application, the new word can be determined efficiently and accurately, and the applicability is high.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, device, and storage medium for discovering new words. Background Technology

[0002] In the era of rapid internet development, with the lowering of the threshold for content production, the amount of various types of content being published is growing exponentially. Instantly discovering new words in massive amounts of content has become a top priority in the content processing stage.

[0003] Common methods for discovering new words often rely on statistical approaches or model predictions to identify new words within massive amounts of content. The former is inefficient, while the latter, because new words are often newly emerging, makes it difficult for models to accurately identify them. Therefore, how to efficiently and accurately identify new words within massive amounts of content has become a pressing problem to be solved. Summary of the Invention

[0004] This application provides a new word discovery method, apparatus, device, and storage medium, which can efficiently and accurately determine new words and has high applicability.

[0005] On the one hand, embodiments of this application provide a new word discovery method, which includes:

[0006] Determine the initial feature sequence corresponding to each character in the sentence to be predicted, input the initial feature sequence into the language model to obtain the predicted feature sequence corresponding to each character in the sentence to be predicted; determine the new words in the sentence to be predicted based on the predicted feature sequence.

[0007] The language model mentioned above was trained in the following way:

[0008] Obtain the training sample set, which includes multiple sample statements;

[0009] For each of the above sample statements, determine the initial word sequence of the sample statement, replace the words to be predicted in the initial word sequence with preset mask characters to obtain the first word sequence, determine the initial character sequence of the sample statement, replace the words to be predicted in the initial character sequence corresponding to the words to be predicted with the preset mask characters to obtain the first character sequence, and determine the sample feature sequence corresponding to the sample statement based on the first word sequence and the first character sequence of the sample statement.

[0010] For each sample sentence, the sample feature sequence corresponding to the sample sentence is input into the initial language model to obtain the sample prediction feature sequence corresponding to each word in the sample sentence. Based on the above sample prediction feature sequence, the prediction word corresponding to the word to be predicted in the sample sentence is determined.

[0011] Based on the word to be predicted and the corresponding predicted word in each of the above sample sentences, the training loss value is determined. The initial language model is iteratively trained according to the training loss value and the training sample set until the training loss value meets the training termination condition. The model at the end of training is then determined as the language model.

[0012] On the other hand, embodiments of this application provide a new word discovery device, which includes:

[0013] The feature processing module is used to determine the initial feature sequence corresponding to each character in the sentence to be predicted, and input the initial feature sequence into the language model to obtain the predicted feature sequence corresponding to each character in the sentence to be predicted.

[0014] The new word determination module is used to determine new words in the above-mentioned sentence to be predicted based on the above-mentioned prediction feature sequence.

[0015] The language model described above is obtained by training using a model training device, which is used for:

[0016] Obtain the training sample set, which includes multiple sample statements;

[0017] For each of the above sample statements, determine the initial word sequence of the sample statement, replace the words to be predicted in the initial word sequence with preset mask characters to obtain the first word sequence, determine the initial character sequence of the sample statement, replace the words to be predicted in the initial character sequence corresponding to the words to be predicted with the preset mask characters to obtain the first character sequence, and determine the sample feature sequence corresponding to the sample statement based on the first word sequence and the first character sequence of the sample statement.

[0018] For each sample sentence, the sample feature sequence corresponding to the sample sentence is input into the initial language model to obtain the sample prediction feature sequence corresponding to each word in the sample sentence. Based on the above sample prediction feature sequence, the prediction word corresponding to the word to be predicted in the sample sentence is determined.

[0019] Based on the word to be predicted and the corresponding predicted word in each of the above sample sentences, the training loss value is determined. The initial language model is iteratively trained according to the training loss value and the training sample set until the training loss value meets the training termination condition. The model at the end of training is then determined as the language model.

[0020] On the other hand, embodiments of this application provide an electronic device, including a processor and a memory, which are interconnected;

[0021] The aforementioned memory is used to store computer programs;

[0022] The processor described above is configured to execute the new word discovery method provided in the embodiments of this application when the computer program described above is invoked.

[0023] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program that is executed by a processor to implement the novel word discovery method provided in embodiments of this application.

[0024] On the other hand, embodiments of this application provide a computer program product, which includes a computer program or computer instructions, and the new word discovery method provided in embodiments of this application is executed by the computer program or computer instructions.

[0025] In this embodiment, since the sample feature sequence corresponding to the sample statement is determined by the first word sequence and the first character sequence corresponding to the sample statement, including preset mask characters, the sample features corresponding to each character in the sample statement can include not only the relevant information of the character but also the relevant information of the word in which the character belongs. Based on this, the language model trained based on the sample feature sequence can determine the prediction feature sequence corresponding to the statement to be predicted by fully considering the meaning of each character in the word to be predicted and the contextual semantics. Thus, based on the prediction feature sequence, new words in the statement to be predicted can be accurately and efficiently determined, with high applicability. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the new word discovery method provided in an embodiment of this application;

[0028] Figure 2 This is a flowchart illustrating the language model training method provided in an embodiment of this application;

[0029] Figure 3a This is a schematic diagram illustrating a scenario for determining the first word sequence provided in an embodiment of this application;

[0030] Figure 3b This is a schematic diagram illustrating a scenario for determining the first character sequence provided in an embodiment of this application;

[0031] Figure 4 This is a schematic diagram of a scenario for determining sample feature sequences provided in an embodiment of this application;

[0032] Figure 5This is a schematic diagram of a scenario of the initial language model provided in an embodiment of this application;

[0033] Figure 6 This is a schematic diagram of the structure of the new word discovery device provided in the embodiments of this application;

[0034] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0036] The new word discovery method provided in this application embodiment is applicable to various fields such as artificial intelligence, big data, and cloud technology. For example, it can discover new words in text based on technologies such as machine learning (ML) and natural language processing (NLP) in the field of artificial intelligence.

[0037] Artificial intelligence (AI) encompasses the theories, methods, technologies, and application systems that utilize digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. For example, embodiments of this application can obtain the language model used in the new word discovery process through machine learning.

[0038] Natural Language Processing (NLP) is an important area within computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close connection with linguistic research.

[0039] The new word discovery method provided in this application can be executed by any terminal device or server. When the new word discovery method provided in this application is executed by a server, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. When the new word discovery method provided in this application is executed by a terminal device, the terminal device can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, vehicle terminal, etc., but is not limited to these.

[0040] See Figure 1 , Figure 1 This is a flowchart illustrating the new word discovery method provided in an embodiment of this application. For example... Figure 1 As shown, the new word discovery method provided in this application includes the following steps:

[0041] Step S11: Determine the initial feature sequence corresponding to each character in the sentence to be predicted, and input the initial feature sequence into the language model to obtain the predicted feature sequence corresponding to each character in the sentence to be predicted.

[0042] The language model provided in this embodiment is obtained through pre-training; the specific training method of this language model can be found in [link to relevant documentation]. Figure 2 , Figure 2 This is a flowchart illustrating the language model training method provided in an embodiment of this application. Figure 2 As shown, the language model training method provided in this application includes the following steps:

[0043] Step S21: Obtain the training sample set, which includes multiple sample statements.

[0044] In some feasible implementations, relevant information from vertical websites can be obtained using big data and web crawling technologies, such as content from gaming websites, video websites, or animation websites, as training corpora. Big data refers to data sets that cannot be captured, managed, and processed within a certain timeframe using conventional software tools. It represents massive, rapidly growing, and diverse information assets that require new processing models to achieve stronger decision-making, insightful discovery, and process optimization capabilities. Big data can effectively obtain the training corpora needed to construct training samples based on technologies such as large-scale parallel processing databases, data mining, distributed file systems, distributed databases, and the aforementioned cloud computing.

[0045] Optionally, daily incremental news and hourly trending news can also be used as training corpora, as well as audio information such as radio and television news converted into text information for use as training corpora. Alternatively, text content accumulated during or historically during the information flow distribution process can also be used as training corpora, such as text content from self-media accounts, user comments, bullet comments, e-books, or various applications.

[0046] The specific source of the training corpus used to construct the training sample set can be determined based on the needs of the actual application scenario, and is not restricted here. Furthermore, because authors may hide text information in the feed corpus to increase the weight and chance of their content being recommended, such as by setting the font color and background to be the same or using hidden tags, it is necessary to remove the hidden text when obtaining the training corpus to eliminate its negative impact.

[0047] Furthermore, after obtaining the training corpus, the repetitive or similar text segments in the training corpus can be filtered to obtain a filtered corpus set. Specifically, similar or identical text segments can be determined by determining the similarity between each text segment, or by determining the simhash value of each text segment. If the simhash value of two text segments is less than a preset threshold, then the two text segments can be determined as similar segments, thereby filtering out repetitive and similar text segments from the training corpus based on the above method.

[0048] The determination of similarity or simhash values ​​between text paragraphs can be achieved through cloud technology. Cloud computing refers to obtaining necessary resources on demand and in a scalable manner via a network. It is a product of the integration and development of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing.

[0049] In some feasible implementations, after obtaining the filtered corpus set, sentences with text lengths within a preset length range in the training corpus set can be used as sample sentences, and the set of sample sentences can be used as the training sample set when training the language model.

[0050] Step S22: For each sample statement, determine the initial word sequence of the sample statement, replace the words to be predicted in the initial word sequence with preset mask characters to obtain the first word sequence, determine the initial character sequence of the sample statement, replace the words to be predicted in the initial character sequence corresponding to the words to be predicted with preset mask characters to obtain the first character sequence, and based on the first word sequence and the first character sequence of the sample statement, determine the sample feature sequence corresponding to each character in the sample statement.

[0051] In some feasible implementations, before the model training process, for each sample sentence, the initial word sequence of the sample sentence can be determined, and the words to be predicted in the initial word sequence can be replaced with preset mask characters to obtain the first word sequence.

[0052] Specifically, the sample statement can be segmented to obtain an initial word sequence. Then, any target word sequence with a preset text length within the initial word sequence can be identified; that is, the target word sequence is a part of the initial word sequence. Further, words belonging to a preset word type within the target word sequence can be identified as words to be predicted, and each word to be predicted can be replaced with a preset mask character.

[0053] The preset word types mentioned above include, but are not limited to, nouns, verbs, adjectives, and gerunds, which can be determined based on the actual application scenario requirements and are not restricted here. The preset mask characters mentioned above can be any characters used to replace the word to be predicted, such as replacing the word to be predicted with [MASK], and are not restricted here.

[0054] like Figure 3a As shown, Figure 3a This is a schematic diagram illustrating a scenario for determining the first word sequence provided in this application embodiment. If a sample statement is "The weather will be very good tomorrow", then the initial word sequence "The weather will be very good tomorrow" is obtained by segmenting the sample statement. If "weather" is determined to be the word to be predicted and [MASK] is a preset mask character, then the word to be predicted "weather" can be replaced with the preset mask character to obtain the first word sequence "The weather will be very good tomorrow" corresponding to the sample statement.

[0055] Optionally, a predetermined proportion of words belonging to a preset type in the initial word sequence of the sample statement can be replaced with other words that do not belong to any word in the initial word sequence to obtain a second word sequence. Then, a target word sequence with a preset text length can be determined from any text in the second word sequence. Words belonging to a preset word type in the target word sequence are identified as words to be predicted, and each word to be predicted is replaced with a preset mask character to obtain a first word sequence.

[0056] In some feasible embodiments, before the model training process, for each sample statement, the initial character sequence of the sample statement can also be determined, that is, the sample statement is arranged in units of characters to obtain the initial character sequence.

[0057] Furthermore, each predicted character corresponding to the predicted word of the sample statement in the initial character sequence is determined, that is, the characters of the predicted word in the initial word sequence used to form the sample statement in the initial character sequence are determined as the predicted characters. For example, if the predicted word in the initial word sequence of the sample statement "The temperature will be very low tomorrow" is "temperature", then the predicted characters in the initial character sequence of the sample statement are "qi" and "wen". After determining the predicted characters in the initial character sequence, the predicted characters in the initial character sequence can also be replaced with a preset mask character, so as to obtain the first character sequence of the sample statement.

[0058] As Figure 3b shown, Figure 3b is a schematic diagram of the scenario for determining the first character sequence provided by an embodiment of the present application. If a sample statement is "The weather will be very good tomorrow", [MASK] is a preset mask character and the predicted word in the initial word sequence of the sample statement is "weather", then the initial character sequence "The weather will be very good tomorrow" of the sample statement can be determined, and the predicted characters corresponding to the predicted word in the initial character sequence are replaced with the preset mask character, resulting in the first character sequence "The MASKMASK will be very good tomorrow" corresponding to the sample statement.

[0059] Optionally, if when determining the first word sequence of the sample statement, in addition to replacing the predicted word with a preset mask character, some words belonging to a preset type are also replaced with other words, then when determining the first character sequence of the sample statement, the characters corresponding to the replaced words in the initial character sequence need to be replaced with the corresponding characters to obtain the second character sequence, and then the predicted characters corresponding to the predicted word in the second character sequence are replaced with the preset mask character to obtain the first character sequence of the sample statement.

[0060] In some feasible embodiments, for any sample statement, since the input of the model is related features represented by vectors, after determining the first word sequence and the first character sequence of the sample statement, the sample feature sequence corresponding to each character in the sample statement can be determined based on the first word sequence and the first character sequence of the sample statement, so as to use the sample feature sequence corresponding to the sample statement as the model input.

[0061] Specifically, each word in the first word sequence of the sample sentence and the preset mask character can be encoded respectively to obtain the initial word features corresponding to each word in the first word sequence. Moreover, each character in the first character sequence of the sample sentence and the preset mask character can be encoded respectively to obtain the initial character features corresponding to each character in the first character sequence. Further, based on the initial word features corresponding to each word in the first word sequence and the initial character features corresponding to each character in the first character sequence, the sample feature sequence corresponding to each character in the sample sentence is determined.

[0062] Among them, when determining the sample feature sequence corresponding to each character in the sample sentence based on the initial word features corresponding to each word in the first word sequence and the initial character features corresponding to each character in the first character sequence, for each character in the first character sequence, the initial word feature corresponding to this character in the first word sequence and the target initial word feature of the target word corresponding to this character in the first word sequence can be determined. Then, based on the initial character feature and the target initial word feature corresponding to this character, the fusion feature corresponding to this character is determined. Based on this, the fusion features corresponding to each character in the first character sequence can be obtained. Further, based on the fusion features corresponding to each character in the first character sequence, the sample feature sequence corresponding to each character in the sample sentence is determined.

[0063] For example, the first character sequence is "tomorrow MASK MASK very low", and the first word sequence is "tomorrow MASK very low". For the character "tian" in the first character sequence, the fusion feature corresponding to "tian" can be determined based on the initial character feature corresponding to "tian" and the target initial word feature of the target word "tomorrow" corresponding to "tian". For any "MASK" in the first character sequence, the fusion feature corresponding to "MASK" in the first character sequence can be determined based on its corresponding initial character feature and the initial word feature corresponding to "MASK" in the first word sequence.

[0064] Among them, for each character, the initial character feature and the target initial word feature corresponding to this character can be concatenated to obtain the fusion feature corresponding to this character. Alternatively, the initial character feature and the target initial word feature corresponding to this character can be fused and encoded to obtain the fusion feature corresponding to this character, which can be specifically determined according to the requirements of the actual application scenario and is not limited here. Based on this method, the fusion feature of each character in the sample sentence can include not only the self-information of this character but also the relevant information of the word where this character is located. Further, in the model training process, the inherent information of this character can be fully understood and the semantic information of this character in the sample sentence can be learned.

[0065] Among them, after determining the integration features corresponding to each character in the first character sequence of the sample statement, the feature sequence composed of the integration features corresponding to each character in the first character sequence can be determined as the sample feature sequence corresponding to each character in the sample statement. Alternatively, the integration features corresponding to each character in the first character sequence can be further encoded, and based on the encoding result, the sample feature sequence corresponding to each character in the sample statement can be obtained.

[0066] See Figure 4 , Figure 4 is a schematic diagram of the scenario for determining the sample feature sequence provided by an embodiment of the present application. As Figure 4 shown, the sample statement is "Tomorrow the weather is very good", the first character sequence corresponding to the sample statement is "Tomorrow MASK MASK very good", and the corresponding first word sequence is "Tomorrow MASK very good". By encoding each character in the first character sequence through the character fine-grained encoding layer, the initial character features "T0, T1, T2, T3, T4, T5" corresponding to each character in the first character sequence are obtained. By encoding each word in the first word sequence through the word coarse-grained encoding layer, the initial word features "S0, S1, S2" corresponding to each word in the first word sequence are obtained.

[0067] Furthermore, for each character in the first character sequence, based on the initial character feature T0 corresponding to "Tomorrow" and the corresponding target word feature S0, the integration feature H0 corresponding to "Tomorrow" can be obtained; based on the initial character feature T1 corresponding to "day" and the corresponding target word feature S0, the integration feature H1 corresponding to "day" can be obtained; based on the initial character feature T2 corresponding to the first MASK and the word feature S1 of the MASK in the corresponding first word sequence, the integration feature H2 corresponding to the first MASK can be obtained; based on the initial character feature T3 corresponding to the second MASK and the word feature S1 of the MASK in the corresponding first word sequence, the integration feature H3 corresponding to the second can be obtained; based on the initial character feature T4 corresponding to "very" and the corresponding target word feature S2, the integration feature H4 corresponding to "very" can be obtained; based on the initial character feature T5 corresponding to "good" and the corresponding target word feature S2, the integration feature H5 corresponding to "good" can be obtained. Based on the above integration features, the sample feature sequence "H0, H1, H2, H3, H4, H5" corresponding to the sample statement can be obtained.

[0068] Step S23: For each sample statement, input the sample feature sequence corresponding to the sample statement into the initial language model to obtain the sample prediction feature sequence corresponding to each character in the sample statement, and determine the predicted word corresponding to the word to be predicted in the sample statement based on the sample prediction feature sequence.

[0069] In some feasible implementations, the initial language model described above can be a language prediction model, including but not limited to models with word prediction capabilities such as the Bidirectional Encoder Representation from Transformers (BERT) model, which is not limited here.

[0070] The initial language model mentioned above can be a combination of a language prediction model and a Conditional Random Field (CRF) model, such as a combination of the BERT model and a CRF model.

[0071] For each sample sentence in the training sample set, sequence labeling can be performed to obtain word labeling information corresponding to each character in the sample sentence. The word labeling information for each character in the sample sentence indicates whether the character is a component of a word in the sentence, and if so, its position within the word. For example, the first character of each word in the sample sentence can be labeled "B," indicating the beginning of the word; the other characters in each word can be labeled "I," indicating they are not part of the beginning of the word; other characters that are not words can be labeled "O"; and the last character in the sample sentence can be labeled "E," indicating it is the end character of the sample sentence.

[0072] Furthermore, the sample feature sequence and corresponding word annotation information corresponding to the sample sentence are input into the initial language model. The language prediction model in the initial language model determines the sample prediction feature sequence corresponding to the sample sentence, and the word annotation information corresponding to each sample feature in the sample feature sequence is learned through conditional random fields. Thus, based on the initial language model, the predicted word annotation information of each predicted character corresponding to the sample prediction feature sequence can be determined. Since each predicted character corresponding to the sample prediction feature sequence includes predicted words, the words in the predicted sentence composed of each predicted character can be determined based on the predicted word annotation information. Based on this, one training of the initial language model can be completed, enabling the initial language model to predict the predicted words in the sample sentence.

[0073] See Figure 5 , Figure 5 This is a schematic diagram of a scenario of the initial language model provided in an embodiment of this application. Figure 5The initial language model shown includes a BERT model and a Conditional Random Field (CRF). A sample feature sequence corresponding to a given sentence is input into the initial language model. The BERT model obtains the predicted feature sequence for that sentence. The Softmax function determines the predicted word corresponding to each predicted feature sequence, and the CRF determines the word tagging information for each predicted word. Based on the word tagging information of each predicted word, the words in the predicted sentence composed of these predicted words are determined, and the predicted word corresponding to the word to be predicted in the sample sentence can also be determined from this information.

[0074] Step S24: Based on the word to be predicted and the corresponding predicted word in each sample sentence, determine the training loss value, and iteratively train the initial language model according to the training loss value and the training sample set until the training loss value meets the training termination condition. Then, determine the model at the end of training as the language model.

[0075] In some feasible implementations, the predicted word corresponding to the word to be predicted in each sample sentence can be determined based on the initial language model. Therefore, based on the word to be predicted and the corresponding predicted word in each sample sentence, the training loss value corresponding to the training process of the initial language model can be determined.

[0076] The training loss value described above can characterize the difference between the word to be predicted and the corresponding predicted word in each sample sentence. Furthermore, the training loss value can be determined based on the negative log-likelihood function, cross-entropy loss function, etc., and is not limited here.

[0077] Furthermore, the initial language model is iteratively trained based on the training loss value and the sample sentences in the training sample set. During the iterative training process, the various model parameters of the initial language model are continuously adjusted until the training loss value meets the training termination condition. The model at the end of training is then determined as the final language model.

[0078] The training termination conditions mentioned above can be that the training loss value tends to stabilize, or that multiple consecutive continuation loss values ​​are less than a preset threshold, or that the difference between multiple consecutive training loss values ​​and the previous training loss value is less than a preset difference, etc. The specific conditions can be determined based on the actual application scenario requirements, and are not limited here.

[0079] Optionally, when the initial language model includes a conditional random field, the first training loss value can be determined based on the word to be predicted and the corresponding predicted word in each sample sentence. The second training loss value is then determined based on the word annotation information corresponding to each sample sentence and the word annotation information determined by the initial language model. This second training loss value characterizes the difference between the true word representation information of the sample sentence and the word annotation information determined by the initial language model. Specifically, it can also be determined based on the negative log-likelihood function or the cross-entropy loss function, etc., without limitation.

[0080] Furthermore, the training loss value during the model training process can be determined based on the first training loss value and the second training loss value, so as to iteratively train the initial language model according to the training loss value and the sample sentences in the training sample set, and when the training loss value meets the end of training conditions, the model at the end of training is determined as the final language model.

[0081] Specifically, the sum of the first training loss value and the second training loss value can be determined as the training loss value during the model training process, or the loss weight corresponding to the first training loss value and the weight corresponding to the second training loss value can be determined, and then the sum of the weights of the first training loss value and the second training loss value can be determined as the training loss value during the model training process.

[0082] It should be noted that the above implementation of determining the training loss value during model training is merely an example, and the specific implementation can be determined based on the actual application scenario requirements, without any restrictions.

[0083] Based on the above training method, the final language model can predict words and determine the word labeling information of the sentence to be predicted. Therefore, when the sentence to be predicted includes uncommon words or new words, the language model can accurately distinguish between uncommon words and new words in the sentence to be predicted.

[0084] Meanwhile, since training corpora within the same domain tend to have similar text types, while training corpora from different domains exhibit significant differences in text types (e.g., training corpora from the gaming domain and urban traffic domain have vastly different text types), training sample sets can be constructed based on training corpora from the same domain, thus yielding training sample sets for different domains. Furthermore, the same initial language model can be trained separately on training sample sets from different domains to obtain a language model suitable for that domain. The implementation method of training the initial language model based on training samples from any one domain will not be elaborated upon here.

[0085] Step S12: Determine new words in the sentence to be predicted based on the predicted feature sequence.

[0086] In some feasible implementations, based on Figure 2 After training the final language model as shown, for any sentence to be predicted, the initial feature sequence corresponding to each character in the sentence can be determined, and the initial feature sequence is input into the language model to obtain the predicted feature sequence corresponding to each character in the sentence. Then, based on the predicted feature sequence, the word labeling information corresponding to each character in the sentence can be determined.

[0087] Based on the word tagging information of each character in the sentence to be predicted, candidate words can be determined. For each character in the sentence to be predicted, based on the corresponding word tagging information, it can be determined whether the character belongs to a word in the sentence. If the character belongs to a word, its specific position within the word can be determined based on the word tagging information. Therefore, each word in the sentence to be predicted can be determined in this way. However, since word tagging information can only determine the words in the sentence to be predicted, but cannot determine whether the word is a new word, the words determined from the sentence based on word tagging information can be considered as candidate words for new words.

[0088] Furthermore, candidate words identified from the statement to be predicted can be matched with a pre-defined lexicon. For any candidate word, if it is included in the pre-defined lexicon, it is determined to be an existing word. If it is not included in the pre-defined lexicon, it is determined to be a new word. Thus, new words in the statement to be predicted can be identified based on the language model and the pre-defined lexicon.

[0089] In some feasible implementations, the preset thesaurus in this application embodiment can be stored in a server, database, cloud storage, or blockchain. When it is necessary to match any candidate word with the preset thesaurus, the candidate word can be directly searched in the storage space of the preset thesaurus to determine whether the preset thesaurus is included.

[0090] In short, a database can be viewed as an electronic filing cabinet—a place to store electronic files. In this application, it can be used to store a pre-defined lexicon and the training sample set used during model training. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods.

[0091] In this application, each data block in the blockchain can store a pre-defined vocabulary and the training sample set used during model training. Cloud storage, an extension and development of cloud computing, refers to the collaborative storage of a large number of various types of storage devices (also known as storage nodes) across a network through application software or interfaces, using cluster applications, grid technology, and distributed storage file systems to jointly store the pre-defined vocabulary and the training sample set used during model training.

[0092] In some feasible implementations, the aforementioned preset vocabulary is a vocabulary database that includes common or frequently used words. It can be constructed by acquiring words through big data, web crawling technology, or based on dictionary data, and there are no restrictions on this.

[0093] It can also periodically discover popular or new words from original corpora in different fields to contribute to the construction and updating of the preset lexicon.

[0094] Specifically, within the same domain, certain new words tend to appear in clusters. For example, science fiction often features authors creating numerous new words, while social media often sees users creating new terms. Therefore, multiple candidate texts potentially containing new words can be selected from the original corpora of different domains. Furthermore, for candidate texts in any given domain, the word features of each character can be determined. Based on these word features, the target word can be identified from the candidate texts in that domain. If the pre-defined vocabulary does not include the target word, it can be added as a new word to update the pre-defined vocabulary.

[0095] Among them, the word features mentioned above include, but are not limited to, one or more of word frequency, mutual information, cohesion, and information entropy. The specific features can be determined based on the actual application scenario requirements and are not limited here.

[0096] As an example, if the frequency of a certain character combination in the candidate text of a certain field is higher than the preset word frequency, then the character combination can be identified as the target word. If the preset word does not include the target word, the preset word can be updated based on the target word.

[0097] Alternatively, the term frequency-inverse document frequency (TF-IDF) weight corresponding to the character combination can be determined based on the term frequency of the character combination. If the TF-IDF weight of the character combination is higher than the preset weight, the character combination can be identified as the target word. If the preset word does not include the target word, the preset word can be updated based on the target word.

[0098] As an example, the cohesion degree corresponding to each character combination in each domain can be determined. Cohesion degree represents the tightness between the characters in a character combination. The higher the cohesion degree, the more likely the character combination is to be a new word. Therefore, character combinations with a cohesion degree greater than a preset cohesion degree can be identified as target words, and if the preset word does not include the target word, the preset word can be updated based on the target word.

[0099] As an example, mutual information reflects the probability of each character in a combination of two characters appearing simultaneously. The higher the mutual information value, the higher the correlation between the two characters, and the greater the probability of forming a new word. Therefore, the mutual information between adjacent words in candidate texts in various fields can be determined. If the mutual information value is greater than the preset mutual information value, the character combination can be identified as a target word. Furthermore, if the preset word does not include the target word, the preset word can be updated based on the target word.

[0100] As an example, the frequency of occurrence of the left and right neighboring characters of any character combination in candidate texts across various domains can be determined. Then, the left information entropy is determined based on the frequency of occurrence of the left neighboring characters and the frequency of occurrence of the character combination, and the right information entropy is determined based on the frequency of occurrence of the right neighboring characters and the frequency of occurrence of the character combination. The larger the left and right entropy values, the richer the surrounding words of the character combination, meaning a greater degree of freedom for the character combination and a higher probability that it will become an independent word. If both the left and right information entropies are greater than a preset threshold, the character combination can be determined as a target word, and if the preset word is not included in the preset dictionary, the preset dictionary can be updated based on the target word.

[0101] As an example, after determining multiple target words based on the aforementioned methods, since the same target word can be determined simultaneously using multiple methods, for each target word, the corresponding determination methods and their weights can be determined, and the sum of the weights of each determination method can be determined. Target words with a weight sum higher than a preset threshold are then matched with a preset vocabulary. If the preset vocabulary does not include the target word, the preset vocabulary is updated based on that target word.

[0102] As an example, after determining multiple target words based on any one or more of the above methods, the target words can be encoded to obtain encoding features. Then, based on each encoding feature, the target words can be clustered to identify potential new words in each set of target words. If the target word is not included in the preset word library, the preset word library can be updated based on the target word.

[0103] Optionally, given the rapid development of social media, news from various platforms often generates numerous comments from netizens, potentially leading to the emergence of new terms within these comments. Therefore, it is possible to obtain comment information corresponding to media news from various fields, such as comments in chat communities and bullet comments on video websites, without any restrictions.

[0104] Furthermore, a first and second original word can be identified from each comment. The first and second original words are any two different words in each comment. For example, names in each comment can be identified based on named entity recognition, and two of these names can be identified as the first and second original words. After identifying the first and second original words, the target comment information that includes both the first and second original words, as well as the total number and total number of likes corresponding to the target comment information, can be determined. Then, based on the number and number of likes corresponding to the target comment information, the evaluation score of the combination word composed of the first and second original words can be determined.

[0105] The higher the evaluation score, the greater the acceptance of the combined words, and the more likely it is to become a new word. Therefore, if the evaluation score of a combined word is greater than the preset evaluation score, it can be added as a new word to the preset word library. Based on the above implementation, word combinations in each comment can be traversed to obtain as many new words generated from the media comment information as possible.

[0106] In this embodiment, since the sample feature sequence corresponding to the sample statement is determined by the first word sequence and the first character sequence corresponding to the sample statement, including preset mask characters, the sample features corresponding to each character in the sample statement can include not only the relevant information of the character but also the relevant information of the word to which the character belongs. Based on this, the language model trained based on the sample feature sequence can determine the prediction feature sequence corresponding to the statement to be predicted, given that it fully understands the meaning of each character in the word to be predicted and the meaning of the statement to be predicted. Therefore, it can accurately and efficiently determine new words in the statement to be predicted based on the prediction feature sequence. It has high applicability.

[0107] See Figure 6 , Figure 6 This is a schematic diagram of the structure of the new word discovery device provided in this application embodiment. The new word discovery device provided in this application embodiment includes:

[0108] Feature processing module 61 is used to determine the initial feature sequence corresponding to each character in the sentence to be predicted, and input the initial feature sequence into the language model to obtain the predicted feature sequence corresponding to each character in the sentence to be predicted.

[0109] The new word determination module 62 is used to determine new words in the above-mentioned sentence to be predicted based on the above-mentioned prediction feature sequence;

[0110] The language model described above is obtained by training using a model training device, which is used for:

[0111] Obtain the training sample set, which includes multiple sample statements;

[0112] For each of the above sample statements, determine the initial word sequence of the sample statement, replace the words to be predicted in the initial word sequence with preset mask characters to obtain the first word sequence, determine the initial character sequence of the sample statement, replace the words to be predicted in the initial character sequence corresponding to the words to be predicted with the preset mask characters to obtain the first character sequence, and determine the sample feature sequence corresponding to the sample statement based on the first word sequence and the first character sequence of the sample statement.

[0113] For each sample sentence, the sample feature sequence corresponding to the sample sentence is input into the initial language model to obtain the sample prediction feature sequence corresponding to each word in the sample sentence. Based on the above sample prediction feature sequence, the prediction word corresponding to the word to be predicted in the sample sentence is determined.

[0114] Based on the word to be predicted and the corresponding predicted word in each of the above sample sentences, the training loss value is determined. The initial language model is iteratively trained according to the training loss value and the training sample set until the training loss value meets the training termination condition. The model at the end of training is then determined as the language model.

[0115] In some feasible implementations, for each of the above sample statements, the above model training apparatus is used to:

[0116] Each word in the first word sequence of the sample statement and the preset mask character are encoded to obtain the initial word features corresponding to each word in the first word sequence.

[0117] Encode each character in the first character sequence of the sample statement and the preset mask character to obtain the initial character features corresponding to each character in the first character sequence.

[0118] Based on the initial word features corresponding to each word in the first word sequence and the initial character features corresponding to each character in the first character sequence, the sample feature sequence corresponding to the sample statement is determined.

[0119] In some feasible implementations, for each of the above sample statements, the above model training apparatus is used to:

[0120] For each character in the first character sequence, determine the target initial word feature of the target word corresponding to that character in the first word sequence, and determine the fusion feature corresponding to that character based on the initial character feature and the target initial word feature.

[0121] Based on the fusion features corresponding to each character in the first character sequence, the sample feature sequence corresponding to the sample statement is determined.

[0122] In some feasible implementations, the aforementioned new word determination module 62 is used for:

[0123] Based on the above predicted feature sequence, the word labeling information of each character in the above sentence to be predicted is determined;

[0124] Based on the above word annotation information, candidate words in the above-mentioned sentence to be predicted were determined;

[0125] If a candidate word is not included in the preset word library, then the candidate word is determined to be a new word.

[0126] In some feasible implementations, the above-described model training apparatus is used for:

[0127] Determine any target word sequence in the above initial word sequence whose text length is a preset text length;

[0128] Words belonging to the preset word type in the above target word sequence are identified as words to be predicted, and each of the above words to be predicted is replaced with a preset mask character.

[0129] In some feasible implementations, the new word discovery device further includes a new word update module 63, which is further configured to:

[0130] Candidate texts were identified from raw corpora from different fields;

[0131] For candidate texts in any domain, determine the word features of each character in the candidate texts in that domain, and determine the first target word from the candidate texts in that domain based on the word features. The word features include at least one of word frequency, mutual information, cohesion, and information entropy.

[0132] The preset vocabulary is updated based on the first target word.

[0133] In some feasible implementations, the aforementioned new word update module 63 is also used for:

[0134] Obtain comment information corresponding to media information in various fields, and determine the first original word and the second original word from each of the above comment information, wherein the first original word and the second original word are any two different words in each of the above comment information;

[0135] Identify target comment information that includes both the first original word and the second original word in each of the aforementioned comment information;

[0136] Based on the number of target comment information and the number of likes corresponding to the target comment information, the evaluation score of the combined words corresponding to the first original word and the second original word is determined;

[0137] If the above evaluation score is greater than the preset evaluation score, then the preset word library will be updated based on the above combination of words.

[0138] In practice, the aforementioned new word discovery device can perform the above-described actions through its built-in functional modules. Figure 1 and / or Figure 2 The implementation methods provided for each step are detailed in the above-mentioned implementation methods, and will not be repeated here.

[0139] See Figure 7 , Figure 7 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 7 As shown, the electronic device 1000 in this embodiment may include: a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the electronic device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to implement communication between these components. The user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1004 may be a high-speed RAM or non-volatile memory, such as at least one disk storage device. The memory 1005 may optionally be at least one storage device located remotely from the processor 1001. Figure 7 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0140] exist Figure 7 In the illustrated electronic device 1000, the network interface 1004 provides network communication functionality; the user interface 1003 is primarily used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0141] Determine the initial feature sequence corresponding to each character in the sentence to be predicted, input the initial feature sequence into the language model to obtain the predicted feature sequence corresponding to each character in the sentence to be predicted; determine the new words in the sentence to be predicted based on the predicted feature sequence.

[0142] The processor 1001 mentioned above is used for training the language model as follows:

[0143] Obtain the training sample set, which includes multiple sample statements;

[0144] For each of the above sample statements, determine the initial word sequence of the sample statement, replace the words to be predicted in the initial word sequence with preset mask characters to obtain the first word sequence, determine the initial character sequence of the sample statement, replace the words to be predicted in the initial character sequence corresponding to the words to be predicted with the preset mask characters to obtain the first character sequence, and determine the sample feature sequence corresponding to the sample statement based on the first word sequence and the first character sequence of the sample statement.

[0145] For each sample sentence, the sample feature sequence corresponding to the sample sentence is input into the initial language model to obtain the sample prediction feature sequence corresponding to each word in the sample sentence. Based on the above sample prediction feature sequence, the prediction word corresponding to the word to be predicted in the sample sentence is determined.

[0146] Based on the word to be predicted and the corresponding predicted word in each of the above sample sentences, the training loss value is determined. The initial language model is iteratively trained according to the training loss value and the training sample set until the training loss value meets the training termination condition. The model at the end of training is then determined as the language model.

[0147] In some feasible implementations, for each of the above sample statements, the processor 1001 is used to:

[0148] Each word in the first word sequence of the sample statement and the preset mask character are encoded to obtain the initial word features corresponding to each word in the first word sequence.

[0149] Encode each character in the first character sequence of the sample statement and the preset mask character to obtain the initial character features corresponding to each character in the first character sequence.

[0150] Based on the initial word features corresponding to each word in the first word sequence and the initial character features corresponding to each character in the first character sequence, the sample feature sequence corresponding to the sample statement is determined.

[0151] In some feasible implementations, for each of the above sample statements, the processor 1001 is used to:

[0152] For each character in the first character sequence, determine the target initial word feature of the target word corresponding to that character in the first word sequence, and determine the fusion feature corresponding to that character based on the initial character feature and the target initial word feature.

[0153] Based on the fusion features corresponding to each character in the first character sequence, the sample feature sequence corresponding to the sample statement is determined.

[0154] In some feasible implementations, the processor 1001 described above is used for:

[0155] Based on the above predicted feature sequence, the word labeling information of each character in the above sentence to be predicted is determined;

[0156] Based on the above word annotation information, candidate words in the above-mentioned sentence to be predicted were determined;

[0157] If a candidate word is not included in the preset word library, then the candidate word is determined to be a new word.

[0158] In some feasible implementations, the processor 1001 described above is used for:

[0159] Determine any target word sequence in the above initial word sequence whose text length is a preset text length;

[0160] Words belonging to the preset word type in the above target word sequence are identified as words to be predicted, and each of the above words to be predicted is replaced with a preset mask character.

[0161] In some feasible implementations, the processor 1001 is further configured to:

[0162] Candidate texts were identified from raw corpora from different fields;

[0163] For candidate texts in any domain, determine the word features of each character in the candidate texts in that domain, and determine the first target word from the candidate texts in that domain based on the word features. The word features include at least one of word frequency, mutual information, cohesion, and information entropy.

[0164] The preset vocabulary is updated based on the first target word.

[0165] In some feasible implementations, the processor 1001 is further configured to:

[0166] Obtain comment information corresponding to media information in various fields, and determine the first original word and the second original word from each of the above comment information, wherein the first original word and the second original word are any two different words in each of the above comment information;

[0167] Identify target comment information that includes both the first original word and the second original word in each of the aforementioned comment information;

[0168] Based on the number of target comment information and the number of likes corresponding to the target comment information, the evaluation score of the combined words corresponding to the first original word and the second original word is determined;

[0169] If the above evaluation score is greater than the preset evaluation score, then the preset word library will be updated based on the above combination of words.

[0170] It should be understood that in some feasible implementations, the processor 1001 described above may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information.

[0171] In specific implementation, the aforementioned electronic device 1000 can perform the above-described actions through its built-in functional modules. Figure 1 and / or Figure 2 The implementation methods provided for each step are detailed in the above-mentioned implementation methods, and will not be repeated here.

[0172] This application also provides a computer-readable storage medium storing a computer program that is executed by a processor to implement... Figure 1 and / or Figure 2 The methods provided in each step are detailed in the implementation methods provided in the above steps, and will not be repeated here.

[0173] The aforementioned computer-readable storage medium can be an internal storage unit of any of the aforementioned new word discovery devices and / or electronic devices, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device. The aforementioned computer-readable storage medium can also include magnetic disks, optical disks, read-only memory (ROM), or random access memory (RAM), etc. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the electronic device. The computer-readable storage medium is used to store the computer program and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0174] This application provides a computer program product, which includes a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor... Figure 1 and / or Figure 2 The methods provided for each step in the process.

[0175] The terms "first," "second," etc., used in the claims, description, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or electronic device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or electronic devices. References to "embodiment" herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The presentation of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. The term "and / or" as used in this application's description and appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0176] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Those skilled in the art can implement the described functions using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.

[0177] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A method for discovering new words, characterized in that, The method includes: Determine the initial feature sequence corresponding to each character in the sentence to be predicted, input the initial feature sequence into the language model to obtain the predicted feature sequence corresponding to each character in the sentence to be predicted, and determine the new words in the sentence to be predicted based on the predicted feature sequence; The language model is trained in the following way: A training sample set is obtained, which includes multiple sample sentences; different training sample sets correspond to different domains, and the same language model is trained based on the training sample sets of different domains respectively. For each sample statement, an initial word sequence is determined, and a target word sequence with a preset text length is identified within the initial word sequence. Words belonging to a preset word type in the target word sequence are identified as words to be predicted, and each word to be predicted is replaced with a preset mask character to obtain a first word sequence. An initial character sequence is determined, and characters in the initial character sequence that form the words to be predicted in the initial word sequence are identified as words to be predicted, and the words to be predicted corresponding to the words to be predicted in the initial character sequence are replaced with the preset mask character to obtain a first character sequence. Based on the first word sequence and the first character sequence of the sample statement, a sample feature sequence corresponding to the sample statement is determined. The preset word type includes at least one of nouns, verbs, adjectives, and gerunds. For each sample sentence, the sample feature sequence corresponding to the sample sentence is input into the initial language model to obtain the sample prediction feature sequence corresponding to each character in the sample sentence. Based on the sample prediction feature sequence, the predicted word corresponding to the word to be predicted in the sample sentence is determined. Based on the word to be predicted and the corresponding predicted word in each of the sample sentences, a training loss value is determined. The initial language model is iteratively trained according to the training loss value and the training sample set until the training loss value meets the training termination condition. The model at the end of training is then determined as the language model. Wherein, for each of the sample statements, determining the sample feature sequence corresponding to the sample statement based on the first word sequence and the first character sequence of the sample statement includes: Each word in the first word sequence of the sample statement and the preset mask characters are encoded to obtain the initial word features corresponding to each word in the first word sequence; Encode each character in the first character sequence of the sample statement and the preset mask character respectively to obtain the initial character features corresponding to each character in the first character sequence; For each character in the first character sequence, determine the target initial word feature of the target word corresponding to that character in the first word sequence, and determine the fusion feature corresponding to that character based on the initial character feature and the target initial word feature; Based on the fusion features corresponding to each character in the first character sequence, the sample feature sequence corresponding to the sample statement is determined.

2. The method according to claim 1, characterized in that, The step of determining new words in the sentence to be predicted based on the predicted feature sequence includes: Based on the predicted feature sequence, the word labeling information of each character in the sentence to be predicted is determined; Based on the word annotation information, candidate words in the sentence to be predicted are determined; If a candidate word is not included in the preset word library, then the candidate word is determined to be a new word.

3. The method according to claim 2, characterized in that, The method further includes: Candidate texts were identified from raw corpora from different fields; For candidate text in any domain, determine the word features of each character in the candidate text in that domain, and determine the first target word from the candidate text in that domain based on the word features. The word features include at least one of word frequency, mutual information, cohesion, and information entropy. The preset vocabulary is updated based on the first target word.

4. The method according to claim 2, characterized in that, The method further includes: Obtain comment information corresponding to media information in various fields, and determine a first original word and a second original word from each of the comment information, wherein the first original word and the second original word are any two different words in each of the comment information; Determine target comment information that includes both the first original word and the second original word in each of the aforementioned comment information; Based on the number of target comment information and the number of likes corresponding to the target comment information, the evaluation score of the combined word corresponding to the first original word and the second original word is determined; If the evaluation score is greater than the preset evaluation score, then the preset word library is updated based on the combined words.

5. A new word discovery device, characterized in that, The device includes: The feature processing module is used to determine the initial feature sequence corresponding to each character in the sentence to be predicted, and input the initial feature sequence into the language model to obtain the predicted feature sequence corresponding to each character in the sentence to be predicted; The new word determination module is used to determine new words in the sentence to be predicted based on the predicted feature sequence. The language model is obtained by training using a model training device, which is used for: A training sample set is obtained, which includes multiple sample sentences; different training sample sets correspond to different domains, and the same language model is trained based on the training sample sets of different domains respectively. For each sample statement, an initial word sequence is determined, and a target word sequence with a preset text length is identified within the initial word sequence. Words belonging to a preset word type in the target word sequence are identified as words to be predicted, and each word to be predicted is replaced with a preset mask character to obtain a first word sequence. An initial character sequence is determined, and the characters in the initial character sequence that form the words to be predicted in the initial word sequence are identified as words to be predicted, and the words to be predicted corresponding to the words to be predicted in the initial character sequence are replaced with the preset mask character to obtain a first character sequence. Based on the first word sequence and the first character sequence of the sample statement, a sample feature sequence corresponding to the sample statement is determined. The preset word type includes at least one of nouns, verbs, adjectives, and gerunds. For each sample sentence, the sample feature sequence corresponding to the sample sentence is input into the initial language model to obtain the sample prediction feature sequence corresponding to each character in the sample sentence. Based on the sample prediction feature sequence, the predicted word corresponding to the word to be predicted in the sample sentence is determined. Based on the word to be predicted and the corresponding predicted word in each of the sample sentences, a training loss value is determined. The initial language model is iteratively trained according to the training loss value and the training sample set until the training loss value meets the training termination condition. The model at the end of training is then determined as the language model. For each of the sample statements, the model training device is configured to: Each word in the first word sequence of the sample statement and the preset mask characters are encoded to obtain the initial word features corresponding to each word in the first word sequence; Encode each character in the first character sequence of the sample statement and the preset mask character respectively to obtain the initial character features corresponding to each character in the first character sequence; For each character in the first character sequence, determine the target initial word feature of the target word corresponding to that character in the first word sequence, and determine the fusion feature corresponding to that character based on the initial character feature and the target initial word feature; Based on the fusion features corresponding to each character in the first character sequence, the sample feature sequence corresponding to the sample statement is determined.

6. The new word discovery device according to claim 5, characterized in that, The new word determination module is used for: Based on the predicted feature sequence, the word labeling information of each character in the sentence to be predicted is determined; Based on the word annotation information, candidate words in the sentence to be predicted are determined; If a candidate word is not included in the preset word library, then the candidate word is determined to be a new word.

7. The new word discovery device according to claim 6, characterized in that, The new word discovery device further includes a new word updating module, which is used for: Candidate texts were identified from raw corpora from different fields; For candidate text in any domain, determine the word features of each character in the candidate text in that domain, and determine the first target word from the candidate text in that domain based on the word features. The word features include at least one of word frequency, mutual information, cohesion, and information entropy. The preset vocabulary is updated based on the first target word.

8. The new word discovery device according to claim 6, characterized in that, The new word discovery device further includes a new word updating module, which is used for: Obtain comment information corresponding to media information in various fields, and determine a first original word and a second original word from each of the comment information, wherein the first original word and the second original word are any two different words in each of the comment information; Determine target comment information that includes both the first original word and the second original word in each of the aforementioned comment information; Based on the number of target comment information and the number of likes corresponding to the target comment information, the evaluation score of the combined word corresponding to the first original word and the second original word is determined; If the evaluation score is greater than the preset evaluation score, then the preset word library is updated based on the combined words.

9. An electronic device, characterized in that, It includes a processor and a memory, which are interconnected; The memory is used to store computer programs; The processor is configured to perform the method as described in any one of claims 1 to 4 when the computer program is invoked.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method of any one of claims 1 to 4.

11. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, implement the method described in any one of claims 1 to 4.

Images