Language model training method, text classification method, device, equipment and medium

Abstract

The application discloses a language model training method, a text classification method, a device, equipment and a medium, and belongs to the technical field of text processing. The method comprises the following steps: acquiring a plurality of first texts; for any first text, acquiring the appearance probability of each character in the first text according to a first network model; adjusting the first network model based on the appearance probability of each character in the plurality of first texts to obtain a first language model, wherein the first language model comprises a first feature extraction model; acquiring the semantic feature of any first text according to the first feature extraction model; adjusting the first feature extraction model based on the semantic feature of the plurality of first texts to obtain a second feature extraction model; and determining a target language model based on the second feature extraction model. The application reduces the dependence of the model on training data, reduces the acquisition time of the training data, and improves the training speed of the model and the efficiency of text classification.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of text processing, and particularly relate to a language model training method, a text classification method, an apparatus, a device, and a medium. BACKGROUND

[0002] In the technical field of text processing, the task of determining the category to which a text belongs is also referred to as a text classification task, and the text classification task is the most basic task in natural language processing (NLP), and can be completed based on a language model.

[0003] A language model is a model for determining the occurrence probability of each character in a text. In related technologies, a large amount of text needs to be obtained as training data, and the language model is obtained by training the training data. When performing text classification, the text to be classified is input into the language model, the language model first extracts the semantic features of the text, then determines the occurrence probability of each character in the text based on the semantic features of the text, and outputs the occurrence probability of each character in the text. Then, the category to which the text belongs is determined based on the occurrence probability of each character in the text.

[0004] In the above technology, a large amount of time is spent on obtaining training data, and the training process has dependency on the training data, thereby affecting the training speed of the model and the text classification efficiency. SUMMARY

[0005] Embodiments of the present application provide a language model training method, a text classification method, an apparatus, a device, and a medium, which can be used to solve the problem of slow model training speed and low text classification efficiency caused by spending a large amount of time to obtain a large amount of training data in related technologies. The technical solution includes the following contents.

[0006] In one aspect, the embodiments of the present application provide a language model training method, which comprises:

[0007] obtaining a plurality of first texts;

[0008] for any one of the first texts, obtaining the occurrence probability of each character in the any one of the first texts according to a first network model;

[0009] adjusting the first network model based on the occurrence probability of each character in the plurality of first texts to obtain a first language model, the first language model comprising a first feature extraction model;

[0010] obtaining semantic features of the any one of the first texts according to the first feature extraction model;

[0011] adjust the first feature extraction model based on the semantic features of the plurality of first texts to obtain a second feature extraction model;

[0012] determine a target language model based on the second feature extraction model.

[0013] In another aspect, an embodiment of the present application provides a text classification method, which comprises:

[0014] obtaining a target text;

[0015] obtaining occurrence probabilities of each character in at least two reconstructed texts according to a target language model, the reconstructed texts comprising the target text and a candidate text category, the target language model being obtained according to the training method of any of the above language models;

[0016] determining a target reconstructed text from the at least two reconstructed texts based on the occurrence probabilities of each character in the at least two reconstructed texts;

[0017] determining the candidate text category in the target reconstructed text as the text category of the target text.

[0018] In another aspect, an embodiment of the present application provides a language model training device, which comprises:

[0019] an obtaining module configured to obtain a plurality of first texts;

[0020] the obtaining module is further configured to, for any one of the first texts, obtain occurrence probabilities of each character in the any one of the first texts according to a first network model;

[0021] an adjusting module configured to adjust the first network model based on the occurrence probabilities of each character in the plurality of first texts to obtain a first language model, the first language model comprising a first feature extraction model;

[0022] the obtaining module is further configured to obtain semantic features of the any one of the first texts according to the first feature extraction model;

[0023] the adjusting module is further configured to adjust the first feature extraction model based on the semantic features of the plurality of first texts to obtain a second feature extraction model;

[0024] a determining module configured to determine a target language model based on the second feature extraction model.

[0025] In a possible implementation, the adjusting module is configured to determine a loss value of each first text based on the semantic feature of the plurality of first texts; determine a loss value of the first feature extraction model based on the loss value of each first text; and adjust the first feature extraction model based on the loss value of the first feature extraction model to obtain a second feature extraction model.

[0026] In a possible implementation, the any first text is any original text or a replacement text corresponding to the any original text, and the replacement text corresponding to the any original text is a text obtained by replacing characters in the any original text.

[0027] The adjusting module is configured to determine a loss value of the any original text based on the semantic feature of each original text and the semantic feature of the replacement text corresponding to each original text; and determine a loss value of the replacement text corresponding to the any original text based on the semantic feature of each original text and the semantic feature of the replacement text corresponding to each original text.

[0028] In a possible implementation, the adjusting module is configured to determine a first similarity between the any original text and the replacement text corresponding to the any original text based on the semantic feature of the any original text and the semantic feature of the replacement text corresponding to the any original text; determine a second similarity between the any original text and other original texts based on the semantic feature of the any original text and the semantic feature of the other original texts, the other original texts being original texts other than the any original text among the plurality of original texts; determine a third similarity between the any original text and replacement texts corresponding to the other original texts based on the semantic feature of the any original text and the semantic feature of the replacement texts corresponding to the other original texts; and determine a loss value of the any original text based on the first similarity, the second similarity, and the third similarity.

[0029] In a possible implementation, the adjusting module is configured to determine a first similarity between the any one original text and the replacement text corresponding to the any one original text based on the semantic feature of the any one original text and the semantic feature of the replacement text corresponding to the any one original text; determine a fourth similarity between the replacement text corresponding to the any one original text and other original texts based on the semantic feature of the replacement text corresponding to the any one original text and the semantic feature of the other original texts, the other original texts being original texts other than the any one original text among the original texts; determine a fifth similarity between the replacement text corresponding to the any one original text and the replacement texts corresponding to the other original texts based on the semantic feature of the replacement text corresponding to the any one original text and the semantic feature of the replacement texts corresponding to the other original texts; and determine a loss value of the replacement text corresponding to the any one original text based on the first similarity, the fourth similarity, and the fifth similarity.

[0030] In a possible implementation, the adjusting module is configured to determine predicted information of each first text based on the semantic feature of the first text, the predicted information of the first text being a probability of each character in the first text being replaced by prediction; obtain labeled information of each first text, the labeled information of the first text being information of whether each character in the first text is replaced by labeling; and adjust the first feature extraction model based on the predicted information of each first text and the labeled information of each first text to obtain a second feature extraction model.

[0031] In a possible implementation, the determining module is configured to obtain a second text and a category label of the second text; obtain a probability of each character in at least two spliced texts based on a second language model, the spliced text including the second text and a candidate text category, the second language model including the second feature extraction model; determine a target spliced text from the at least two spliced texts based on the probability of each character in the at least two spliced texts; and adjust the second language model based on the candidate text category in the target spliced text and the category label of the second text to obtain the target language model.

[0032] In a possible implementation, the determining module is configured to splice the second text and the candidate text category to obtain any one of the at least two spliced texts; input the any one spliced text into the second language model to determine a text vector of the any one spliced text by the second language model, and determine the probability of each character in the any one spliced text based on the text vector of the any one spliced text.

[0033] In a possible implementation, the determining module is configured to input the second text into the second language model to determine a text vector of the second text by the second language model; splice the text vector of the second text and a text vector of the candidate text category to obtain a text vector of any one of the at least two spliced texts by the second language model; and determine, by the second language model, an occurrence probability of each character in the any one spliced text based on the text vector of the any one spliced text.

[0034] In a possible implementation, the determining module is configured to determine perplexity of each spliced text based on the occurrence probability of each character in the at least two spliced texts, where the perplexity of the spliced text represents a fluency degree of the spliced text; and determine, based on the perplexity of each spliced text, that a spliced text corresponding to the perplexity that meets a condition is the target spliced text.

[0035] In a possible implementation, the determining module is configured to determine an occurrence probability of each spliced text based on the occurrence probability of each character in the at least two spliced texts; and determine the perplexity of each spliced text based on the occurrence probability of each spliced text.

[0036] In a possible implementation, the apparatus further includes:

[0037] The obtaining module is further configured to obtain the semantic feature of the any one first text according to the second network model.

[0038] The adjusting module is further configured to adjust the second network model based on the semantic features of the plurality of first texts to obtain a third feature extraction model.

[0039] The constructing module is configured to construct the first network model based on the third feature extraction model.

[0040] In another aspect, an embodiment of the present application provides a language model training apparatus, and the apparatus includes:

[0041] The obtaining module is configured to obtain a target text.

[0042] The obtaining module is further configured to obtain an occurrence probability of each character in at least two reconstructed texts according to a target language model, where the reconstructed texts include the target text and a candidate text category, and the target language model is obtained according to the language model training method described above.

[0043] The determining module is configured to determine a target reconstructed text from the at least two reconstructed texts based on the occurrence probability of each character in the at least two reconstructed texts.

[0044] The determining module is further configured to determine a candidate text category in the target reconstructed text as the text category of the target text.

[0045] In a possible implementation, the obtaining module is configured to splice the target text and the candidate text category to obtain any one of the at least two reconstructed texts; input the any one of the reconstructed texts into the target language model to determine a text vector of the any one of the reconstructed texts by the target language model; and determine the appearance probability of each character in the any one of the reconstructed texts based on the text vector of the any one of the reconstructed texts.

[0046] In a possible implementation, the obtaining module is configured to input the target text into the target language model to determine a text vector of the target text by the target language model; splice, by the target language model, the text vector of the target text and a text vector of the candidate text category to obtain a text vector of any one of the at least two reconstructed texts; and determine, by the target language model, the appearance probability of each character in the any one of the reconstructed texts based on the text vector of the any one of the reconstructed texts.

[0047] In another aspect, an electronic device is provided, which includes a processor and a memory, and the memory stores at least one piece of program code, which is loaded and executed by the processor, so that the electronic device implements the language model training method or the text classification method described above.

[0048] In another aspect, a computer readable storage medium is also provided, which stores at least one piece of program code, which is loaded and executed by a processor, so that the computer implements the language model training method or the text classification method described above.

[0049] In another aspect, a computer program or computer program product is also provided, which stores at least one piece of computer instruction, which is loaded and executed by a processor, so that the computer implements the language model training method or the text classification method described above.

[0050] The technical solutions provided in the embodiments of the present application at least bring the following beneficial effects:

[0051] The technical solution provided in this application involves first training a first language model using multiple first texts. This first language model includes a first feature extraction model. Then, the first feature extraction model is adjusted using the semantic features of the multiple first texts obtained from the first feature extraction model to obtain a second feature extraction model. Finally, a target language model is obtained based on the second feature extraction model. This achieves two-stage training of the language model using first texts, enabling the training of a highly accurate language model with a small amount of first texts. It reduces the model's dependence on training data, thereby reducing the time required to acquire training data and improving the training speed and text classification efficiency. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram illustrating the implementation environment of a language model training method or a text classification method provided in an embodiment of this application.

[0054] Figure 2 This is a flowchart of a language model training method provided in an embodiment of this application;

[0055] Figure 3 This is a schematic diagram of the structure of a BERT model provided in an embodiment of this application;

[0056] Figure 4 This is a schematic diagram illustrating the processing of original text and replacement text provided in an embodiment of this application;

[0057] Figure 5 This is a schematic diagram of a soft coding method provided in an embodiment of this application;

[0058] Figure 6 This is a flowchart of a text classification method provided in an embodiment of this application;

[0059] Figure 7 This is a schematic diagram illustrating the information processing of a target language model according to an embodiment of this application;

[0060] Figure 8 This is a schematic diagram of the structure of a language model training device provided in an embodiment of this application;

[0061] Figure 9 This is a schematic diagram of the structure of a text classification device provided in an embodiment of this application;

[0062] Figure 10 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0063] Figure 11 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the terms involved in the various optional embodiments of this application will be explained below.

[0065] A language model (LM), also known as a standard language model, is a probability distribution model that aims to evaluate the probability p(S) of generating any string in a language, where S = (w1, w2, w3, ..., wn), S is a string containing n characters, and wi is the i-th character of the string.

[0066] Few-Shot Learning (FSL): Given a task-specific dataset DT containing a small amount of supervised information and an auxiliary dataset DA unrelated to T, the goal of few-shot learning is to construct a function f for task T. This function utilizes the limited supervised information in DT and the knowledge in DA to map the input to the target. In short, few-shot learning aims to learn a model to solve a problem using a small number of samples. When the available labeled dataset DT is empty, this task is called zero-shot learning (ZSL).

[0067] Pre-trained models: These are task-independent models trained from large-scale data through self-supervised learning. Because they learn a wealth of information during pre-training, these models, after fine-tuning, can be used for specific tasks.

[0068] Text classification refers to the process of processing natural language text T into a computer-recognizable encoding through steps such as text preprocessing, feature extraction, text representation, and classifier construction, and then inputting the input text into a classifier F to classify the input text into one or more classes from a specified set of categories G. Common text classification tasks include sentiment analysis, news classification, and spam filtering.

[0069] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0070] Figure 1 This is a schematic diagram illustrating the implementation environment of a language model training method or text classification method provided in this application embodiment, such as... Figure 1The implementation environment shown includes an electronic device 11. The language model training method or the text classification method in the embodiments of the present application can be executed by the electronic device 11. Exemplarily, the electronic device 11 can include at least one of a terminal device or a server.

[0071] The terminal device can be at least one of a smartphone, a game console, a desktop computer, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, and a laptop computer.

[0072] The server can be a server, or a server cluster composed of multiple servers, or any one of a cloud computing platform and a virtualization center, and the embodiments of the present application do not limit this. The server can be communicatively connected to the terminal device through a wired network or a wireless network. The server can have functions of data processing, data storage, and data transceiving, and the embodiments of the present application do not limit this.

[0073] The optional embodiments of the present application can be implemented based on artificial intelligence technology. Artificial intelligence (AI) is a theory, method, technology and application system for using a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive an environment, acquire knowledge and use the knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0074] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware-level technologies and software-level technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, autonomous driving, intelligent transportation, etc.

[0075] Nature Language Processing (NLP) is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, the research in this field will involve natural language, i.e. the language used in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question and answer, knowledge graph, etc.

[0076] With the research and progress of artificial intelligence technology, artificial intelligence technology has been researched and applied in many fields, such as common smart home, smart wearable device, virtual assistant, smart speaker, smart marketing, unmanned vehicle, autonomous driving, unmanned aerial vehicle, robot, smart medical treatment, smart customer service, Internet of Vehicles, autonomous driving, intelligent transportation, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0077] Based on the above implementation environment, the embodiment of the present application provides a language model training method to Figure 2 The flowchart of the language model training method provided by the embodiment of the present application is taken as an example. The method can be executed by the electronic device 11 in Figure 1 As shown in Figure 2 The method includes steps 201 to 206.

[0078] Step 201, obtaining a plurality of first texts.

[0079] The number, content, length, and text type of the first text are not limited in the embodiment of the present application. For example, the first text is the text in multimedia information or the text such as bullet screen and comment for multimedia information.

[0080] Step 202, for any first text, obtaining the appearance probability of each character in the first text according to the first network model.

[0081] For any first text, the first network model is input into the first text, and the first network model determines the semantic features of the first text first, and then determines the appearance probability of each character in the first text based on the semantic features of the first text. The semantic features of the first text include the character semantic features of each character in the first text, and the model structure and model size of the first network model are not limited in the embodiment of the present application.

[0082] Optionally, the first network model includes a generative pre-training (GPT) model, which can be a GPT-1 model, a GPT-2 model, etc. In this case, the probability of any character appearing in the first text can be the probability of that character appearing based on at least one character preceding it in the first text. For example, the probability of any character appearing in the first text satisfies P(w i |w1, w2, ..., w i-1 ), where w i Let be the i-th character in the first text. In other words, the probability of the i-th character appearing in the first text is the probability of the i-th character appearing based on the probability of the i-th character appearing based on the probability of the characters preceding the i-th character in the first text.

[0083] Optionally, the first network model includes a Bidirectional Encoder Representations from Transformers (BERT) model. For example... Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a BERT model provided in an embodiment of this application. For text consisting of characters 1, 2, ..., N, after inputting the text into the BERT model, the text vectors of each character are first determined, that is, the text vectors of character 1, character 2, ..., character N are determined first. Then, the text vectors of each character are input into the converter, which includes multiple conversion parts. Each conversion part can update the text vector of one character based on the text vectors of each character. Finally, the converter outputs the semantic features of each character, that is, the converter outputs the semantic features of character 1, character 2, ..., character N.

[0084] When the first network model includes a BERT model, the probability of any character appearing in the first text can be the probability of that character appearing in addition to at least one other character in the first text. For example, the probability of any character appearing in the first text satisfies P(w i |w1,…,w i-1 w i+1 ,…,w k ), where w i Let be the i-th character in the first text, and k be the number of characters in the first text. k is a positive integer. In other words, the probability of the i-th character appearing in the first text is the probability of the i-th character appearing based on the probability of each character appearing in the first text excluding the i-th character.

[0085] Step 203: Based on the occurrence probability of each character in multiple first texts, adjust the first network model to obtain a first language model, which includes a first feature extraction model.

[0086] In this embodiment, the loss value of the first network model is calculated based on the occurrence probability of each character in multiple first texts. The model parameters of the first network model are then updated based on this loss value to obtain an updated first network model. When the first training termination condition is met, the updated first network model is used as the first language model. When the first training termination condition is not met, the updated first network model is used as the first network model for the next training iteration. Based on multiple first texts, the first network model is updated and trained at least once according to steps 202 to 203 until the first language model is obtained. This embodiment does not limit the scope of the first training termination condition; for example, meeting the first training termination condition means that the number of training iterations reaches a first target number (e.g., 500).

[0087] Optionally, according to the formula The loss value of the first network model is calculated based on the occurrence probability of each character in multiple first texts. This represents the loss value of the first network model. These are the model parameters for the first network model. Let P be the i-th character in the first text, and let P represent the probability of the character appearing. It is a positive integer greater than or equal to 1.

[0088] In this embodiment, the first language model includes a first feature extraction model and a regression model. This embodiment does not limit the regression model; for example, the regression model is a Softmax model.

[0089] In this embodiment of the application, a first language model is obtained based on the occurrence probability of each character in multiple first texts and a first network model, so that the first language model learns the occurrence probability of each character in the text, which facilitates text classification based on the occurrence probability of each character in the text.

[0090] Step 204: Obtain the semantic features of any first text based on the first feature extraction model.

[0091] In this embodiment of the application, any first text is input into a first feature extraction model, and the first feature extraction model outputs the semantic features of any first text. The semantic features of any first text include the textual semantic features of the first text and / or the character semantic features of each character in the first text.

[0092] Optionally, the first feature extraction model includes an encoder and a converter. Any first text is input into the first feature extraction model. The encoder determines the text vector of the first text, and the converter determines the semantic features of the first text based on the text vector. The text vector of any first text includes the character vector of each character in the first text, the position vector of each character in the first text, and the paragraph vector of each character in the first text.

[0093] It should be noted that the text vector of any character includes the character vector of any character, the position vector of any character, and the paragraph vector of any character. In other words, the text vector of any first text includes the text vectors of each character in the first text.

[0094] Step 205: Based on the semantic features of multiple first texts, adjust the first feature extraction model to obtain the second feature extraction model.

[0095] In this embodiment, based on the semantic features of multiple first texts, the first feature extraction model is updated once to obtain the updated first feature extraction model. If the second training termination condition is met, the updated first feature extraction model is used as the second feature extraction model; if the second training termination condition is not met, the updated first feature extraction model is used as the first feature extraction model for the next training iteration. Based on multiple first texts, the first feature extraction model is updated and trained at least once according to steps 204 to 205 until the second feature extraction model is obtained. This embodiment does not limit the satisfaction of the second training termination condition; for example, satisfying the second training termination condition means that the number of training iterations reaches a second target number (e.g., 1500).

[0096] In one possible implementation (denoted as implementation A1), a second feature extraction model is obtained by adjusting the first feature extraction model based on the semantic features of multiple first texts, including: determining the loss value of each first text based on the semantic features of multiple first texts; determining the loss value of the first feature extraction model based on the loss value of each first text; and adjusting the first feature extraction model based on the loss value of the first feature extraction model to obtain the second feature extraction model.

[0097] In the embodiments of the present application, the loss value of any first text is determined based on the semantic features of the plurality of first texts, and in this way, the loss values of the first texts can be determined. Then, the sum of the loss values of the first texts is taken as the loss value of the first feature extraction model. Subsequently, the model parameters of the first feature extraction model are updated once based on the loss value of the first feature extraction model, to obtain an updated first feature extraction model, and the second feature extraction model is obtained based on the updated first feature extraction model.

[0098] Optionally, any first text is any original text or a replacement text corresponding to any original text, and the replacement text corresponding to any original text is a text obtained by replacing characters in any original text; the loss value of each first text is determined based on the semantic features of the plurality of first texts, comprising: for any original text, the loss value of any original text is determined based on the semantic features of each original text and the semantic features of the replacement text corresponding to each original text; for any replacement text corresponding to any original text, the loss value of any replacement text corresponding to any original text is determined based on the semantic features of each original text and the semantic features of the replacement text corresponding to each original text.

[0099] In the embodiments of the present application, the implementation manner A1 can be implemented based on contrastive learning (CTL). At this time, any first text is any original text or a replacement text corresponding to any original text, that is, the plurality of first texts includes at least two original texts and the replacement text corresponding to each original text. Wherein, the replacement text corresponding to any original text is a text obtained by replacing at least one character in any original text, and the at least one character can be continuous or discontinuous characters. For example, the original text is segmented to obtain at least two words, each character in any word is replaced to obtain the replacement text corresponding to the original text, and any word of the original text can be an entity.

[0100] In a possible implementation manner, the replacement of at least one character in the original text is implemented by synonym replacement of the word vector model. That is, the original text is segmented to obtain at least two words, for any randomly selected word, the word with the maximum cosine similarity to the any word is queried in the word table, and the any word is replaced by the queried word to obtain the replacement text corresponding to the original text, that is, the original text and the replacement text corresponding to the original text are similar texts. Wherein, the cosine similarity between the queried word and the any word is not less than a similarity threshold (for example, 0.8), and the word vector model includes but is not limited to word2vector model, Glove model and FastText model.

[0101] In the embodiments of the present application, the loss value of any first text is determined based on the semantic features of the plurality of first texts, i.e., the loss value of any original text or the loss value of the replacement text corresponding to any original text is determined based on the semantic features of the plurality of first texts. Then, the loss value of the first feature extraction model is determined based on the loss value of each original text and the loss value of the replacement text corresponding to each original text. The determination manner of the loss value of the first feature extraction model is shown in the following formula (1).

[0102] Formula (1)

[0103] wherein, is the loss value of the first feature extraction model, is the original text, is the replacement text corresponding to the original text, is the number of original texts and / or the number of replacement texts, is the loss value of the original text, is the loss value of the replacement text corresponding to the original text.

[0104] The determination of the loss value of any original text based on the semantic features of the plurality of first texts is introduced as follows. Optionally, the loss value of any original text is determined based on the semantic features of each original text and the semantic features of the replacement text corresponding to each original text, including: determining the first similarity between any original text and the replacement text corresponding to any original text based on the semantic features of any original text and the semantic features of the replacement text corresponding to any original text; determining the second similarity between any original text and other original texts based on the semantic features of any original text and the semantic features of other original texts, wherein the other original texts are the original texts other than any original text among the original texts; determining the third similarity between any original text and the replacement texts corresponding to other original texts based on the semantic features of any original text and the semantic features of the replacement texts corresponding to other original texts; and determining the loss value of any original text based on the first similarity, the second similarity and the third similarity.

[0105] In the embodiments of the present application, when calculating the loss value of any one original text, first, the first similarity is determined based on the semantic feature of any one original text and the semantic feature of the replacement text corresponding to any one original text, then the second similarity is determined based on the semantic feature of any one original text and the semantic features of other original texts, and the third similarity is determined based on the semantic feature of any one original text and the semantic features of the replacement texts corresponding to other original texts. Subsequently, the loss value of any one original text is determined based on the first similarity, the second similarity and the third similarity. The above process of calculating the loss value of any one original text is shown in the following formula (2).

[0106] Formula (2)

[0107] wherein, is the loss value of any one original text, is the symbol of the similarity, is the semantic feature of any one original text, is the semantic feature of the replacement text corresponding to any one original text, is the semantic feature of other original texts, is the semantic feature of the replacement text corresponding to other original texts, and are sequence numbers, is a scaling factor.

[0108] The embodiments of the present application do not limit the calculation method of the scaling factor. For example, the scaling factor is determined based on the training times. In one possible implementation, the calculation formula of the scaling factor is shown in the following formula (3).

[0109] Formula (3)

[0110] wherein, is the scaling factor corresponding to the current training times, is the current training times, is the second target training times.

[0111] The following introduces determining the loss value of the replacement text corresponding to any original text based on the semantic features of multiple first texts. Optionally, the loss value of the replacement text corresponding to any original text is determined based on the semantic features of each original text and the semantic features of the replacement text corresponding to each original text, including: determining the first similarity between any original text and the replacement text corresponding to any original text based on the semantic features of any original text and the semantic features of the replacement text corresponding to any original text; determining the fourth similarity between the replacement text corresponding to any original text and other original texts based on the semantic features of the replacement text corresponding to any original text and the semantic features of other original texts, the other original texts being original texts other than any original text among the original texts; determining the fifth similarity between the replacement text corresponding to any original text and the replacement texts corresponding to other original texts based on the semantic features of the replacement text corresponding to any original text and the semantic features of the replacement texts corresponding to other original texts; and determining the loss value of the replacement text corresponding to any original text based on the first similarity, the fourth similarity, and the fifth similarity.

[0112] In the embodiments of the present application, when calculating the loss value of the replacement text corresponding to any original text, the first similarity is determined based on the semantic features of any original text and the semantic features of the replacement text corresponding to any original text, the fourth similarity is determined based on the semantic features of the replacement text corresponding to any original text and the semantic features of other original texts, and the fifth similarity is determined based on the semantic features of the replacement text corresponding to any original text and the semantic features of the replacement texts corresponding to other original texts. Then, the loss value of the replacement text corresponding to any original text is determined based on the first similarity, the fourth similarity, and the fifth similarity. The above process of calculating the loss value of the replacement text corresponding to any original text is shown in the following formula (4).

[0113] Formula (4)

[0114] wherein, is the loss value of the replacement text corresponding to any original text, is the symbol of the similarity, is the semantic feature of any original text, is the semantic feature of the replacement text corresponding to any original text, is the semantic feature of other original texts, is the semantic feature of the replacement text corresponding to other original texts, and are sequence numbers, is a scaling factor, and the description of the scaling factor can be found above and will not be repeated here.

[0115] Next, please refer to Figure 4 , Figure 4 is a processing schematic diagram of original text and replacement text provided by an embodiment of the present application. In the embodiment of the present application, the replacement text corresponding to the original text 1 is the replacement text 1, the replacement text corresponding to the original text 2 is the replacement text 2, the replacement text corresponding to the original text 3 is the replacement text 3, and the original text 1, the original text 2 and the original text 3 are different texts.

[0116] Firstly, the original text 1 to 3 and the replacement text 1 to 3 are input into the first feature extraction model, and the semantic features of the original text 1 to 3 and the replacement text 1 to 3 are output by the first feature extraction model. Then, the semantic features of one original text and the semantic features of the replacement text corresponding to the original text are taken as positive samples, for example, the semantic features of the original text 1 and the semantic features of the replacement text 1 are taken as positive samples, and the semantic features of one original text and the semantic features of another original text, the semantic features of one original text and the semantic features of the replacement text corresponding to another original text, the semantic features of the replacement text corresponding to one original text and the semantic features of the replacement text corresponding to another original text are taken as negative samples, for example, the semantic features of the original text 1 and the semantic features of the original text 2, the semantic features of the original text 1 and the semantic features of the original text 3, the semantic features of the original text 1 and the semantic features of the replacement text 2, the semantic features of the original text 1 and the semantic features of the replacement text 3 are all taken as negative samples. After that, the similarities between the positive samples and the similarities between the negative samples are calculated.

[0117] After the similarities between the positive samples and the similarities between the negative samples are calculated, the loss values of the first feature extraction model are calculated by using formulas (1) to (4), and based on the loss values of the first feature extraction model, the second feature extraction model is obtained according to the first feature extraction model.

[0118] In the embodiment of the present application, by determining the semantic features of each original text and the semantic features of the replacement text corresponding to each original text, the first feature extraction model is adjusted based on the semantic features of each original text and the semantic features of the replacement text corresponding to each original text to obtain the second feature extraction model, which can accurately determine the semantic features of different texts. When the replacement text corresponding to the original text and the original text are similar texts, the second feature extraction model can accurately identify and determine the semantic features of the similar texts, improve the robustness of the second feature extraction model to noise, and improve the accuracy of the second feature extraction model.

[0119] In another possible implementation (referred to as implementation A2), the first feature extraction model is adjusted based on semantic features of the plurality of first texts to obtain a second feature extraction model, including: determining predicted information of each first text based on the semantic features of the first text, the predicted information of the first text being a probability of each character in the first text being replaced obtained by prediction; obtaining labeled information of each first text, the labeled information of the first text being information of whether each character in the first text is replaced obtained by labeling; and adjusting the first feature extraction model based on the predicted information of each first text and the labeled information of each first text to obtain the second feature extraction model.

[0120] In the embodiments of the present application, implementation A2 is realized based on Replaced Token Detection (RTD). That is, after the semantic features of each first text are obtained based on the first feature extraction model, the semantic features of each first text are input into an activation model, and the activation model outputs the predicted information of each first text. The model structure and model size of the activation model are not limited in the embodiments of the present application. For example, the activation model is a sigmoid model.

[0121] For any first text, the predicted information of the first text is a probability of each character in the first text being replaced obtained by prediction, and the probability of any character being replaced is greater than or equal to 0 and less than or equal to 1. The labeled information of each first text can also be obtained, and the labeled information of any first text is information of whether each character in the first text is replaced obtained by labeling, and the information of whether any character is replaced is 0 or 1, where 0 indicates that the character is not replaced, and 1 indicates that the character is replaced.

[0122] Next, the loss value of the first feature extraction model is calculated based on the predicted information of each first text and the labeled information of each first text. The calculation formula of the loss value of the first feature extraction model is shown in the following formula (5).

[0123] Formula (5)

[0124] wherein, L represents the loss value of the first feature extraction model, is any character in the first text, E represents expectation, and is the number of characters in the first text, is the position of the character, represents the character after the character is replaced. and are characters are replaced, when , i.e., the character is not replaced, , the value of is 0; when , i.e., the character is replaced, , the value of is 1, represents the probability that the t-th character in the first text activated by the model output is replaced.

[0125] After calculating the loss value of the first feature extraction model, the first feature extraction model is updated once based on the loss value of the first feature extraction model to obtain an updated first feature extraction model, and a second feature extraction model is obtained according to the updated first feature extraction model.

[0126] It can be understood that the first text in the embodiments of the present application is a text obtained by replacing at least one character in the original text, which can be consecutive or non-consecutive characters. In a possible implementation, the at least one character in the original text is replaced by synonym replacement of a word vector model. That is, the original text is first segmented to obtain at least two words, and for a randomly selected any one word, a word with a cosine similarity not greater than a similarity threshold is queried in a word table, and the any one word is replaced by the queried word to obtain the first text. In the embodiments of the present application, the similarity threshold is not limited, and the similarity threshold is exemplarily 0.5, and the word vector model includes but is not limited to a word2vector model, a Glove model, and a FastText model.

[0127] In the embodiments of the present application, since at least one character in the first text is replaced, after obtaining the second feature extraction model based on the prediction information of each first text and the annotation information of each first text, the second feature extraction model can pay more attention to the fluency of the sentence, so that the second feature extraction model can accurately determine the semantic features of the replaced text, and the accuracy of the second feature extraction model is increased.

[0128] It should be noted that the embodiments of the present application can first perform the implementation manner A1 and then perform the implementation manner A2, that is, first determine the loss value of each first text based on the semantic features of the plurality of first texts, then determine the loss value of the first feature extraction model based on the loss value of each first text, and adjust the first feature extraction model based on the loss value of the first feature extraction model to obtain an intermediate feature extraction model, then determine the predicted information of each first text based on the semantic features of each first text, obtain the labeled information of each first text, and then adjust the intermediate feature extraction model based on the predicted information of each first text and the labeled information of each first text to obtain the second feature extraction model. The implementation manner A2 can be performed first and then the implementation manner A1, that is, first determine the predicted information of each first text based on the semantic features of each first text, obtain the labeled information of each first text, then adjust the first feature extraction model based on the predicted information of each first text and the labeled information of each first text to obtain an intermediate feature extraction model, then determine the loss value of each first text based on the semantic features of the plurality of first texts, then determine the loss value of the intermediate feature extraction model based on the loss value of each first text, and adjust the intermediate feature extraction model based on the loss value of the intermediate feature extraction model to obtain the second feature extraction model.

[0129] In step 206, the second feature extraction model is used to determine the target language model.

[0130] After the second feature extraction model is determined, the second feature extraction model is spliced with the regression model to obtain the target language model, so as to realize the classification processing of the target text by using the target language model. The target language model is obtained by zero-shot learning (ZSL) training in the embodiments of the present application, which reduces the labeling process, accelerates the training speed of the model, and improves the text classification efficiency.

[0131] Optionally, the language model obtained by splicing the second feature extraction model and the regression model can also be trained by using a small amount of labeled samples to obtain the target language model. The target language model is obtained by few-shot learning (FSL) training in the embodiments of the present application, which reduces the number of samples that need to be labeled, accelerates the training speed of the model, and improves the text classification efficiency.

[0132] Optionally, based on the second feature extraction model, the target language model is determined, including: obtaining the second text and a category label of the second text; obtaining, according to the second language model, an occurrence probability of each character in at least two spliced texts, the spliced texts including the second text and a candidate text category, and the second language model including the second feature extraction model; determining, based on the occurrence probability of each character in the at least two spliced texts, a target spliced text from the at least two spliced texts; and adjusting the second language model based on the candidate text category in the target spliced text and the category label of the second text to obtain the target language model.

[0133] In the embodiment of the application, the second feature extraction model and the regression model are spliced to obtain the second language model. For any second text, the second text is input into the second language model, the semantic features of the spliced text are determined by the second feature extraction model, and the occurrence probability of each character in the spliced text is determined by the regression model based on the semantic features of the spliced text. The semantic features of the spliced text include the character semantic features of each character in the spliced text, and the spliced text includes the second text and a candidate text category.

[0134] In the embodiment of the application, the candidate text category is not limited, and the candidate text category may be, for example, a sentiment category such as good, general, bad, positive, and negative, a content category such as entertainment, animation, and food, or a quality category such as junk text and non-junk text.

[0135] In a possible implementation, the occurrence probability of each character in the at least two spliced texts is obtained according to the second language model, including: splicing the second text and the candidate text category to obtain any one of the at least two spliced texts; inputting the any one of the spliced texts into the second language model to determine a text vector of the any one of the spliced texts, and determining the occurrence probability of each character in the any one of the spliced texts based on the text vector of the any one of the spliced texts.

[0136] In the embodiment of the application, the second text and the candidate text category are spliced by using a hard coding method to obtain the spliced text, the semantic features of the spliced text are determined by the second feature extraction model, and the occurrence probability of each character in the spliced text is determined by the regression model based on the semantic features of the spliced text, so as to realize outputting the occurrence probability of each character in the spliced text by the second language model.

[0137] The second feature extraction model includes an encoder and a converter. The spliced text is input into the second language model, the text vector of the spliced text is determined by the encoder, the semantic feature of the spliced text is determined by the converter based on the text vector of the spliced text, and the occurrence probability of each character in the spliced text is determined by the regression model based on the semantic feature of the spliced text. The text vector of the spliced text includes the character vector of each character in the spliced text, the position vector of each character in the spliced text, and the paragraph vector of each character in the spliced text.

[0138] Optionally, in addition to the second text and the candidate text category, the spliced text can also include other texts, as shown in Table 1 below.

[0139] Table 1

[0140]

[0141] In Table 1, the second text is “The movie is amazing and fascinating.”, “The content of the movie is just so-so, and the completion degree is acceptable.”, or “Junk movie, waste of time.”, other texts “feel really” are spliced after the second text, and the candidate text category includes “good”, “general” and “bad”.

[0142] Hard coding is to directly splice the second text and the candidate text category to obtain the spliced text. For different scenarios, different hard-coded texts need to be constructed, for example, for the text “Men's autumn and winter sweater large size with fleece”, it is not appropriate to splice the hard-coded texts “feel really good”, “feel really bad” and “feel really general” after the text, therefore, the corresponding hard-coded text needs to be constructed for the text. This also leads to weak generality of hard coding and difficulty in designing hard-coded texts. Based on this, the embodiment of the present application also designs a soft coding mode, splices the text vector of the second text and the text vector of the candidate text category to obtain the text vector of the spliced text, and through multiple training and learning, the model learns the soft-coded text and improves the generality of the model. The soft coding mode in the embodiment of the present application is introduced below.

[0143] In another possible implementation, the occurrence probability of each character in at least two spliced texts is obtained according to the second language model, including: inputting the second text into the second language model to determine the text vector of the second text by the second language model; splicing the text vector of the second text and the text vector of the candidate text category by the second language model to obtain the text vector of any one of the at least two spliced texts; and determining the occurrence probability of each character in the any one of the at least two spliced texts based on the text vector of the any one of the at least two spliced texts by the second language model.

[0144] In the embodiment of the present application, the second language model comprises a second feature extraction model and a regression model, and the second feature extraction model comprises an encoder and a converter. The second text is input into the second language model in a soft coding manner, the text vector of the second text and the text vector of the candidate text category are determined by the encoder, the text vector of the second text and the text vector of the candidate text category are spliced to obtain the text vector of the spliced text, the semantic feature of the spliced text is determined by the converter based on the text vector of the spliced text, and the occurrence probability of each character in the spliced text is determined by the regression model based on the semantic feature of the spliced text.

[0145] The text vector of the text (including but not limited to the first text, the second text, the spliced text, and the candidate text category in various optional embodiments) comprises the character vector of each character in the text, the position vector of each character in the text, and the paragraph vector of each character in the text.

[0146] Optionally, the text vector of the spliced text can also include the text vector of other text in addition to the text vector of the second text and the text vector of the candidate text category. The content of the other text is not limited in the embodiment of the present application. For example, the other text is "feeling true".

[0147] Please refer to Figure 5 , Figure 5 is a schematic diagram of soft coding provided by the embodiment of the present application. In the embodiment of the present application, the second text comprises character 1, character 2 and character 3, the second language model comprises an encoder and a converter, the encoder comprises a character encoder, a position encoder and a paragraph encoder, and the converter comprises converter 1, converter 2 to converter L, and L is a positive integer.

[0148] The second text is input into the second language model, the character vector of each character in the second text is determined by the character encoder, the position vector of each character in the second text is determined by the position encoder, and the paragraph vector of each character in the second text is determined by the paragraph encoder, thereby obtaining the text vector of the second text. Based on the same principle, the text vector of other text and the text vector of the candidate text category are determined by the character encoder, the position encoder and the paragraph encoder.

[0149] Then, the text vector of the second text, the text vector of other text and the text vector of the candidate text category are spliced to obtain the text vector of the spliced text. The text vector of the spliced text is input into the converter, and the semantic feature of the spliced text is finally output after the converter 1, the converter 2 to the converter L, wherein the semantic feature of the spliced text comprises the character semantic feature of each character in the spliced text, and the semantic feature of each character in the spliced text comprises the character semantic features 1 to 5.

[0150] The second language model in the embodiments of the present application further comprises a regression model (not shown) Figure 5 The regression model is used to determine the appearance probability of each character in the spliced text based on the semantic features of the spliced text.

[0151] In the embodiments of the present application, after obtaining the appearance probability of each character in the at least two spliced texts, the target spliced text is determined from the at least two spliced texts based on the appearance probability of each character in the at least two spliced texts.

[0152] Optionally, determining the target spliced text from the at least two spliced texts based on the appearance probability of each character in the at least two spliced texts comprises: determining the perplexity of each spliced text based on the appearance probability of each character in the at least two spliced texts, wherein the perplexity of the spliced text represents the fluency degree of the spliced text; and determining the spliced text corresponding to the perplexity satisfying the condition as the target spliced text based on the perplexity of each spliced text.

[0153] In the embodiments of the present application, for any spliced text, the perplexity of the spliced text is determined based on the appearance probability of each character in the spliced text. The perplexity (PPL) is a text measurement method used to represent the fluency of the text. The greater the perplexity of the spliced text, the less fluent the spliced text is, and the smaller the perplexity of the spliced text, the more fluent the spliced text is.

[0154] Optionally, determining the perplexity of each spliced text based on the appearance probability of each character in the at least two spliced texts comprises: determining the appearance probability of each spliced text based on the appearance probability of each character in the at least two spliced texts; and determining the perplexity of each spliced text based on the appearance probability of each spliced text.

[0155] In the embodiments of the present application, for any spliced text, the appearance probability of the spliced text is determined based on the appearance probability of each character in the spliced text. The appearance probability of the spliced text is shown in the following formula (6-1) or (6-2).

[0156] Formula (6-1)

[0157] Formula (6-2)

[0158] wherein, and are the appearance probability of the spliced text, , , and are the appearance probability of the character in the spliced text, and The number of characters in the spliced text.

[0159] After determining the occurrence probabilities of the at least two spliced texts, the perplexity of each spliced text is determined based on the occurrence probability of each spliced text. The perplexity of the spliced text is determined according to formula (7) as follows.

[0160] Formula (7)

[0161] Wherein, is the perplexity of the spliced text, is the occurrence probability of the spliced text, is the number of characters in the spliced text.

[0162] After calculating the perplexities of the at least two spliced texts, the spliced text corresponding to the perplexity meeting the condition is determined as the target spliced text. The perplexity meeting the condition is not limited in the embodiments of the present application. For example, the perplexity meeting the condition is the minimum perplexity, that is, the spliced text with the minimum perplexity is taken as the target spliced text. Alternatively, the perplexity meeting the condition is the perplexity not greater than the perplexity threshold, that is, the spliced text with the perplexity not greater than the perplexity threshold is taken as the target spliced text. Then, the candidate text category in the target spliced text is determined as the text category of the second text, as shown in Table 2 below.

[0163] Table 2

[0164]

[0165] In Table 2, the second text is “movie is wonderful, fascinating.”, one spliced text is “movie is wonderful, fascinating. It feels really good”, the corresponding perplexity is “1.0116761350214916”, another spliced text is “movie is wonderful, fascinating. It feels really average”, the corresponding perplexity is “1.1787686347935873”, and another spliced text is “movie is wonderful, fascinating. It feels really bad”, the corresponding perplexity is “1.2385989684455543”. Based on the three perplexities, the spliced text with the minimum perplexity is determined as the target spliced text, that is, the target spliced text is “movie is wonderful, fascinating. It feels really good”, the target spliced text includes the second text “movie is wonderful, fascinating.”, other text “It feels really”, and candidate text category “good”, at this time, “good” is taken as the text category of the second text “movie is wonderful, fascinating.”.

[0166] Afterwards, based on the text category of the second text and the category label of the second text, a loss value of the second language model is determined, and the second language model is updated once based on the loss value of the second language model to obtain an updated second language model. If a third training end condition is met, the updated second language model is taken as the target language model, and if the third training end condition is not met, the updated second language model is taken as the second language model for the next training, and the second language model is updated at least once based on the second text and the category label of the second text until the target language model is obtained. In the embodiments of the present application, the third training end condition is not limited, and for example, the third training end condition is that the number of training times reaches a third target training number (for example, 50 times). The process of updating the second language model at least once is described above and will not be repeated here.

[0167] In the embodiments of the present application, the second language model is trained based on the second text and the category label of the second text to obtain the target language model. In the soft coding mode, the target language model can learn the soft coded text, which increases the versatility of the target language model. At the same time, the model can accurately output the appearance probability of each character in the spliced text, so as to accurately determine the text category based on the appearance probability of each character in the spliced text.

[0168] It can be understood that the embodiments of the present application can take the spliced text corresponding to the perplexity threshold value not greater than the perplexity threshold value as the target spliced text, adjust the second language model based on the candidate text category in the target spliced text and the category label of the second text, and obtain the target language model. In the process of training the target language model, the embodiments of the present application can constantly adjust the size of the perplexity threshold value to improve the accuracy of the perplexity threshold value.

[0169] It should be noted that the embodiments of the present application select the target spliced text from at least two spliced texts based on the perplexity of each spliced text. In application, other evaluation indexes can be used to evaluate the fluency of the spliced text to select the target spliced text from the spliced texts.

[0170] In a possible implementation, before obtaining the appearance probability of each character in any first text according to the first network model, the method further includes: obtaining semantic features of any first text according to a second network model; adjusting the second network model based on the semantic features of the plurality of first texts to obtain a third feature extraction model; and constructing the first network model based on the third feature extraction model.

[0171] In the embodiments of the present application, any one of the first texts is input into the second network model, and semantic features of any one of the first texts are output by the second network model. Optionally, the second network model comprises an encoder and a converter. Any one of the first texts is input into the second network model, the text vector of any one of the first texts is determined by the encoder, and the semantic features of any one of the first texts are determined by the converter based on the text vector of any one of the first texts. In this way, the semantic features of the plurality of first texts can be determined.

[0172] Afterwards, the second network model is updated based on the semantic features of the plurality of first texts to obtain an updated second network model. If the fourth training end condition is met, the updated second network model is taken as the third feature extraction model, and if the fourth training end condition is not met, the updated second network model is taken as the second network model for the next training, and the second network model is updated at least once based on the plurality of first texts in the manner of the embodiments of the present application until the third feature extraction model is obtained. In the embodiments of the present application, the fourth training end condition is not limited, and exemplarily, the fourth training end condition is that the number of training reaches the fourth target training number.

[0173] It should be noted that the implementation manner of obtaining the semantic features of any one of the first texts based on the second network model, adjusting the second network model based on the semantic features of the plurality of first texts, and obtaining the third feature extraction model is similar to the implementation manner of steps 204-205, and the description of steps 204-205 can be referred to.

[0174] After the third feature extraction model is obtained, the third feature extraction model and the third network model are spliced to obtain the first network model. The first language model is obtained based on the plurality of first texts in the manner of steps 201-203. The first language model comprises the first feature extraction model and the regression model, the first feature extraction model is obtained by adjusting the third feature extraction model, and the regression model is obtained by adjusting the third network model.

[0175] The above method first trains the first language model based on the plurality of first texts, the first language model comprises the first feature extraction model, then adjusts the first feature extraction model based on the semantic features of the plurality of first texts obtained based on the first feature extraction model to obtain the second feature extraction model, and obtains the target language model based on the second feature extraction model. The two-stage training of the language model based on the first texts is realized, so that the language model with high accuracy can be trained based on a small amount of first texts, the dependence of the model on the training data is reduced, and thus the acquisition time of the training data is reduced, and the training speed of the model and the efficiency of the text classification are improved.

[0176] And the application can improve the accuracy of the feature extraction model by training the feature extraction model, thereby improving the accuracy of the target language model containing the second feature extraction model and ensuring the accuracy of text classification.

[0177] Based on the above implementation environment, the embodiment of the application provides a text classification method, which comprises the following steps: Figure 6 As shown in the flowchart of the text classification method provided by the embodiment of the application, the method can be executed by the electronic device 11 in the Figure 1 As shown in the flowchart of the text classification method provided by the embodiment of the application, the method can be executed by the electronic device 11 in the Figure 6 The method comprises steps 601 to 604.

[0178] Step 601: obtaining a target text.

[0179] The embodiment of the application does not limit the content, length, text type, etc. of the target text. For example, the target text is text in multimedia information and text such as bullet screen and comments for multimedia information.

[0180] Step 602: obtaining the appearance probability of each character in at least two reconstructed texts according to a target language model, wherein the reconstructed texts comprise the target text and a candidate text category.

[0181] The target language model is obtained according to the training method of the language model provided in the above optional embodiments.

[0182] In the embodiment of the application, the second feature extraction model is spliced with the regression model to obtain a second language model, and the target language model is the second language model or is obtained by adjusting the second language model. Therefore, the target language model comprises a feature extraction network and a regression network, the feature extraction network is the second feature extraction model or is obtained by adjusting the second feature extraction model, and the regression network is the regression model or is obtained by adjusting the regression model.

[0183] The target text is input into the target language model, the semantic features of the reconstructed text are determined by the feature extraction network, and the appearance probability of each character in the reconstructed text is determined by the regression network based on the semantic features of the reconstructed text. The semantic features of the reconstructed text comprise the character semantic features of each character in the reconstructed text, and the reconstructed text comprises the target text and a candidate text category.

[0184] In a possible implementation, the obtaining, according to the target language model, of the appearance probability of each character in the at least two reconstructed texts comprises: splicing the target text and the candidate text category to obtain any one of the at least two reconstructed texts; inputting the any one of the reconstructed texts into the target language model to determine a text vector of the any one of the reconstructed texts, and determining the appearance probability of each character in the any one of the reconstructed texts based on the text vector of the any one of the reconstructed texts.

[0185] In the embodiments of the present application, the target text and the candidate text category are spliced in a hard coding manner to obtain the reconstructed text, the reconstructed text is input into the target language model, the semantic feature of the reconstructed text is determined by the feature extraction network, and the appearance probability of each character in the reconstructed text is determined by the regression network based on the semantic feature of the reconstructed text, so as to realize outputting the appearance probability of each character in the reconstructed text by the target language model.

[0186] The feature extraction network comprises an encoder and a converter. The reconstructed text is input into the target language model, the text vector of the reconstructed text is determined by the encoder, the semantic feature of the reconstructed text is determined by the converter based on the text vector of the reconstructed text, and the appearance probability of each character in the reconstructed text is determined by the regression model based on the semantic feature of the reconstructed text.

[0187] In another possible implementation, the obtaining, according to the target language model, of the appearance probability of each character in the at least two reconstructed texts comprises: inputting the target text into the target language model to determine a text vector of the target text; splicing, by the target language model, the text vector of the target text and a text vector of the candidate text category to obtain a text vector of any one of the at least two reconstructed texts; and determining, by the target language model, the appearance probability of each character in the any one of the reconstructed texts based on the text vector of the any one of the reconstructed texts.

[0188] In the embodiments of the present application, the target language model comprises a feature extraction network and a regression network, and the feature extraction network comprises an encoder and a converter. The target text is input into the target language model in a soft coding manner, the text vector of the target text and the text vector of the candidate text category are determined by the encoder, the text vector of the target text and the text vector of the candidate text category are spliced to obtain a text vector of the reconstructed text, the semantic feature of the reconstructed text is determined by the converter based on the text vector of the reconstructed text, and the appearance probability of each character in the reconstructed text is determined by the regression network based on the semantic feature of the reconstructed text.

[0189] It should be noted that the description of step 602 can be seen in the description of the “obtaining, according to the second language model, of the appearance probability of each character in the at least two spliced texts” above, and the implementation principles are the same, which will not be described here.

[0190] In step 603, the target reconstructed text is determined from the at least two reconstructed texts based on the appearance probability of each character in the at least two reconstructed texts.

[0191] After obtaining the appearance probability of each character in the at least two reconstructed texts, the target reconstructed text is determined from the at least two reconstructed texts based on the appearance probability of each character in the at least two reconstructed texts.

[0192] In one possible implementation, the target reconstructed text is determined from the at least two reconstructed texts based on the appearance probability of each character in the at least two reconstructed texts, including: determining the perplexity of each reconstructed text based on the appearance probability of each character in the at least two reconstructed texts, where the perplexity of the reconstructed text represents the fluency of the reconstructed text; and determining the reconstructed text corresponding to the perplexity satisfying a condition as the target reconstructed text based on the perplexity of each reconstructed text.

[0193] In the embodiments of the present application, for any reconstructed text, the perplexity of the reconstructed text is determined based on the appearance probability of each character in the reconstructed text.

[0194] Optionally, the perplexity of each reconstructed text is determined based on the appearance probability of each character in the at least two reconstructed texts, including: determining the appearance probability of each reconstructed text based on the appearance probability of each character in the at least two reconstructed texts; and determining the perplexity of each reconstructed text based on the appearance probability of each reconstructed text.

[0195] In the embodiments of the present application, for any reconstructed text, the appearance probability of the reconstructed text is determined according to formula (6-1) or (6-2) first, and then the perplexity of the reconstructed text is determined based on the appearance probability of the reconstructed text according to formula (7).

[0196] After calculating the perplexity of the at least two reconstructed texts, the reconstructed text corresponding to the perplexity satisfying a condition is determined as the target reconstructed text. The embodiments of the present application do not limit the perplexity satisfying the condition, and exemplarily, the perplexity satisfying the condition is the minimum perplexity, or the perplexity satisfying the condition is the perplexity not greater than a perplexity threshold.

[0197] It should be noted that the description of step 603 can be found in the description of “determining the target spliced text from the at least two spliced texts based on the appearance probability of each character in the at least two spliced texts” above, and the implementation principles are the same, which will not be described here.

[0198] In step 604, the candidate text category in the target reconstructed text is determined as the text category of the target text.

[0199] In the embodiment of the present application, the target reconstructed text includes a target text and a candidate text category, and the candidate text category in the target reconstructed text is taken as the text category of the target text. For example, the target reconstructed text is "The movie is wonderful and fascinating. It feels really good", which includes the target text "The movie is wonderful and fascinating", the other text "It feels really good", and the candidate text category "good". At this time, "good" is taken as the text category of the target text "The movie is wonderful and fascinating".

[0200] It should be noted that the description of step 604 can be seen in the description of "determining the candidate text category in the target spliced text as the text category of the second text" above, and the implementation principles are the same, which will not be described here.

[0201] The determination manner of the target language model in the above method is: first, a first language model is trained by using a plurality of first texts, the first language model includes a first feature extraction model, then the first feature extraction model is adjusted based on the semantic features of the plurality of first texts obtained based on the first feature extraction model, and then the target language model is determined based on a second feature extraction model. Two-stage training of the language model is realized by using the first text, so that a language model with high accuracy can be trained by using a small amount of first text, the dependence of the model on training data is reduced, thereby reducing the acquisition time of the training data and improving the training speed of the model and the efficiency of text classification. By training the first feature extraction model, the accuracy of the feature extraction model can be improved, thereby improving the accuracy of the target language model and ensuring the accuracy of text classification.

[0202] The training method of the language model and the text classification method of the embodiment of the present application are described from the perspective of the method steps above, and the training method of the language model and the text classification method of the embodiment of the present application will be introduced comprehensively from the perspective of the scene. The scene of the embodiment of the present application is a scene of an information recommendation type application program. For the information in the information recommendation type application program, a plurality of first texts are constructed based on the text in the information and the comment text, barrage text, etc. for the information.

[0203] The first network model is trained based on the plurality of first texts to obtain a first language model, and the first feature extraction model in the first language model is trained based on the plurality of first texts to obtain a second feature extraction model, and the target language model is obtained based on the second feature extraction model. The process of determining the target language model based on the plurality of first texts is described in steps 201 to 206, which will not be described here.

[0204] Optionally, the target language model comprises an encoder, a converter and a Softmax model. When applied, the target language model can be used to determine the text category of the target text. Taking the information in the information recommendation application as an example, please refer to Figure 7 , Figure 7 is a processing schematic diagram of the information by the target language model provided in an embodiment of the present application.

[0205] For the information in the information recommendation application, the information comprises character 1, character 2 (and so on), when the information is input into the target language model, a special character “CLS” is added before the information, that is, the CLS, character 1, character 2 are input into the target language model, the text vector of the CLS, the text vector of the character 1 and the text vector of the character 2 are determined by the encoder, and at the same time, the candidate text category comprises character 3, and the text vector of the character 3 is determined by the encoder. Wherein, the text vector of the reconstructed text comprises the text vector of each of the character 1, character 2 and character 3.

[0206] The semantic features of the CLS, the semantic features of the character 1, the semantic features of the character 2 and the semantic features of the character 3 are determined by the converter based on the text vectors of the CLS, character 1, character 2 and character 3 respectively, and the Softmax model outputs the appearance probability of the character 1, the appearance probability of the character 2 and the appearance probability of the character 3 based on the semantic features of the CLS, character 1, character 2 and character 3 respectively. Wherein, the semantic features of the CLS represent the text semantic features of the first text, and the semantic features of the characters 1-3 represent the character semantic features of each character in the reconstructed text.

[0207] Then, based on the character semantic features of each character in the reconstructed text, the appearance probability of the reconstructed text is determined according to formula (6-1) or (6-2) first, and then the perplexity of the reconstructed text is determined based on the appearance probability of the reconstructed text according to formula (7). In this way, the perplexities of at least two reconstructed texts can be determined, and then the reconstructed text corresponding to the perplexity meeting the condition is determined as the target reconstructed text, and the candidate text category in the target reconstructed text is determined as the text category of the target text. Wherein, the process of determining the text category of the target text based on the target language model is described in steps 601-604, which will not be repeated here.

[0208] The language model training method provided in the embodiments of the present application uses the first text to train the target language model. Since the first text does not need to be labeled, the training time of the model can be shortened, the iteration efficiency of the model can be accelerated, and the accuracy of the model is relatively high. The embodiments of the present application can not use the second text and the category label of the second text to fine-tune the model, that is, the number of the second text can be 0, realizing zero-shot learning, or the embodiments of the present application can use a small amount of the second text and the category label of the second text to fine-tune the model, realizing few-shot learning. Whether it is zero-shot learning or few-shot learning, the accuracy of the target language model provided in the embodiments of the present application is relatively high, which can ensure the text classification accuracy.

[0209] Next, please refer to Figure 8 , Figure 8 The structure diagram of the language model training device provided in the embodiments of the present application is shown in FIG. 8, which includes: Figure 8

[0210] The obtaining module 801 is configured to obtain a plurality of first texts.

[0211] The obtaining module 801 is further configured to, for any one of the first texts, obtain the occurrence probability of each character in the any one of the first texts according to the first network model.

[0212] The adjusting module 802 is configured to adjust the first network model based on the occurrence probability of each character in the plurality of first texts to obtain a first language model, and the first language model includes a first feature extraction model.

[0213] The obtaining module 801 is further configured to obtain the semantic feature of any one of the first texts according to the first feature extraction model.

[0214] The adjusting module 802 is further configured to adjust the first feature extraction model based on the semantic feature of the plurality of first texts to obtain a second feature extraction model.

[0215] The determining module 803 is configured to determine a target language model based on the second feature extraction model.

[0216] In a possible implementation, the adjusting module 802 is configured to determine the loss value of each of the first texts based on the semantic feature of the plurality of first texts, determine the loss value of the first feature extraction model based on the loss value of each of the first texts, and adjust the first feature extraction model based on the loss value of the first feature extraction model to obtain the second feature extraction model.

[0217] ​In a possible implementation, any first text is any original text or a replacement text corresponding to any original text, and the replacement text corresponding to any original text is a text obtained by replacing characters in the original text.

[0218] The adjusting module 802 is configured to determine a loss value of any original text based on the semantic feature of the original text and the semantic feature of a replacement text corresponding to the original text, and determine a loss value of the replacement text corresponding to any original text based on the semantic feature of the original text and the semantic feature of the replacement text corresponding to the original text.

[0219] In a possible implementation, the adjusting module 802 is configured to determine a first similarity between any original text and a replacement text corresponding to the original text based on the semantic feature of the original text and the semantic feature of the replacement text corresponding to the original text, determine a second similarity between any original text and other original texts based on the semantic feature of the original text and the semantic feature of the other original texts, the other original texts being original texts other than the original text, determine a third similarity between any original text and replacement texts corresponding to the other original texts based on the semantic feature of the original text and the semantic feature of the replacement texts corresponding to the other original texts, and determine the loss value of any original text based on the first similarity, the second similarity, and the third similarity.

[0220] In a possible implementation, the adjusting module 802 is configured to determine a first similarity between any original text and a replacement text corresponding to the original text based on the semantic feature of the original text and the semantic feature of the replacement text corresponding to the original text, determine a fourth similarity between the replacement text corresponding to any original text and other original texts based on the semantic feature of the replacement text corresponding to the original text and the semantic feature of the other original texts, the other original texts being original texts other than the original text, determine a fifth similarity between the replacement text corresponding to any original text and replacement texts corresponding to the other original texts based on the semantic feature of the replacement text corresponding to the original text and the semantic feature of the replacement texts corresponding to the other original texts, and determine the loss value of the replacement text corresponding to any original text based on the first similarity, the fourth similarity, and the fifth similarity.

[0221] In a possible implementation, the adjusting module 802 is configured to determine predicted information of each first text based on semantic features of the first text, the predicted information of the first text being a probability of each character in the first text being replaced by prediction; obtain labeled information of each first text, the labeled information of the first text being information of whether each character in the first text is replaced by labeling; and adjust the first feature extraction model based on the predicted information of each first text and the labeled information of each first text to obtain a second feature extraction model.

[0222] In a possible implementation, the determining module 803 is configured to obtain the second text and a category label of the second text; obtain, according to a second language model, a probability of each character in at least two spliced texts, the spliced text including the second text and a candidate text category, the second language model including the second feature extraction model; determine a target spliced text from the at least two spliced texts based on the probability of each character in the at least two spliced texts; and adjust the second language model based on the candidate text category in the target spliced text and the category label of the second text to obtain a target language model.

[0223] In a possible implementation, the determining module 803 is configured to splice the second text and the candidate text category to obtain any one of the at least two spliced texts; input the any one of the at least two spliced texts into the second language model to determine a text vector of the any one of the at least two spliced texts by the second language model, and determine the probability of each character in the any one of the at least two spliced texts based on the text vector of the any one of the at least two spliced texts.

[0224] In a possible implementation, the determining module 803 is configured to input the second text into the second language model to determine a text vector of the second text by the second language model; splice, by the second language model, the text vector of the second text and a text vector of the candidate text category to obtain a text vector of any one of the at least two spliced texts; and determine, by the second language model, the probability of each character in the any one of the at least two spliced texts based on the text vector of the any one of the at least two spliced texts.

[0225] In a possible implementation, the determining module 803 is configured to determine perplexity of each spliced text based on the probability of each character in the at least two spliced texts, the perplexity of the spliced text representing a fluency degree of the spliced text; and determine, based on the perplexity of each spliced text, that a spliced text corresponding to perplexity satisfying a condition is the target spliced text.

[0226] In a possible implementation, the determining module 803 is configured to determine the probability of each spliced text based on the probability of each character in the at least two spliced texts; and determine the perplexity of each spliced text based on the probability of each spliced text.

[0227] In a possible implementation, the apparatus further includes:

[0228] The obtaining module 801 is further configured to obtain semantic features of any first text according to the second network model;

[0229] The adjusting module 802 is further configured to adjust the second network model based on the semantic features of the plurality of first texts to obtain a third feature extraction model;

[0230] The constructing module is configured to construct the first network model based on the third feature extraction model.

[0231] The apparatus described above first trains a first language model using a plurality of first texts, the first language model including a first feature extraction model, then adjusts the first feature extraction model based on semantic features of the plurality of first texts obtained based on the first feature extraction model to obtain a second feature extraction model, and obtains a target language model based on the second feature extraction model. Two stages of training of the language model using the first texts are implemented, so that a language model with high accuracy can be trained using a small amount of first texts, reducing the dependence of the model on training data, thereby reducing the acquisition time of the training data and improving the training speed of the model and the efficiency of text classification. By training the feature extraction model, the accuracy of the feature extraction model can be improved, thereby improving the accuracy of the target language model containing the second feature extraction model, and ensuring the accuracy of text classification.

[0232] It should be understood that the above Figure 8 The apparatus provided by the embodiments of the present application, when implementing its functions, is only exemplified by the above division of functional modules, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the apparatus and method embodiments provided by the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be described here.

[0233] Next, please refer to Figure 9 , Figure 9 A structural schematic diagram of a language model training apparatus provided by the embodiments of the present application is shown in FIG. 1, which includes: Figure 9

[0234] The obtaining module 901 is configured to obtain a target text.

[0235] The obtaining module 901 is further configured to obtain occurrence probabilities of characters in at least two reconstructed texts according to a target language model, the reconstructed texts including the target text and a candidate text category, and the target language model being obtained according to the training method of any of the above language models; ​

[0236] The determination module 902 is configured to determine a target reconstructed text from the at least two reconstructed texts based on the appearance probability of each character in the at least two reconstructed texts.

[0237] The determination module 902 is further configured to determine the candidate text category in the target reconstructed text as the text category of the target text.

[0238] In a possible implementation, the acquisition module 901 is configured to splice the target text and the candidate text category to obtain any one of the at least two reconstructed texts; input the any one of the reconstructed texts into the target language model, determine a text vector of the any one of the reconstructed texts by the target language model, and determine the appearance probability of each character in the any one of the reconstructed texts based on the text vector of the any one of the reconstructed texts.

[0239] In a possible implementation, the acquisition module 901 is configured to input the target text into the target language model, and determine a text vector of the target text by the target language model; splice, by the target language model, the text vector of the target text and a text vector of the candidate text category to obtain a text vector of any one of the at least two reconstructed texts; and determine, by the target language model, the appearance probability of each character in the any one of the reconstructed texts based on the text vector of the any one of the reconstructed texts.

[0240] The determination manner of the target language model in the apparatus is as follows: a first language model is trained by using a plurality of first texts, the first language model includes a first feature extraction model, the first feature extraction model is adjusted based on semantic features of the plurality of first texts obtained based on the first feature extraction model, and then the target language model is determined based on a second feature extraction model. The two-stage training of the language model by using the first texts is implemented, so that the language model with high accuracy can be trained by using a small amount of first texts, the dependence of the model on training data is reduced, and thus the acquisition time of the training data is reduced, and the training speed of the model and the efficiency of text classification are improved. The accuracy of the feature extraction model is improved by training the first feature extraction model, and thus the accuracy of the target language model is improved, and the accuracy of text classification is ensured.

[0241] It should be understood that the above Figure 9 When the apparatuses provided in the embodiments are implemented, the above division of the functional modules is only an example and actual division of the functional modules can be differently implemented in actual application. That is, internal structures of the devices can be divided into different functional modules to implement all or part of the above described functions. In addition, the apparatuses and method embodiments provided in the above embodiments belong to the same technical field and have the same technical concept. Therefore, details are not described here.

[0242] Figure 10A structure block diagram of a terminal device 1000 provided by an example embodiment of the present application is shown. The terminal device 1000 can be a portable mobile terminal, such as a smartphone, a tablet computer, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a notebook computer, or a desktop computer. The terminal device 1000 can also be referred to as a user equipment, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0243] Generally, the terminal device 1000 includes a processor 1001 and a memory 1002.

[0244] The processor 1001 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1001 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 1001 can also include a main processor and a coprocessor, the main processor being a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit), and the coprocessor being a low-power processor for processing data in a standby state. In some embodiments, the processor 1001 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content required to be displayed by a display screen. In some embodiments, the processor 1001 can also include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning.

[0245] The memory 1002 can include one or more computer-readable storage media. The computer-readable storage media can be non-transitory. The memory 1002 can also include high-speed random access memory and can include nonvolatile memory, such as one or more magnetic disk storage devices, optical storage devices, flash memory devices, or other nonvolatile solid-state storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1002 is used to store at least one instruction for execution by the processor 1001 to implement the method for training a language model or the method for text classification provided by the method embodiments of the present application.

[0246] In some embodiments, the terminal device 1000 can also optionally include a peripheral device interface 1003 and at least one peripheral device. The processor 1001, the memory 1002, and the peripheral device interface 1003 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1003 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007, and a power supply 1009.

[0247] The peripheral device interface 1003 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1001 and the memory 1002. In some embodiments, the processor 1001, the memory 1002, and the peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1001, the memory 1002, and the peripheral device interface 1003 can be implemented on a separate chip or circuit board, and the present embodiment does not limit this.

[0248] The radio frequency circuit 1004 is configured to receive and send RF (Radio Frequency, radio frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1004 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 1004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1004 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1004 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity, wireless fidelity) network. In some embodiments, the radio frequency circuit 1004 can also include NFC (Near Field Communication, near field communication) related circuitry, which is not limited by the present application.

[0249] The display screen 1005 is configured to display a UI (User Interface, user interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1005 is a touch display screen, the display screen 1005 also has the ability to collect touch signals on or above the surface of the display screen 1005. The touch signals can be input as control signals to the processor 1001 for processing. At this time, the display screen 1005 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1005 can be one, arranged on the front panel of the terminal device 1000; in other embodiments, the display screen 1005 can be at least two, arranged on different surfaces of the terminal device 1000 or in a folding design; in other embodiments, the display screen 1005 can be a flexible display screen, arranged on a curved surface or a folding surface of the terminal device 1000. Even, the display screen 1005 can also be arranged in an irregular shape other than a rectangle, that is, a special-shaped screen. The display screen 1005 can be made of LCD (Liquid Crystal Display, liquid crystal display), OLED (Organic Light-Emitting Diode, organic light-emitting diode) and the like.

[0250] The camera component 1006 is configured to capture images or videos. Optionally, the camera component 1006 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is disposed on the front panel of the terminal device, and the rear-facing camera is disposed on the back of the terminal device. In some embodiments, the rear-facing camera is at least two, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a long-focus camera, to realize the background blur function by fusing the main camera and the depth-of-field camera, the panorama shooting and VR (Virtual Reality) shooting function by fusing the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera component 1006 can further include a flash. The flash can be a single-color-temperature flash or a dual-color-temperature flash. The dual-color-temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0251] The audio circuit 1007 can include a microphone and a speaker. The microphone is configured to capture sound waves of a user and the environment, and convert the sound waves into an electrical signal input to the processor 1001 for processing or to the radio frequency circuit 1004 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, which are respectively disposed at different parts of the terminal device 1000. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is configured to convert an electrical signal from the processor 1001 or the radio frequency circuit 1004 into sound waves. The speaker can be a traditional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert an electrical signal into a sound wave audible to humans, but also convert an electrical signal into an inaudible sound wave to humans for ranging purposes. In some embodiments, the audio circuit 1007 can further include a headphone jack.

[0252] The power supply 1009 is configured to supply power to each component in the terminal device 1000. The power supply 1009 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1009 includes a rechargeable battery, the rechargeable battery can be a wired charging battery or a wireless charging battery. The wired charging battery is a battery that is charged through a wired line, and the wireless charging battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0253] In some embodiments, the terminal device 1000 further includes one or more sensors 1010. The one or more sensors 1010 include, but are not limited to, an acceleration sensor 1011, a gyroscope sensor 1012, a pressure sensor 1013, an optical sensor 1015, and a proximity sensor 1016.

[0254] The acceleration sensor 1011 can detect the acceleration magnitude in three coordinate axes of the coordinate system established by the terminal device 1000. For example, the acceleration sensor 1011 can be used to detect the components of the gravitational acceleration in three coordinate axes. The processor 1001 can control the display screen 1005 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signals collected by the acceleration sensor 1011. The acceleration sensor 1011 can also be used for game or user motion data collection.

[0255] The gyroscope sensor 1012 can detect the body direction and rotation angle of the terminal device 1000, and the gyroscope sensor 1012 can collect the 3D motion of the user to the terminal device 1000 in cooperation with the acceleration sensor 1011. The processor 1001 can implement the following functions according to the data collected by the gyroscope sensor 1012: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization when shooting, game control, and inertial navigation.

[0256] The pressure sensor 1013 can be arranged on the side frame of the terminal device 1000 and / or the lower layer of the display screen 1005. When the pressure sensor 1013 is arranged on the side frame of the terminal device 1000, the user's holding signal to the terminal device 1000 can be detected, and the left-hand or right-hand recognition or shortcut operation can be performed by the processor 1001 according to the holding signal collected by the pressure sensor 1013. When the pressure sensor 1013 is arranged on the lower layer of the display screen 1005, the processor 1001 can control the operable control on the UI interface according to the user's pressure operation on the display screen 1005. The operable control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0257] The optical sensor 1015 is used to collect the ambient light intensity. In one embodiment, the processor 1001 can control the display brightness of the display screen 1005 according to the ambient light intensity collected by the optical sensor 1015. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1005 is increased; when the ambient light intensity is low, the display brightness of the display screen 1005 is decreased. In another embodiment, the processor 1001 can also dynamically adjust the shooting parameters of the camera assembly 1006 according to the ambient light intensity collected by the optical sensor 1015.

[0258] The proximity sensor 1016, also known as a distance sensor, is typically installed on the front panel of the terminal device 1000. The proximity sensor 1016 is used to detect the distance between the user and the front of the terminal device 1000. In one embodiment, when the proximity sensor 1016 detects that the distance between the user and the front of the terminal device 1000 is gradually decreasing, the processor 1001 controls the display screen 1005 to switch from a screen-on state to a screen-off state; when the proximity sensor 1016 detects that the distance between the user and the front of the terminal device 1000 is gradually increasing, the processor 1001 controls the display screen 1005 to switch from a screen-off state to a screen-on state.

[0259] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on the terminal device 1000, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0260] Figure 11 This is a schematic diagram of the server structure provided in the embodiments of this application. The server 1100 can vary considerably due to different configurations or performance. It may include one or more processors 1101 and one or more memories 1102. The one or more memories 1102 store at least one line of program code, which is loaded and executed by the one or more processors 1101 to implement the language model training method or text classification method provided in the above-described method embodiments. For example, the processor 1101 is a CPU. Of course, the server 1100 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 1100 may also include other components for implementing device functions, which will not be elaborated here.

[0261] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one piece of program code, which is loaded and executed by a processor to enable an electronic device to implement any of the above-described language model training methods or text classification methods.

[0262] Optionally, the aforementioned computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0263] In an exemplary embodiment, a computer program or computer program product is also provided, which stores at least one computer instruction, which is loaded and executed by a processor to enable the computer to implement any of the above-described language model training methods or text classification methods.

[0264] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0265] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0266] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for training a language model, characterized in that, The method comprises: obtaining a plurality of first texts; for any one first text, obtaining the appearance probability of each character in the any one first text according to a first network model; adjusting the first network model based on the appearance probability of each character in the plurality of first texts to obtain a first language model, the first language model comprising a first feature extraction model; obtaining the semantic feature of the any one first text according to the first feature extraction model; adjusting the first feature extraction model based on the semantic feature of the plurality of first texts to obtain a second feature extraction model; obtaining a second text and a category label of the second text; obtaining the appearance probability of each character in at least two spliced texts according to a second language model, the spliced texts comprising the second text and a candidate text category, the second language model comprising the second feature extraction model; determining a target spliced text from the at least two spliced texts based on the appearance probability of each character in the at least two spliced texts; adjusting the second language model based on the candidate text category in the target spliced text and the category label of the second text to obtain a target language model.

2. The method of claim 1, wherein, The adjusting of the first feature extraction model based on the semantic feature of the plurality of first texts to obtain a second feature extraction model comprises: determining the loss value of each first text based on the semantic feature of the plurality of first texts; determining the loss value of the first feature extraction model based on the loss value of each first text; adjusting the first feature extraction model based on the loss value of the first feature extraction model to obtain a second feature extraction model.

3. The method of claim 2, wherein, The any one first text is any one original text or a replacement text corresponding to the any one original text, the replacement text corresponding to the any one original text being a text obtained by replacing characters in the any one original text; The determining of the loss value of each first text based on the semantic feature of the plurality of first texts comprises: for the any one original text, determining the loss value of the any one original text based on the semantic feature of each original text and the semantic feature of the replacement text corresponding to the any one original text; for the replacement text corresponding to the any one original text, determining the loss value of the replacement text corresponding to the any one original text based on the semantic feature of each original text and the semantic feature of the replacement text corresponding to the any one original text.

4. The method of claim 3, wherein, The determining of the loss value of the any one original text based on the semantic feature of each original text and the semantic feature of the replacement text corresponding to the any one original text comprises: determining the first similarity between the any one original text and the replacement text corresponding to the any one original text based on the semantic feature of the any one original text and the semantic feature of the replacement text corresponding to the any one original text; determine a second similarity between the any one original text and other original texts based on the semantic feature of the any one original text and the semantic feature of the other original texts, the other original texts being the original texts other than the any one original text among the original texts; determine a third similarity between the any one original text and the replaced text corresponding to the other original text based on the semantic feature of the any one original text and the semantic feature of the replaced text corresponding to the other original text; determine the loss value of the any one original text based on the first similarity, the second similarity and the third similarity.

5. The method of claim 3, wherein, The determining the loss value of the replaced text corresponding to the any one original text based on the semantic feature of the original texts and the semantic feature of the replaced texts corresponding to the original texts comprises: determine a first similarity between the any one original text and the replaced text corresponding to the any one original text based on the semantic feature of the any one original text and the semantic feature of the replaced text corresponding to the any one original text; determine a fourth similarity between the replaced text corresponding to the any one original text and other original texts based on the semantic feature of the replaced text corresponding to the any one original text and the semantic feature of the other original texts, the other original texts being the original texts other than the any one original text among the original texts; determine a fifth similarity between the replaced text corresponding to the any one original text and the replaced text corresponding to the other original text based on the semantic feature of the replaced text corresponding to the any one original text and the semantic feature of the replaced text corresponding to the other original text; determine the loss value of the replaced text corresponding to the any one original text based on the first similarity, the fourth similarity and the fifth similarity.

6. The method of claim 1, wherein, The adjusting the first feature extraction model based on the semantic features of the plurality of first texts to obtain a second feature extraction model comprises: determine predicted information of each first text based on the semantic feature of the first text, the predicted information of the first text being the probability of each character in the first text being replaced obtained by prediction; obtain labeled information of each first text, the labeled information of the first text being the information of whether each character in the first text is replaced obtained by labeling; adjust the first feature extraction model based on the predicted information of each first text and the labeled information of each first text to obtain a second feature extraction model.

7. The method of claim 1, wherein, The obtaining the probability of occurrence of each character in the at least two spliced texts according to the second language model comprises: splicing the second text and the candidate text category to obtain any one of the at least two spliced texts; inputting the any one spliced text into the second language model to determine a text vector of the any one spliced text by the second language model, and determining the probability of occurrence of each character in the any one spliced text based on the text vector of the any one spliced text.

8. The method of claim 1, wherein, The method comprises: obtaining the probability of occurrence of each character in the at least two spliced texts according to a second language model, comprising: inputting the second text into the second language model to determine a text vector of the second text by the second language model; splicing the text vector of the second text and the text vector of the candidate text category by the second language model to obtain a text vector of any one of the at least two spliced texts; 9. The method of claim 1, wherein, determining the probability of occurrence of each character in the any one of the at least two spliced texts by the second language model based on the text vector of the any one of the at least two spliced texts. The method comprises: determining the perplexity of each spliced text based on the probability of occurrence of each character in the at least two spliced texts, wherein the perplexity of the spliced text represents the fluency of the spliced text; 10. The method of claim 9, wherein, determining the spliced text corresponding to the perplexity meeting the condition as the target spliced text based on the perplexity of each spliced text. The method comprises: determining the probability of occurrence of each character in the at least two spliced texts based on the probability of occurrence of each character in the at least two spliced texts, wherein the perplexity of the spliced text represents the fluency of the spliced text; 11. The method according to any one of claims 1 to 6, characterized in that, determining the perplexity of each spliced text based on the probability of occurrence of each character in the at least two spliced texts. The method comprises: obtaining the probability of occurrence of each character in the at least two spliced texts according to a second language model, comprising: inputting the second text into the second language model to determine a text vector of the second text by the second language model; 12. A method of text classification, characterized by, splicing the text vector of the second text and the text vector of the candidate text category by the second language model to obtain a text vector of any one of the at least two spliced texts; determining the probability of occurrence of each character in the any one of the at least two spliced texts by the second language model based on the text vector of the any one of the at least two spliced texts. The method comprises: obtaining the probability of occurrence of each character in the at least two spliced texts according to a second language model, comprising: inputting the second text into the second language model to determine a text vector of the second text by the second language model; 13. The method of claim 12, wherein, splicing the text vector of the second text and the text vector of the candidate text category by the second language model to obtain a text vector of any one of the at least two spliced texts; determining the probability of occurrence of each character in the any one of the at least two spliced texts by the second language model based on the text vector of the any one of the at least two spliced texts. ​ 14. The method of claim 12, wherein, ​ ​ concatenate, by the target language model, the text vector of the target text and the text vector of the candidate text category to obtain a text vector of any one of the at least two reconstructed texts; determine, by the target language model based on the text vector of the any one of the reconstructed texts, an occurrence probability of each character in the any one of the reconstructed texts.

15. An apparatus for training a language model, the apparatus comprising: The apparatus comprises: an obtaining module configured to obtain a plurality of first texts; the obtaining module is further configured to, for any one of the first texts, obtain an occurrence probability of each character in the any one of the first texts according to a first network model; an adjusting module configured to adjust the first network model based on the occurrence probability of each character in the plurality of first texts to obtain a first language model, the first language model comprising a first feature extraction model; the obtaining module is further configured to obtain a semantic feature of the any one of the first texts according to the first feature extraction model; the adjusting module is further configured to adjust the first feature extraction model based on the semantic feature of the plurality of first texts to obtain a second feature extraction model; a determining module configured to obtain a second text and a category label of the second text; obtain an occurrence probability of each character in at least two concatenated texts according to a second language model, the concatenated texts comprising the second text and a candidate text category, the second language model comprising the second feature extraction model; determine a target concatenated text from the at least two concatenated texts based on the occurrence probability of each character in the at least two concatenated texts; and adjust the second language model based on the candidate text category in the target concatenated text and the category label of the second text to obtain a target language model.

16. The apparatus of claim 15, wherein, the adjusting module is configured to determine a loss value of each of the first texts based on the semantic feature of the plurality of first texts; determine a loss value of the first feature extraction model based on the loss value of each of the first texts; and adjust the first feature extraction model based on the loss value of the first feature extraction model to obtain the second feature extraction model.

17. The apparatus of claim 16, wherein, the any one of the first texts is any one of original texts or replacement texts corresponding to the any one of the original texts, the replacement texts corresponding to the any one of the original texts being texts obtained by replacing characters in the any one of the original texts; the adjusting module is configured to, for the any one of the original texts, determine a loss value of the any one of the original texts based on a semantic feature of each of the original texts and a semantic feature of the replacement text corresponding to the any one of the original texts; and for the replacement text corresponding to the any one of the original texts, determine a loss value of the replacement text corresponding to the any one of the original texts based on the semantic feature of each of the original texts and the semantic feature of the replacement text corresponding to the any one of the original texts.

18. The apparatus of claim 17, wherein, The adjustment module is configured to determine a first similarity between the any original text and the replacement text corresponding to the any original text based on the semantic feature of the any original text and the semantic feature of the replacement text corresponding to the any original text; determine a second similarity between the any original text and other original texts based on the semantic feature of the any original text and the semantic features of the other original texts, the other original texts being the original texts other than the any original text among the original texts; determine a third similarity between the any original text and the replacement texts corresponding to the other original texts based on the semantic feature of the any original text and the semantic features of the replacement texts corresponding to the other original texts; and determine a loss value of the any original text based on the first similarity, the second similarity, and the third similarity.

19. The apparatus of claim 17, wherein, The adjustment module is configured to determine a first similarity between the any original text and the replacement text corresponding to the any original text based on the semantic feature of the any original text and the semantic feature of the replacement text corresponding to the any original text; determine a fourth similarity between the replacement text corresponding to the any original text and other original texts based on the semantic feature of the replacement text corresponding to the any original text and the semantic features of the other original texts, the other original texts being the original texts other than the any original text among the original texts; determine a fifth similarity between the replacement text corresponding to the any original text and the replacement texts corresponding to the other original texts based on the semantic feature of the replacement text corresponding to the any original text and the semantic features of the replacement texts corresponding to the other original texts; and determine a loss value of the replacement text corresponding to the any original text based on the first similarity, the fourth similarity, and the fifth similarity.

20. The apparatus of claim 15, wherein, The adjustment module is configured to determine prediction information of each first text based on the semantic feature of each first text, the prediction information of the first text being a probability of each character in the first text being replaced obtained by prediction; obtain labeled information of each first text, the labeled information of the first text being information of whether each character in the first text is replaced obtained by labeling; and adjust the first feature extraction model based on the prediction information of each first text and the labeled information of each first text to obtain a second feature extraction model.

21. The apparatus of claim 15, wherein, The determination module is configured to splice the second text and the candidate text category to obtain any one of the at least two spliced texts; input the any one of the spliced texts into the second language model to determine a text vector of the any one of the spliced texts by the second language model, and determine an occurrence probability of each character in the any one of the spliced texts based on the text vector of the any one of the spliced texts.

22. The apparatus of claim 15, wherein, The determination module is configured to input the second text into the second language model to determine a text vector of the second text by the second language model; splice the text vector of the second text and a text vector of the candidate text category by the second language model to obtain a text vector of any one of the at least two spliced texts; and determine, by the second language model, an occurrence probability of each character in the any one of the at least two spliced texts based on the text vector of the any one of the at least two spliced texts.

23. The apparatus of claim 15, wherein, The determination module is configured to determine a perplexity of each of the at least two spliced texts based on the occurrence probability of each character in the at least two spliced texts, where the perplexity of the spliced text represents a fluency degree of the spliced text; and determine, based on the perplexity of each of the at least two spliced texts, that a spliced text corresponding to a perplexity meeting a condition is the target spliced text.

24. The apparatus of claim 23, wherein, The determination module is configured to determine an occurrence probability of each of the at least two spliced texts based on the occurrence probability of each character in the at least two spliced texts; and determine a perplexity of each of the at least two spliced texts based on the occurrence probability of each of the at least two spliced texts.

25. The apparatus of any one of claims 15 to 20, wherein, The apparatus further includes: The acquisition module is further configured to acquire semantic features of the any one of the first texts according to a second network model. The adjustment module is further configured to adjust the second network model based on the semantic features of the plurality of first texts to obtain a third feature extraction model. The construction module is configured to construct the first network model based on the third feature extraction model.

26. A text classification apparatus characterized by comprising: The apparatus includes: An acquisition module is configured to acquire a target text. The acquisition module is further configured to acquire occurrence probabilities of characters in at least two reconstructed texts according to a target language model, where the reconstructed texts include the target text and a candidate text category, and the target language model is obtained according to the method in any one of claims 1 to 11. A determination module is configured to determine a target reconstructed text from the at least two reconstructed texts based on the occurrence probabilities of the characters in the at least two reconstructed texts. The determination module is further configured to determine the candidate text category in the target reconstructed text as a text category of the target text.

27. The apparatus of claim 26, wherein, The acquisition module is configured to splice the target text and the candidate text category to obtain any one of the at least two reconstructed texts; input the any one of the at least two reconstructed texts into the target language model to determine a text vector of the any one of the at least two reconstructed texts by the target language model; and determine occurrence probabilities of characters in the any one of the at least two reconstructed texts based on the text vector of the any one of the at least two reconstructed texts.

28. The apparatus of claim 26, wherein, The acquisition module is configured to input the target text into the target language model to determine a text vector of the target text by the target language model; splice the text vector of the target text and a text vector of the candidate text category by the target language model to obtain a text vector of any one of the at least two reconstructed texts; and determine, by the target language model, occurrence probabilities of characters in the any one of the at least two reconstructed texts based on the text vector of the any one of the at least two reconstructed texts.

29. An electronic device, comprising: The electronic device includes a processor and a memory, and the memory stores at least one program code, which is loaded and executed by the processor, so that the electronic device implements the language model training method according to any one of claims 1 to 11 or the text classification method according to any one of claims 12 to 14.

30. A computer-readable storage medium, characterized in that, The computer readable storage medium stores at least one program code, which is loaded and executed by the processor, so that the computer implements the language model training method according to any one of claims 1 to 11 or the text classification method according to any one of claims 12 to 14.

31. A computer program product, characterised in that, The computer program product stores at least one computer instruction, which is loaded and executed by the processor, so that the computer implements the language model training method according to any one of claims 1 to 11 or the text classification method according to any one of claims 12 to 14.

Images