Training method of language model, text classification method and related device

Abstract

Embodiments of the present application relate to the field of artificial intelligence, and provide a language model training method, a text classification method and related devices. The language model training method comprises: obtaining a first text feature queue and a second text feature queue; performing feature extraction on a first text sample based on a first language model to obtain a first text feature; classifying the first text sample based on the first text feature to obtain a first classification correct probability of the first text sample; determining a classification loss based on a sample label of the first text sample and the first classification correct probability of the first text sample; determining a contrast loss based on the first text feature, the sample label of the first text sample, the first text feature queue and the second text feature queue; determining a total loss based on the classification loss and the contrast loss; and iteratively updating the first language model based on the total loss to obtain a target language model. The present application can improve the robustness and generalization of the language model, and improve the accuracy of text classification.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically to a method for training a language model, a text classification method, and related apparatus. Background Technology

[0002] Large Language Models (LLMs), with their superior natural language generation capabilities, such as closed-source models GPT-4 and Gemini, and open-source models ChatGLM3 and LLAMA, are able to generate grammatically correct, persuasive text that is highly similar to human writing. LLMs have been applied in various fields, bringing convenience and efficiency improvements and playing a crucial role in areas such as education, content creation, language translation, programming assistance, scientific research, information retrieval, entertainment, and games. However, technological development has two sides, and LLMs also present some potential harms and risks. On the one hand, due to the inherent "illusion" problem of LLMs, they may promote the spread of misinformation. On the other hand, LLMs can be maliciously abused, such as generating fake news, spreading misleading information, and affecting educational equity. Therefore, there is an urgent need to research effective artificial intelligence (AI) generated text detection technologies to promote the healthy development of LLMs and prevent their misuse. The core objective of AI-generated text detection is to accurately identify whether a given text sample was written by a human or generated by an AI system. Currently, AI-generated text detection mainly includes methods based on supervised learning. Supervised learning methods rely on a large amount of human-generated text and AI-generated text to train classifiers. Based on different modeling features, supervised learning methods can be further divided into methods based on pre-trained language models and methods based on linguistic features. Methods based on pre-trained language models mainly add multi-layer perceptrons (MLPs) and fine-tune them after pre-trained models such as BERT (Bidirectional Encoder Representations from Transformers) and RoBERTa[9] to achieve binary classification of text. Methods based on linguistic features focus on using artificially constructed linguistic statistical features combined with traditional machine learning models to distinguish between human text and AI text. However, the robustness and generalization of language models trained by existing technologies are low. Summary of the Invention

[0003] This application provides a language model training method, a text classification method, and related apparatus, which can improve the robustness and generalization of the language model and improve the accuracy of text classification.

[0004] In a first aspect, embodiments of this application provide a method for training a language model, the method comprising:

[0005] Obtain a first text feature queue and a second text feature queue, wherein the sample label of each text feature in the first text feature queue is a first text type, and the sample label of each text feature in the second text feature queue is a second text type;

[0006] Obtain the first language model, the first text sample, and the corresponding sample labels;

[0007] Based on the first language model, feature extraction is performed on the first text sample to obtain the first text features;

[0008] Based on the first text features, the first text sample is classified to obtain the first classification correct probability of the first text sample;

[0009] The classification loss is determined based on the sample label of the first text sample and the first classification correct probability of the first text sample;

[0010] The contrast loss is determined based on the first text feature, the sample label of the first text sample, the first text feature queue, and the second text feature queue.

[0011] The total loss is determined based on the classification loss and the contrast loss.

[0012] The model parameters of the first language model are iteratively updated based on the total loss until a preset stopping condition is met, thereby obtaining the target language model.

[0013] In one implementation, obtaining the first language model, the first text sample, and the corresponding sample label includes:

[0014] Obtain the initial text sample and its corresponding sample label;

[0015] The initial text sample is augmented to obtain the target augmented text sample;

[0016] The initial text sample and the target augmented text sample are placed into the target sample set;

[0017] A portion of the samples in the target sample set are identified as the first text sample.

[0018] In one implementation, the method for training the language model includes:

[0019] Obtain the second language model;

[0020] Based on the second language model, feature extraction is performed on the second text sample to obtain the second text features, wherein the second text sample is the sample in the target sample set after removing the first text sample;

[0021] Calculate the first similarity between the second text feature and multiple first sample prototypes respectively, wherein the sample labels of the multiple first sample prototypes are the same as the sample labels of the second text feature;

[0022] The second text features with a first similarity less than a preset similarity are identified as hard sample features;

[0023] If the sample label of the difficult sample feature is a first text type, then the difficult sample is placed in the first text feature queue;

[0024] If the sample label of the difficult sample feature is a second text type, then the difficult sample is placed in the second text feature queue.

[0025] In one implementation, the step of data augmentation of the initial text sample to obtain the target augmented text sample includes:

[0026] The initial text sample is split into multiple text characters; the multiple text characters in the initial text sample are subjected to character-level perturbation a target preset number of times to obtain the target augmented text sample, wherein the character-level perturbation includes random character insertion, character deletion, character replacement, or character swapping.

[0027] Alternatively, the initial text sample can be split into multiple text words; the multiple text words in the initial text sample can be subjected to word-level perturbation a target preset number of times to obtain the target augmented text sample, wherein the word-level perturbation is synonym replacement, word insertion, word deletion or random word swapping.

[0028] In one implementation, determining the total loss based on the classification loss and the contrast loss includes:

[0029] Obtain multiple second sample prototypes whose sample labels are the same as the first text sample;

[0030] Calculate the second similarity between multiple second sample prototypes and the first text sample respectively;

[0031] The probability of attribution is determined based on multiple second similarities, which shows that the sample label of the first text sample is the same as the sample label of multiple second sample prototypes.

[0032] The second classification correctness probability of the first text sample is determined based on the probability that the sample label of the first text sample is the same as the sample label of multiple second sample prototypes;

[0033] The prototype update loss is determined based on the second classification correct probability of the first text sample;

[0034] The total loss is determined based on the prototype update loss, the classification loss, and the comparison loss.

[0035] In one implementation, obtaining the second language model includes:

[0036] Obtain the model parameters of the first language model during the current batch training and the model parameters of the second language model during the previous batch training;

[0037] The current model parameters of the second language model are determined based on the model parameters of the first language model during the current batch training, the model parameters of the second language model during the previous batch training, and the preset momentum coefficient.

[0038] Secondly, embodiments of this application provide a text classification method, the text classification method comprising:

[0039] Get the text to be categorized;

[0040] The text to be classified is input into a target language model to obtain target text features, wherein the target language model is any one of the target language models described in the first aspect;

[0041] Based on the target text features, the text type of the text to be classified is obtained.

[0042] Thirdly, embodiments of this application provide a language model training apparatus, which has the function of implementing a training method corresponding to the language model provided in the first aspect above. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above function, and the modules can be software and / or hardware.

[0043] Fourthly, embodiments of this application provide a text classification apparatus having functions corresponding to the text classification method provided in the second aspect above. These functions can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and these modules can be software and / or hardware.

[0044] Fifthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the language model training method as described in the first aspect or the text classification method as described in the second aspect.

[0045] In a sixth aspect, embodiments of this application provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the language model training method described in the first aspect or the text classification method described in the second aspect.

[0046] In a seventh aspect, embodiments of this application provide a chip that includes a processor coupled to a transceiver of a terminal device, for executing the technical solutions provided in the first and second aspects of embodiments of this application.

[0047] Eighthly, embodiments of this application provide a chip system including a processor for supporting terminal devices in implementing the functions involved in the first and second aspects described above.

[0048] In one possible design, the aforementioned chip system also includes a memory for storing program instructions and data necessary for the terminal. The chip system can be composed of chips or may include chips and other discrete components.

[0049] Ninthly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on a computer, cause the computer to execute the language model training method described in the first aspect or the text classification method described in the second aspect.

[0050] Compared to existing technologies, in this application embodiment, a first text feature queue and a second text feature queue are obtained. The sample label of each text feature in the first text feature queue is a first text type, and the sample label of each text feature in the second text feature queue is a second text type. A first language model, a first text sample, and corresponding sample labels are obtained. Features are extracted from the first text sample based on the first language model to obtain first text features. The first text sample is classified based on the first text features to obtain a first classification correct probability of the first text sample. A classification loss is determined based on the sample labels of the first text sample and the first classification correct probability of the first text sample. A contrast loss is determined based on the first text features, the sample labels of the first text sample, the first text feature queue, and the second text feature queue. A total loss is determined based on the classification loss and the contrast loss. The model parameters of the first language model are iteratively updated based on the total loss until a preset stopping condition is met to obtain the target language model. This application can improve the robustness and generalization of the language model and improve the accuracy of text classification. Attached Figure Description

[0051] The objectives, features, and advantages of the embodiments of this application will become readily understood by referring to the accompanying drawings and the detailed description of the embodiments. Wherein:

[0052] Figure 1 This is a schematic diagram of a text classification system for training a language model in an embodiment of this application;

[0053] Figure 2 A flowchart illustrating a language model training method provided in an embodiment of this application;

[0054] Figure 3 A schematic diagram of information processing in a language model training method provided in an embodiment of this application;

[0055] Figure 4 A flowchart illustrating a text classification method provided in this application embodiment;

[0056] Figure 5 This is a schematic diagram of the structure of a language model training device according to an embodiment of this application;

[0057] Figure 6 This is a schematic diagram of the structure of the text classification device according to an embodiment of this application;

[0058] Figure 7 This is a schematic diagram of the structure of a computing device according to an embodiment of this application;

[0059] Figure 8 This is a schematic diagram of the structure of a mobile phone in one embodiment of this application;

[0060] Figure 9 This is a schematic diagram of a server structure in one embodiment of this application.

[0061] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0062] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products, or devices. The division of modules in the embodiments of this application is merely a logical division; in actual applications, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be omitted or not performed. Additionally, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interface, indirect coupling between modules, or electrical or other similar forms of communication connection, none of which are limited in the embodiments of this application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed among multiple circuit modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiments of this application.

[0063] Generative artificial intelligence methods and software have been widely used, which has created potential risks and hidden dangers.

[0064] The general-purpose model is currently widely used in applications such as chat dialogue, text editing, artistic creation, coding, mathematical reasoning, and bioinformatics. Although it has created many new business models and is very powerful, after the general-purpose model is launched for users, there are mainly algorithmic risks, data risks, and application risks in the three types of applications: translation, chat, and collaboration.

[0065] This application also provides a language model training method, a text classification method, and related apparatus, which can be applied to a text classification system. The text classification system may include a language model training device, which can be deployed integratedly or separately. The language model training device is at least used to acquire a first text feature queue and a second text feature queue. The sample label of each text feature in the first text feature queue is a first text type, and the sample label of each text feature in the second text feature queue is a second text type. The device acquires a first language model, a first text sample, and corresponding sample labels. It extracts features from the first text sample based on the first language model to obtain first text features. It classifies the first text sample based on the first text features to obtain a first classification correct probability of the first text sample. It determines a classification loss based on the sample labels of the first text sample and the first classification correct probability of the first text sample. It determines a contrastive loss based on the first text features, the sample labels of the first text sample, the first text feature queue, and the second text feature queue. It determines a total loss based on the classification loss and the contrastive loss. It iteratively updates the model parameters of the first language model based on the total loss until a preset stopping condition is met to obtain a target language model.

[0066] The solutions provided in this application involve technologies such as Artificial Intelligence (AI), Natural Language Processing (NLP), and Machine Learning (ML), which are specifically illustrated through the following embodiments:

[0067] AI, or Artificial Intelligence, refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, Artificial Intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine capable of reacting in a manner similar to human intelligence. Artificial Intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0068] AI technology is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0069] Compared to existing technologies, in this application embodiment, a first text feature queue and a second text feature queue are obtained. The sample label of each text feature in the first text feature queue is a first text type, and the sample label of each text feature in the second text feature queue is a second text type. A first language model, a first text sample, and the corresponding sample label are obtained. Features are extracted from the first text sample based on the first language model to obtain the first text features. The first text sample is classified based on the first text features to obtain the first classification correct probability of the first text sample. A classification loss is determined based on the sample label of the first text sample and the first classification correct probability of the first text sample. A contrast loss is determined based on the first text features, the sample label of the first text sample, the first text feature queue, and the second text feature queue. A total loss is determined based on the classification loss and the contrast loss. The model parameters of the first language model are iteratively updated based on the total loss until a preset stopping condition is met to obtain the target language model. This application can improve the robustness and generalization of the language model and improve the accuracy of text classification.

[0070] In some implementations, reference is made to Figure 1 The language model training method provided in this application embodiment can be based on Figure 1 The illustration shows an implementation of a text classification system. This text classification system may include an electronic device 100 and a memory 200. The electronic device 100 may be a server or a terminal device.

[0071] It should be noted that the server involved in the embodiments of this application can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0072] The terminal devices involved in the embodiments of this application can be devices that provide voice and / or data connectivity to users, handheld devices with wireless connectivity, or other processing devices connected to a wireless modem. Examples include mobile phones (or "cellular" phones) and computers with mobile terminals, such as portable, pocket-sized, handheld, computer-embedded, or vehicle-mounted mobile devices that exchange voice and / or data with a wireless access network. Examples include Personal Communication Service (PCS) phones, cordless phones, Session Initiation Protocol (SIP) phones, Wireless Local Loop (WLL) stations, Personal Digital Assistants (PDAs), and other similar devices.

[0073] Reference Figure 2 , Figure 2 This is a flowchart illustrating a language model training method provided in an embodiment of this application. The method can be executed by a language model training device. The method includes steps 101-108:

[0074] Step 101: Obtain the first text feature queue and the second text feature queue.

[0075] In the first text feature queue, the sample label of each text feature is the first text type, and in the second text feature queue, the sample label of each text feature is the second text type.

[0076] In this embodiment, the first text type is "human text," represented by 0; the second text type is "AI text," represented by 1. The first text type is the human text feature queue q. r The second text type is the AI ​​text feature queue q. f .

[0077] In this embodiment of the application, during the initial stage of model training, the human text feature queue q is initialized. r AI text feature queue q f Human text feature queue q r AI text feature queue q f The queue length is K. Human text feature queue q r AI text feature queue q f Each text feature has a preset number of dimensions.

[0078] In this embodiment of the application, the first text feature queue and the second text feature queue are updated dynamically.

[0079] Step 102: Obtain the first language model, the first text sample, and the corresponding sample label.

[0080] In this embodiment of the application, the first language model can be a RoBERTa model structure and initialized with pre-trained parameters.

[0081] In one specific embodiment, such as Figure 3 As shown, let the first language model be the pre-trained language model f. q First text sample x i The number can be multiple, for example, the first text sample x i The quantity is N. The first text sample x i The corresponding sample labels are of either the first text type or the second text type, and can be pre-labeled manually.

[0082] In this embodiment of the application, obtaining the first language model, the first text sample, and the corresponding sample label includes:

[0083] (1) Obtain the initial text sample and the corresponding sample label.

[0084] In this embodiment of the application, the initial text sample and the corresponding sample label are manually pre-annotated.

[0085] In a specific embodiment, such as Figure 3 As shown, the initial text sample x i The set is Where x i Let y represent the i-th initial text sample. i ∈{0,1} represents the sample label of the i-th initial text sample, where 0 represents "human-written" and 1 represents "AI-generated", and N represents the sample data.

[0086] (2) Data augmentation is performed on the initial text sample to obtain the target augmented text sample.

[0087] Data augmentation specifically involves text augmentation. Text augmentation encompasses various transformations of text data, such as substitution and back-translation methods based on masked language models. Masked language models, through pre-training, acquire the ability to predict masked words in text based on context, and can be used for text augmentation. Specifically, some words in the text are replaced with masks, and then the trained masked language model predicts the word at that location to complete the sentence information. Back-translation methods involve translating the original document into another language, and then translating it back to obtain new text in the original language, thus achieving data augmentation.

[0088] To better perform data augmentation, in a specific embodiment, data augmentation is performed on the initial text sample to obtain the target augmented text sample, including: splitting the initial text sample into multiple text characters; performing character-level perturbation on the multiple text characters in the initial text sample a target preset number of times to obtain the target augmented text sample, wherein the character-level perturbation includes random character insertion, character deletion, character replacement, or character swapping.

[0089] Specifically, the number of characters in the multiple text characters obtained from the initial text sample is obtained, and the target preset number of perturbations is determined based on the number of characters. Specifically, the range of perturbation frequency is determined based on the number of characters, and the target preset number of perturbations is randomly selected within this range. The target preset number of character-level perturbations is then applied to the multiple text characters in the initial text sample to obtain the target augmented text sample.

[0090] Specifically, the initial text sample x i The number of characters len(x) i The relationship between the perturbation number range [1, T] and the target preset number t is shown in the following formula.

[0091]

[0092] t∈[1,T]

[0093] After obtaining the target preset number of times t, the initial text sample x i Multiple text characters in the target text are perturbed by a predetermined number of times (t) to obtain the target augmented text sample. The data augmentation operation is denoted as π. Target Augmented Text Sample

[0094] In another specific embodiment, data augmentation is performed on the initial text sample to obtain a target augmented text sample, including: splitting the initial text sample into multiple text words; and performing word-level perturbation on the multiple text words in the initial text sample a target preset number of times to obtain the target augmented text sample. The word-level perturbation includes synonym replacement, word insertion, word deletion, or random word swapping. Each word consists of at least two characters.

[0095] Specifically, the number of words in the multiple text words obtained from the initial text sample is obtained, and the target preset number of operations is determined based on the number of words. Specifically, the range of perturbation number is determined based on the number of words, and the target preset number of operations is randomly selected within the range of perturbation number. The target preset number of word-level operations are then performed on multiple text words in the initial text sample to obtain the target augmented text sample.

[0096] In another specific embodiment, data augmentation is performed on the initial text sample to obtain a target augmented text sample, including: splitting the initial text sample into multiple text characters; performing a first preset number of character-level perturbations on the multiple text characters in the initial text sample to obtain the initial augmented text sample, wherein the character-level perturbations include random character insertion, character deletion, character replacement, or character swapping. The initial text sample is then split into multiple text words; performing a second preset number of word-level perturbations on the multiple text words in the initial text sample to obtain the target augmented text sample, wherein the word-level perturbations include synonym replacement, word insertion, word deletion, or random word swapping. Each word consists of at least two characters.

[0097] Specifically, the number of words in the initial text sample is obtained from multiple text words, and a target preset number of times is determined based on the number of words. Specifically, a range of perturbation times is determined based on the number of words, and a target preset number of times is randomly selected within this range. This target preset number of times is divided into a first preset number of times and a second preset number of times. The ratio of the first preset number of times to the second preset number of times is the ratio of the number of characters to the number of words.

[0098] (3) Put the initial text sample and the target augmented text sample into the target sample set.

[0099] In this embodiment of the application, N initial text samples x i and N target augmented text samples Add the text samples to the target sample set. The target sample set contains 2N text samples.

[0100] (4) Select a portion of the target sample set as the first text sample.

[0101] In a specific embodiment, such as Figure 3 As shown, the initial text sample x in the target sample set i This is identified as the first text sample. The initial text sample x is removed from the target sample set. i The latter was determined to be the second text sample.

[0102] In another specific embodiment, N text samples are randomly selected from the target sample set as the first text samples.

[0103] Step 103: Extract features from the first text sample based on the first language model to obtain the first text features.

[0104] In this embodiment, the first text sample is input into a first language model for feature extraction to obtain the first text features. Specifically, the first text sample x... i Input first language model f qFeature extraction is performed to obtain the first text feature h. i[CLS] .

[0105] Step 104: Classify the first text sample based on the first text features to obtain the first classification correct probability of the first text sample.

[0106] Wherein, the first classification correct probability is the first classification correct probability of the first text sample being classified as the first text sample's sample label.

[0107] In this embodiment of the application, the first text feature h i[CLS] Input a preset classification model to obtain the probability that the first text sample is classified as a first text type and the probability that the first text sample is classified as a second text type. If the sample label of the first text sample is the first text type, then the probability that the first text sample is classified as the first text type is determined as the first correct classification probability of the first text sample; if the sample label of the first text sample is the second text type, then the probability that the first text sample is classified as the second text type is determined as the first correct classification probability of the first text sample.

[0108] like Figure 3 As shown, the preset classification model can be a binary classifier f. c For example, the preset classification model is CNN, etc. The first text feature h... i[CLS] Feed into binary classifier f c We obtain the i-th first text sample x. i The correct probability p of the first classification i .

[0109] Step 105: Determine the classification loss based on the sample label of the first text sample and the first classification correct probability of the first text sample.

[0110] In this embodiment, the classification loss is determined using the cross-entropy loss function based on the sample label of the first text sample and the first classification correct probability of the first text sample.

[0111] Specifically, classification loss The calculation method is shown in the following formula.

[0112]

[0113] in, For classification loss, y i Let p be the sample label of the i-th first text sample. i Let be the first classification correct probability of the first text sample, and N be the total number of the first text samples, where N is the number of certificates greater than 1.

[0114] Step 106: Determine the contrast loss based on the first text features, the sample labels of the first text samples, the first text feature queue, and the second text feature queue.

[0115] In this embodiment of the application, comparative loss is used. The calculation method is shown in the following formula.

[0116]

[0117] Where N is the total number of the first text samples, σ represents the cosine similarity calculation, τ represents the temperature parameter, τ is a pre-set parameter, and p + Represents the first text feature queue q r Second text feature queue q f The queue with the same sample label as the first text sample, p - Represents the first text feature queue q r Second text feature queue q f The queue with the opposite sample labels to the first text sample, the z-th q This is the first text feature.

[0118] Step 107: Determine the total loss based on classification loss and contrast loss.

[0119] In one specific embodiment, the classification loss and the contrast loss are weighted and summed based on the weight coefficients of the classification loss and the contrast loss to obtain the total loss.

[0120] Specifically, the formula for calculating the total loss is as follows:

[0121]

[0122] in, For the total loss, It is classification loss. It is a contrastive loss, λ ce λ is the weighting coefficient of the classification loss. scl These are the weighting coefficients for the comparative loss.

[0123] In another specific embodiment, the total loss is determined based on the classification loss and the contrastive loss, including:

[0124] (1) Obtain multiple second sample prototypes with the same sample labels as the first text sample.

[0125] Among them, the sample labels of multiple second sample prototypes are the same as those of the first text sample.

[0126] In this embodiment of the application, the dimension of the second sample prototype is a preset number of dimensions.

[0127] In this embodiment of the application, the target prototype set is obtained. Where W represents the number of sample prototypes in each class, C represents the number of channels in the sample prototype vector, i = 0 represents the sample prototype of the first text type, i = 1 represents the sample prototype of the second text type, and j represents the index of the sample prototype in the corresponding type. Multiple second sample prototypes with the same sample label as the first text sample are obtained from the target prototype set H.

[0128] For example, i=0 represents the "human text" prototype, and i=1 represents the "AI text" prototype.

[0129] (2) Calculate the second similarity between multiple second sample prototypes and the first text sample respectively.

[0130] In this embodiment, the second similarity can be Euclidean distance. Of course, in other embodiments, the second similarity can be cosine similarity.

[0131] In a specific embodiment, such as Figure 3 As shown, the first text feature h of the first text sample is obtained. i[CLS] The first text feature h i[CLS] Input first projection layer mlp q The first projection feature z of the first text sample is obtained. q Calculate the first projected feature z. q Second sample prototype m ij The second similarity is used to determine the first projected feature z. q Second sample prototype m ij The second similarity is determined as the second similarity between the second sample prototype and the first text sample.

[0132] Specifically, the second similarity d(z) between the second sample prototype and the first text sample q ,m ij The calculation formula for ) is as follows:

[0133]

[0134] Among them, z q For the first projection feature, m ij This is the second prototype sample.

[0135] (3) Determine the probability that the sample label of the first text sample is the same as the sample label of the multiple second sample prototypes based on multiple second similarities.

[0136] In this embodiment of the application, the attribution probability p(x∈m) ij The formula for calculating |x) is shown below.

[0137]

[0138] Where, m ij p(x∈m) represents the j-th second sample prototype with sample label i. ij |x) represents the sample label of the first text sample and the prototype m of the second sample. ij The probability of belonging to the same sample label, d(z) q ,m ij ) is the second sample prototype m ij The second similarity with the first text sample.

[0139] (4) The probability of the second classification of the first text sample is determined based on the probability that the sample label of the first text sample is the same as the sample label of multiple second sample prototypes.

[0140] In this embodiment of the application, the formula for calculating the correct probability of the second classification of the first text sample is as follows:

[0141]

[0142] Where p(y|x) is the second classification correct probability that the first text sample is classified as the sample label of the first text sample, p(x∈m) ij |x) represents the sample label of the first text sample and the prototype m of the second sample. ij The probability of belonging to the same sample label, d(z) q ,m ij ) is the second sample prototype m ij The second similarity with the first text sample.

[0143] (5) Determine the prototype update loss based on the second classification correct probability of the first text sample.

[0144] In this embodiment of the application, the prototype update loss The calculation formula is as follows:

[0145]

[0146] in, Let p(y|x) be the prototype update loss, where p(y|x) is the second classification correct probability of the first text sample being classified as the first text sample label, and N is the total number of the first text samples.

[0147] (6) Determine the total loss based on prototype update loss, classification loss and comparison loss.

[0148] In this embodiment of the application, the total loss is determined based on the prototype update loss, classification loss, and comparison loss, including: weighting and summing the classification loss, prototype update loss, and comparison loss based on the weight coefficients of the classification loss, the prototype update loss, and the comparison loss to obtain the total loss.

[0149] Specifically, the formula for calculating the total loss is as follows:

[0150]

[0151] in, For the total loss, It is classification loss. It is a prototype update loss. It is a contrastive loss, λ ce λ is the weighting coefficient of the classification loss. pro These are the weighting coefficients of the prototype update loss, λ. scl These are the weighting coefficients for the comparative loss.

[0152] Step 108: Iteratively update the model parameters of the first language model based on the total loss until the preset stopping condition is met, and obtain the target language model.

[0153] In one specific embodiment, the preset stopping condition is that the total loss is not greater than a preset loss value. It is determined whether the total loss is greater than the preset loss value. If the total loss is greater than the preset loss value, the model parameters of the first language model are updated to obtain a new total loss. If the new total loss is not greater than the preset loss value, the preset stopping condition is determined to be met, and the first language model at this point is identified as the target language model.

[0154] In another specific embodiment, the preset stopping condition is that the total loss is not greater than a preset loss value or the number of iterations is greater than a preset number of iterations. It is determined whether the total loss is greater than the preset loss value and whether the number of iterations is greater than the preset number of iterations. If the total loss is greater than the preset loss value and the number of iterations is not greater than the preset number of iterations, the model parameters of the first language model are updated, and the number of iterations is also updated, resulting in a new total loss and a new number of iterations. When the new total loss is not greater than the preset loss value or the new number of iterations is greater than the preset number of iterations, the preset stopping condition is determined to be met, and the first language model at this point is identified as the target language model.

[0155] Furthermore, in order to dynamically update the first and second text feature queues more quickly, the language model training method in this embodiment further includes:

[0156] (1) Obtain the second language model.

[0157] In this embodiment, the second language model can be a RoBERTa model structure and initialized with pre-trained parameters. Let the second language model be the pre-trained language model f. k .

[0158] In one specific embodiment, the first language model and the second language model have the same network structure but different network parameters. The network parameters of the first language model and the second language model are updated synchronously and dynamically. Of course, in other embodiments, the second language model may not be updated, depending on the specific circumstances.

[0159] In this embodiment of the application, obtaining the second language model includes:

[0160] First, obtain the model parameters of the first language model during the current batch training and the model parameters of the second language model during the previous batch training.

[0161] Then, the current model parameters of the second language model are determined based on the model parameters of the first language model during the current batch training, the model parameters of the second language model during the previous batch training, and the preset momentum coefficient.

[0162] In one specific embodiment, the second language model f is updated. k The network parameters are given. Let f be the first language model. q The network parameter is θ q Second language model f k The network parameters are θ k Momentum updates are used, calculated as follows:

[0163] θ k ←∈θ k +(1-∈)θ q

[0164] Where ∈ represents the momentum coefficient.

[0165] In another specific embodiment, the relationship between the network parameters of the first language model and the second language model satisfies the following formula:

[0166] θ k =∈θ k-1 +(1-∈)ɑ k

[0167] Where, θ k For the second language model f during the kth batch of training k Network parameters, θ k-1 For the (k-1)th batch of training, the second language model f k The network parameters are ∈, where ∈ is the preset momentum coefficient, and α k For the first language model f during the kth batch of training q The network parameters. When k=1, θ k-1 =0.

[0168] (2) Based on the second language model, feature extraction is performed on the second text sample to obtain the second text features.

[0169] The second text sample is the sample in the target sample set after removing the first text sample.

[0170] In one specific embodiment, the second text sample Inputting the second language model for feature extraction yields the second text features.

[0171] (3) Calculate the first similarity between the second text features and multiple first sample prototypes respectively.

[0172] In this process, multiple first sample prototypes have the same sample labels as the second text features. Multiple first sample prototypes with the same sample labels as the second text samples are obtained from the target prototype set H.

[0173] In one specific embodiment, the second text feature Input the second projection layer mlp k The second text sample was obtained. The second projection feature z k Calculate the second projection feature z. k and the first sample prototype m ij The first similarity is used to determine the second projected feature z. k and the first sample prototype m ij The first similarity is determined as the first similarity between the first sample prototype and the second text sample.

[0174] The first similarity can be the cosine similarity.

[0175] (4) The second text features with a first similarity less than the preset similarity are identified as difficult sample features.

[0176] The preset similarity η can be set according to the specific situation.

[0177] like Figure 3 As shown, the second projection feature z k and the first sample prototype m ij First similarity is used for hard sample mining to update human text feature queue q r AI text feature queue q f .

[0178] (5) If the sample label of the difficult sample feature is the first text type, then the difficult sample is placed in the first text feature queue.

[0179] (6) If the sample label of the difficult sample feature is the second text type, then the difficult sample is placed in the second text feature queue.

[0180] Reference Figure 4 , Figure 4 This is a flowchart illustrating a text classification method provided in an embodiment of this application. The method can be executed by a text classification device. The method includes steps 301-303:

[0181] Step 301: Obtain the text to be classified.

[0182] Step 302: Input the text to be classified into the target language model to obtain the target text features.

[0183] The target language model can be any of the target language models mentioned above.

[0184] Step 303: Classify the text based on the features of the target text to obtain the text type of the text to be classified.

[0185] See Figure 5 ,like Figure 5 The diagram shows a structural schematic of a language model training device. The language model training device in this embodiment can achieve the above-described... Figure 2 The steps of the language model training method executed in the corresponding embodiments are described below. The functions implemented by the language model training device can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware.

[0186] The language model training device 50 includes:

[0187] The first acquisition module 501 is configured to acquire a first text feature queue and a second text feature queue, wherein the sample label of each text feature in the first text feature queue is a first text type, and the sample label of each text feature in the second text feature queue is a second text type.

[0188] The second acquisition module 502 is configured to acquire the first language model, the first text sample, and the corresponding sample label;

[0189] Feature extraction module 503 is configured to extract features from the first text sample based on the first language model to obtain the first text features;

[0190] The classification module 504 is configured to classify the first text sample based on the first text features to obtain the first classification correct probability of the first text sample;

[0191] The first determining module 505 is configured to determine the classification loss based on the sample label of the first text sample and the first classification correct probability of the first text sample;

[0192] The second determining module 506 is configured to determine the contrast loss based on the first text features, the sample labels of the first text samples, the first text feature queue, and the second text feature queue.

[0193] The third determination module 507 is configured to determine the total loss based on classification loss and contrast loss;

[0194] The iterative update module 508 is configured to iteratively update based on the total loss.

[0195] See Figure 6 ,like Figure 6 The diagram shows a structural schematic of a text classification device. The text classification device in this embodiment can achieve the functions described above. Figure 4 The steps of the text classification method executed in the corresponding embodiments are described below. The functions implemented by the language model training device can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware.

[0196] The text classification device 60 includes:

[0197] The acquisition module 601 is configured to acquire text to be classified.

[0198] The input module 602 is configured to input the text to be classified into the target language model to obtain the target text features, wherein the target language model is any of the above-mentioned target language models.

[0199] The classification module 603 is configured to classify based on the features of the target text to obtain the text type of the text to be classified.

[0200] The language model training device 50 in this application embodiment has been described above from the perspective of modular functional entities. The language model training device in this application embodiment will be described below from the perspective of hardware processing.

[0201] Figure 6 The devices shown can all have the following characteristics: Figure 7 The structure shown, when Figure 6 The training device 60 for the language model shown has, for example, Figure 7 When the structure shown is used, Figure 7 The processor and transceiver in the device can perform the same or similar functions as the modules provided in the aforementioned embodiments of the corresponding device. Figure 7 The memory storage processor in the memory needs to call computer programs when executing the training method of the above language model.

[0202] This application also provides a terminal device, such as... Figure 8As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. The terminal device can be any terminal device including mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, etc. Taking a mobile phone as an example:

[0203] Figure 8 This diagram illustrates a partial structural representation of a mobile phone related to the terminal device provided in this embodiment. (Reference) Figure 8 The mobile phone includes components such as a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art will understand that... Figure 8 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0204] The following is combined with Figure 8 A detailed introduction to each component of a mobile phone:

[0205] The RF circuit 1010 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with the processor 1080; additionally, it transmits uplink data to the base station. Typically, the RF circuit 1010 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the RF circuit 1010 can also communicate wirelessly with networks and other devices. The aforementioned wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).

[0206] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 1020. The memory 1020 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0207] The input unit 1030 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 1031), and drive the corresponding connection devices according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 1080, and can also receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1031, the input unit 1030 may also include other input devices 1032. Specifically, other input devices 1032 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0208] The display unit 1040 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1040 may include a display panel 1041, which may optionally be configured as a liquid crystal display (LCD), organic light-emitting diode (OLED), or similar display panel 1041. Further, a touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it transmits the information to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 8 In this embodiment, the touch panel 1031 and the display panel 1041 are two separate components to realize the input and output functions of the mobile phone. However, in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0209] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 1041 according to the ambient light level, and the proximity sensor can turn off the display panel 1041 and / or the backlight when the phone is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, taps), etc. Other sensors that may be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0210] The audio circuit 1060, speaker 1061, and microphone 1062 provide an audio interface between the user and the mobile phone. The audio circuit 1060 converts the received audio data into electrical signals and transmits them to the speaker 1061, where the speaker 1061 converts them into sound signals for output. On the other hand, the microphone 1062 converts the collected sound signals into electrical signals, which are then received by the audio circuit 1060, converted into audio data, and then processed by the processor 1080 before being transmitted via the RF circuit 1010 to, for example, another mobile phone, or the audio data can be output to the memory 1020 for further processing.

[0211] Wi-Fi is a short-range wireless transmission technology. Through the Wi-Fi module 1070, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 8 The Wi-Fi module 1070 is shown, but it is understood that it is not an essential component of a mobile phone and can be omitted as needed without changing the essence of the invention.

[0212] The processor 1080 is the control center of the mobile phone, connecting various parts of the phone through various interfaces and lines. It executes software programs and / or modules stored in the memory 1020 and calls data stored in the memory 1020 to perform various functions and process data, thereby providing overall monitoring of the phone. Optionally, the processor 1080 may include one or more processing units; optionally, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the aforementioned modem processor may also not be integrated into the processor 1080.

[0213] The mobile phone also includes a power supply 1090 (such as a battery) that supplies power to various components. Optionally, the power supply can be logically connected to the processor 1080 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0214] Although not shown, mobile phones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0215] In this embodiment of the application, the processor 1080 included in the mobile phone also has a method flow for controlling and executing the language model training method executed by the language model training device.

[0216] This application also provides a server; please refer to [link / reference]. Figure 9 , Figure 9 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1100 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1122 (e.g., one or more processors) and memory 1132, and one or more storage media 1130 (e.g., one or more mass storage devices) for storing application programs 1142 or data 1144. The memory 1132 and storage media 1130 may be temporary or persistent storage. The program stored in the storage media 1130 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the CPU 1122 may be configured to communicate with the storage media 1130 and execute the series of instruction operations in the storage media 1130 on the server 1100.

[0217] Server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input / output interfaces 1158, and / or one or more operating systems 1141, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0218] The steps performed by the server in the above embodiments can be based on this Figure 9 The structure of server 1100 shown. For example, as in the above embodiment, by Figure 9 The steps performed by the training device 60 of the language model shown can be based on this Figure 9 The server architecture is shown. For example, the central processing unit 1122 performs the following operations by calling instructions from memory 1132:

[0219] Obtain a first text feature queue and a second text feature queue. The sample label of each text feature in the first text feature queue is the first text type, and the sample label of each text feature in the second text feature queue is the second text type. Obtain a first language model, a first text sample, and the corresponding sample label. Extract features from the first text sample based on the first language model to obtain the first text features. Classify the first text sample based on the first text features to obtain the first classification correct probability of the first text sample. Determine the classification loss based on the sample label of the first text sample and the first classification correct probability of the first text sample. Determine the contrast loss based on the first text features, the sample label of the first text sample, the first text feature queue, and the second text feature queue. Determine the total loss based on the classification loss and the contrast loss. Iteratively update the model parameters of the first language model based on the total loss until a preset stopping condition is met to obtain the target language model.

[0220] Alternatively, obtain the text to be classified; input the text to be classified into the target language model to obtain the target text features; classify the text based on the target text features to obtain the text type of the text to be classified.

[0221] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0222] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0223] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or modules, and may be electrical, mechanical, or other forms.

[0224] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0225] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0226] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations described above.

[0227] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0228] A computer program product includes one or more computer instructions. When a computer program is loaded and executed on a computer, it produces, in whole or in part, the flow or function according to the embodiments of this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0229] The technical solutions provided in the embodiments of this application have been described in detail above. Specific examples have been used in the embodiments of this application to illustrate the principles and implementation methods of the embodiments of this application. The description of the above embodiments is only for the purpose of helping to understand the methods and core ideas of the embodiments of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the embodiments of this application. Therefore, the content of this specification should not be construed as a limitation on the embodiments of this application.

Claims

1. A method for training a language model, characterized in that, The training methods for the language model include: Obtain a first text feature queue and a second text feature queue, wherein the sample label of each text feature in the first text feature queue is a first text type, and the sample label of each text feature in the second text feature queue is a second text type; Obtain the first language model, the first text sample, and the corresponding sample labels; Based on the first language model, feature extraction is performed on the first text sample to obtain the first text features; Based on the first text features, the first text sample is classified to obtain the first classification correct probability of the first text sample; The classification loss is determined based on the sample label of the first text sample and the first classification correct probability of the first text sample; A contrast loss is determined based on the first text feature, the sample label of the first text sample, the first text feature queue, and the second text feature queue. Specifically, a second language model is obtained; features are extracted from the second text sample based on the second language model to obtain the second text feature, wherein the second text sample is the sample in the target sample set after removing the first text sample; a first similarity is calculated between the second text feature and multiple first sample prototypes, wherein the sample labels of the multiple first sample prototypes are the same as the sample labels of the second text feature; second text features with a first similarity less than a preset similarity are identified as difficult sample features; if the sample label of the difficult sample feature is a first text type, the difficult sample is placed in the first text feature queue; if the sample label of the difficult sample feature is a second text type, the difficult sample is placed in the second text feature queue. The total loss is determined based on the classification loss and the contrast loss, wherein: multiple second sample prototypes with the same sample labels as the first text sample are obtained; a second similarity between the multiple second sample prototypes and the first text sample is calculated respectively; the probability of the first text sample having the same sample label as the multiple second sample prototypes is determined based on the multiple second similarities; a second classification correct probability of the first text sample is determined based on the probability of the first text sample having the same sample label as the multiple second sample prototypes; a prototype update loss is determined based on the second classification correct probability of the first text sample; and the total loss is determined based on the prototype update loss, the classification loss, and the contrast loss. The model parameters of the first language model are iteratively updated based on the total loss until a preset stopping condition is met, thereby obtaining the target language model.

2. The method for training a language model according to claim 1, characterized in that, The process of obtaining the first language model, the first text sample, and the corresponding sample labels includes: Obtain the initial text sample and its corresponding sample label; The initial text sample is augmented to obtain the target augmented text sample; The initial text sample and the target augmented text sample are placed into the target sample set; A portion of the samples in the target sample set are identified as the first text sample.

3. The language model training method according to claim 2, characterized in that, The step of data augmentation of the initial text sample to obtain the target augmented text sample includes: The initial text sample is split into multiple text characters; the multiple text characters in the initial text sample are subjected to character-level perturbation a target preset number of times to obtain the target augmented text sample, wherein the character-level perturbation includes random character insertion, character deletion, character replacement, or character swapping. Alternatively, the initial text sample can be split into multiple text words; the multiple text words in the initial text sample can be subjected to word-level perturbation a target preset number of times to obtain the target augmented text sample, wherein the word-level perturbation is synonym replacement, word insertion, word deletion or random word swapping.

4. The language model training method according to claim 2, characterized in that, The acquisition of the second language model includes: Obtain the model parameters of the first language model during the current batch training and the model parameters of the second language model during the previous batch training; The current model parameters of the second language model are determined based on the model parameters of the first language model during the current batch training, the model parameters of the second language model during the previous batch training, and the preset momentum coefficient.

5. A text classification method, characterized in that, The text classification method includes: Get the text to be categorized; The text to be classified is input into the target language model to obtain the target text features, wherein the target language model is the target language model according to any one of claims 1-4; Based on the target text features, the text type of the text to be classified is obtained.

6. A training device for a language model, characterized in that, The training apparatus for this language model includes: The first acquisition module is configured to acquire a first text feature queue and a second text feature queue, wherein the sample label of each text feature in the first text feature queue is a first text type, and the sample label of each text feature in the second text feature queue is a second text type. The second acquisition module is configured to acquire the first language model, the first text sample, and the corresponding sample labels; The feature extraction module is configured to extract features from the first text sample based on the first language model to obtain the first text features; The classification module is configured to classify the first text sample based on the first text features to obtain the first classification correct probability of the first text sample; The first determining module is configured to determine the classification loss based on the sample label of the first text sample and the first classification correct probability of the first text sample; The second determining module is configured to determine a contrast loss based on the first text features, the sample labels of the first text samples, the first text feature queue, and the second text feature queue. This includes: acquiring a second language model; extracting features from the second text samples based on the second language model to obtain second text features, wherein the second text samples are samples from the target sample set after removing the first text samples; calculating the first similarity between the second text features and multiple first sample prototypes, wherein the sample labels of the multiple first sample prototypes are the same as the sample labels of the second text features; determining second text features with a first similarity less than a preset similarity as difficult sample features; if the sample label of the difficult sample feature is a first text type, then placing the difficult sample into the first text feature queue; if the sample label of the difficult sample feature is a second text type, then placing the difficult sample into the second text feature queue. The third determining module is configured to determine the total loss based on the classification loss and the contrast loss, wherein: multiple second sample prototypes with the same sample labels as the first text sample are obtained; a second similarity between the multiple second sample prototypes and the first text sample is calculated respectively; the probability of the first text sample having the same sample label as the sample labels of the multiple second sample prototypes is determined based on the multiple second similarities; a second classification correct probability of the first text sample is determined based on the probability of the first text sample having the same sample label as the sample labels of the multiple second sample prototypes; a prototype update loss is determined based on the second classification correct probability of the first text sample; and the total loss is determined based on the prototype update loss, the classification loss, and the contrast loss. The iterative update module is configured to iteratively update the model parameters of the first language model based on the total loss until a preset stopping condition is met, thereby obtaining the target language model.

7. A text classification device, characterized in that, The text classification device includes: The acquisition module is configured to acquire the text to be categorized. The input module is configured to input the text to be classified into a target language model to obtain target text features, wherein the target language model is the target language model according to any one of claims 1-4; The classification module is configured to classify the text based on the features of the target text to obtain the text type of the text to be classified.

8. A computing device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any one of claims 1-4 or the method as described in any one of claims 5.

Images