Text classification model training method, device, electronic device and storage medium

Abstract

The present invention relates to the fields of digital medicine and artificial intelligence technology, employing deep learning technology, and relates to a training method, device, electronic device, and storage medium for a text classification model. The method comprises: receiving a classification model training instruction; inputting training text into an encoder to generate a text feature vector; utilizing a linear layer to obtain a preset number of text labels and calculate a probability distribution of predicted labels; randomly initializing the text label vectors, inputting the obtained random label feature vectors into two self-attention layers and a linear layer, utilizing an algorithm to obtain a confused label distribution, combining the confused label distribution with the true label distribution to obtain a simulated label distribution; calculating the loss value between the simulated label distribution and the probability distribution of predicted labels according to a loss function, adjusting the parameters of the text classification model according to the loss value, and obtaining a trained text classification model. In the medical field, the present invention can improve the accuracy of medical text classification.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, applies deep learning technology, and relates to a training method, device, electronic device and storage medium for a text classification model. Background Art

[0002] Currently in the medical field, traditional medical text classification models typically use deep networks (DNNs, such as LSTM, CNN, BERT, etc.) to obtain vector representations, use linear layers to map dimensions to the number of labels, and then output the predicted label probability distribution through the SoftMax algorithm. Cross-entropy is then used to calculate the loss between the true label (one-hot representation) and p to perform classification optimization. However, the above method severs the relationship between labels and ignores the relationship between the true label and other labels, resulting in the classification model being unable to learn the information between labels and poor generalization ability. When faced with easily confused and noisy classification data, the model is more affected and the classification performance is poor. Summary of the Invention

[0003] In view of the above, it is necessary to provide a training method for text classification models, whose main purpose is to learn the relationship between the true label and other labels.

[0004] To achieve the above object, the present invention provides a text classification model training method, comprising:

[0005] S1. Receive a classification model training instruction, and obtain training data from a database according to the classification model training instruction, wherein the training data includes a training text and a true label distribution corresponding to the training text;

[0006] S2. Inputting the training text into the encoder, extracting text information features of the training text to generate a text feature vector, and processing the text feature vector using the first linear layer to obtain a preset number of text labels and a probability distribution of predicted labels;

[0007] S3, calculating the vectors of the preset number of text labels and performing random initialization processing to obtain multiple random label feature vectors, inputting the obtained multiple random label feature vectors into the first self-attention layer, and outputting multiple first correlation feature vectors;

[0008] S4, inputting the multiple first association feature vectors and the text feature vector into the second self-attention layer, and outputting a second label feature vector;

[0009] S5. Perform dimension mapping on the second label feature vector using the second linear layer to obtain a mapped label feature vector, calculate the mapped label feature vector using the SoftMax algorithm to obtain a confused label distribution, and combine the confused label distribution with the true label distribution to obtain a simulated label distribution.

[0010] S6. Calculate the loss value of the simulated label distribution and the probability distribution of the predicted label according to the loss function. If the loss value is greater than or equal to a preset loss threshold, update the parameters of the text classification model and return to step S2. If the loss value is less than the preset loss threshold, obtain the trained text classification model, and return the trained text classification model to the terminal of the classification model training instruction for the terminal to perform text classification.

[0011] Optionally, extracting text information features of the training text to generate a text feature vector includes:

[0012] The formula for extracting the text information features of the training text to generate a text feature vector is:

[0013] Hw=F(x)=F(x1,x2,...xn)

[0014] Among them, Hw is the text feature vector, F(x) is the classification function of the encoder, and x1 to xn refer to multiple text input information of the model.

[0015] Optionally, processing the text feature vector using the first linear layer includes:

[0016] Use the first linear layer to perform dimension mapping on the text feature vector to obtain a preset number of text labels;

[0017] The probability distribution of predicted labels is obtained by calculating the preset number of text labels using a SoftMax algorithm, wherein the predicted labels are used to represent the categories of the training texts.

[0018] Optionally, the formula for calculating the probability distribution of the predicted labels by calculating the preset number of text labels using the SoftMax algorithm is:

[0019] Yp=softmax(H)

[0020] Among them, Yp is the probability distribution of the predicted label, softmax is the softmax function used to calculate the probability distribution of the label, and H is the preset number of text labels.

[0021] Optionally, the multiple first associated feature vectors and the text feature vector are input into the second self-attention layer, and a second label feature vector is output, which is formulated as follows:

[0022] Hc2=Attention(Hc1)

[0023] Among them, Hc2 is the second label feature vector, Attention is the second self-attention layer function used to calculate the label feature vector, and Hc1 is the first association feature vector.

[0024] Optionally, combining the obfuscated label distribution with the true label distribution to obtain a simulated label distribution includes:

[0025] The value of the confused label distribution is added to the value of the true label distribution, and the calculated value is normalized to obtain a simulated label distribution.

[0026] Optionally, the loss function is a KL divergence algorithm, the input of the loss function is the probability distribution of the predicted label and the sum of the confusion label distribution and the true label distribution, and the output is the loss value.

[0027] In addition, to achieve the above-mentioned purpose, the present invention also provides a text classification model training device, which includes:

[0028] A data receiving module is used to receive a classification model training instruction and obtain training data from a database according to the classification model training instruction;

[0029] a text feature extraction module, configured to input the training text into the encoder, extract text information features of the training text to generate a text feature vector, and process the text feature vector using the first linear layer to obtain a preset number of text labels and a probability distribution of predicted labels;

[0030] A first self-attention layer module is configured to calculate vectors of the preset number of text labels and perform random initialization processing, input the obtained multiple random label feature vectors into the first self-attention layer, and output multiple first correlation feature vectors;

[0031] A second self-attention layer module is configured to input the plurality of first associated feature vectors and the text feature vector into the second self-attention layer, and output a second label feature vector;

[0032] a simulated label distribution module, configured to perform dimension mapping on the second label feature vector using the second linear layer to obtain a mapped label feature vector, then calculate the mapped label feature vector using a SoftMax algorithm to obtain a confused label distribution, and combine the confused label distribution with the true label distribution to obtain a simulated label distribution;

[0033] The loss value calculation module is used to calculate the loss value of the probability distribution of the simulated label distribution and the predicted label according to the loss function. If the loss value is greater than or equal to the preset loss threshold, the parameters of the text classification model are updated and the method returns to step S2. If the loss value is less than the preset loss threshold, the trained text classification model is obtained, and the trained text classification model is returned to the terminal of the classification model training instruction for the terminal to perform text classification.

[0034] In addition, to achieve the above-mentioned object, the present invention further provides an electronic device, comprising:

[0035] a memory storing at least one computer program; and

[0036] The processor executes the program stored in the memory to implement the above-mentioned training method of the text classification model.

[0037] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned text classification model training method.

[0038] The embodiment of the present invention extracts text feature vectors from training text through an encoder, then uses a first linear layer to perform dimension mapping on the text feature vectors to obtain a preset number of text labels, and obtains a probability distribution of predicted labels. The vectors of the preset number of text labels are randomly initialized to obtain multiple random label feature vectors, and then these multiple random label feature vectors are gradually constructed into a label distribution by similarity with the input text vector. Because such a label distribution can reflect the relationship between labels and samples, and between labels, and contains more useful information than one-hot, it can bring performance improvements to the classification model. It does not rely on any external knowledge and fully uses the information used in the training model. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart of a method for training a text classification model provided by one embodiment of the present invention;

[0040] Figure 2 A schematic diagram of a flow chart of a text classification model training apparatus provided in one embodiment of the present invention;

[0041] Figure 3 A schematic diagram of the internal structure of an electronic device for implementing a text classification model training method provided by one embodiment of the present invention;

[0042] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0044] It should be noted that the descriptions of "first", "second", etc. in the present invention are for descriptive purposes only and should not be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0045] Reference Figure 1 FIG. 1 is a flow chart of a method for training a text classification model according to an embodiment of the present invention. In the embodiment of the present invention, the method for training a text classification model includes the following steps S1-S6:

[0046] S1. Receive a classification model training instruction, and obtain training data from a database according to the classification model training instruction, wherein the training data includes a training text and a true label distribution corresponding to the training text;

[0047] In an embodiment of the present invention, the training instruction has an identifier for marking the corresponding training data, and the training data is extracted according to the identifier; the training data includes training text and true label distribution, and the training text is a medical text with a certain label, for example, the medical text is a case report with a chronic bronchitis label, a case report with a favism label, and a case report with a diabetes label. The embodiment of the present invention does not limit the method for obtaining the training text, that is, any acquisition method can be used to obtain sample text. The true label distribution represents the correct label distribution for the training text, and the text classification model is used to classify the text, that is, the category to which the text belongs can be obtained by inputting the text into the trained text classification model.

[0048] The text classification model includes but is not limited to: an encoder, a first linear layer, a second linear layer, a first self-attention layer, a second self-attention layer, and a loss function.

[0049] S2. Inputting the training text into the encoder, extracting text information features of the training text to generate a text feature vector, and processing the text feature vector using the first linear layer to obtain a preset number of text labels and a probability distribution of predicted labels;

[0050] In an embodiment of the present invention, the text information features of the training text are extracted to generate a text feature vector, and the formula is Hw=F(x)=F(x1, x2, ... xn), wherein the encoder is a classification network (DNN, such as LSTM, CNN, BERT, etc.), F(x) is the classification function of the encoder, and x1 to xn refer to multiple text input information of the model;

[0051] Specifically, the text information features are extracted using natural language technology. For example, keyword recognition is used to extract text information features from a chronic bronchitis case report to obtain the keyword "chronic bronchitis." The preset number of text tags are "chronic bronchitis," "favism," and "diabetes."

[0052] S3, calculating the vectors of the preset number of text labels and performing random initialization processing to obtain multiple random label feature vectors, inputting the obtained multiple random label feature vectors into the first self-attention layer, and outputting multiple first correlation feature vectors;

[0053] In an embodiment of the present invention, the random initialization method adopted is an initialization method that satisfies the normal distribution; the multiple random label feature vectors include Q, K, and V, and the Q, K, and V are passed through the first self-attention layer to obtain multiple first associated feature vectors Hc1, Hc1 = SelfAttention (Hl), and SelfAttention is the first self-attention layer.

[0054] For example, Q, K, and V are the label vectors of “chronic bronchitis”, “favism”, and “diabetes” respectively.

[0055] S4, inputting the multiple first association feature vectors and the text feature vector into the second self-attention layer, and outputting a second label feature vector;

[0056] In the embodiment of the present invention, the first correlation feature vector Hc1 is represented by K1 and V1, and the text feature vector, V1, and K1 are input into the second self-attention layer to obtain the second label feature vector Hc2. The formula is Hc2 = Attention(Hc1), where Attention is the second self-attention layer;

[0057] For example, V1 and K1 are the label vectors of diabetes cases and favism cases;

[0058] In an embodiment of the present invention, the second self-attention layer is a cross-attention layer that fuses the text feature vector and the second label feature vector.

[0059] S5. Perform dimension mapping on the second label feature vector using the second linear layer to obtain a mapped label feature vector, calculate the mapped label feature vector using the SoftMax algorithm to obtain a confused label distribution, and combine the confused label distribution with the true label distribution to obtain a simulated label distribution.

[0060] In the embodiment of the present invention, the vector obtained by the dimensional mapping of the linear layer is used to represent the score of each label. For example, the label score of the diabetes case is 0.8 points, and the label score of the favism case is 0.2 points.

[0061] In an embodiment of the present invention, combining the obfuscated label distribution with the true label distribution to obtain a simulated label distribution includes: adding the numerical value of the obfuscated label distribution to the numerical value of the true label distribution, and normalizing the calculated numerical value to obtain a final label distribution, i.e., the simulated label distribution.

[0062] S6. Calculate the loss value of the simulated label distribution and the probability distribution of the predicted label according to the loss function. If the loss value is greater than or equal to a preset loss threshold, update the parameters of the text classification model and return to step S2. If the loss value is less than the preset loss threshold, obtain the trained text classification model, and return the trained text classification model to the terminal of the classification model training instruction for the terminal to perform text classification.

[0063] In an embodiment of the present invention, the loss function is a KL divergence algorithm, the input of the loss function is the probability distribution of the predicted label and the sum of the confusion label distribution and the true label distribution, and the output is the loss value;

[0064] In an embodiment of the present invention, a loss threshold is preset. When the loss value is greater than or equal to the loss threshold, it means that the text classification model does not meet the requirements. Therefore, the parameters of the text classification model are changed and training is continued until the loss value is less than the loss threshold, thereby obtaining a trained text classification model.

[0065] The present invention extracts text feature vectors from training text through an encoder, then uses a first linear layer to perform dimension mapping on the text feature vectors to obtain a preset number of text labels, and obtains a probability distribution of predicted labels. After randomly initializing the vectors of the preset number of text labels, multiple random label feature vectors are obtained. These multiple random label feature vectors are then gradually constructed into a label distribution by similarity with the input text vector. Because such a label distribution can reflect the relationship between labels and samples, and between labels, and contains more useful information than one-hot, it can improve the performance of the classification model. It does not rely on any external knowledge and fully utilizes the information used in the training model.

[0066] like Figure 2 , which is a functional module diagram of the training device for the text classification model of the present invention.

[0067] The text classification model training device 100 of the present invention can be installed in an electronic device. Depending on the functions to be implemented, the text classification model training device can include a data receiving module 101, a text feature extraction module 102, a first self-attention layer module 103, a second self-attention layer module 104, a simulated label distribution module 105, and a loss value calculation module 106. The modules described herein can also be referred to as units, which refer to a series of computer program segments that can be executed by an electronic device processor and can perform fixed functions, and are stored in the memory of the electronic device.

[0068] In this embodiment, the functions of each module / unit are as follows:

[0069] The data receiving module 101 is used to receive a classification model training instruction and obtain training data from a database according to the classification model training instruction;

[0070] The text feature extraction module 102 is used to input the training text into the encoder, extract the text information features of the training text to generate a text feature vector, and use the first linear layer to process the text feature vector to obtain a preset number of text labels and a probability distribution of predicted labels;

[0071] The first self-attention layer module 103 is used to calculate the vectors of the preset number of text labels and perform random initialization processing, input the obtained multiple random label feature vectors into the first self-attention layer, and output multiple first correlation feature vectors;

[0072] The second self-attention layer module 104 is used to input the multiple first association feature vectors and the text feature vector into the second self-attention layer, and output a second label feature vector;

[0073] The simulated label distribution module 105 is used to perform dimension mapping on the second label feature vector through the second linear layer to obtain a mapped label feature vector, and then calculate the mapped label feature vector using the SoftMax algorithm to obtain a confused label distribution, and combine the confused label distribution with the true label distribution to obtain a simulated label distribution;

[0074] The loss value calculation module 106 is used to calculate the loss value of the simulated label distribution and the probability distribution of the predicted label according to the loss function. If the loss value is greater than or equal to the preset loss threshold, the parameters of the text classification model are updated and the process returns to step S2. If the loss value is less than the preset loss threshold, the trained text classification model is obtained, and the trained text classification model is returned to the terminal of the classification model training instruction for the terminal to perform text classification.

[0075] In detail, each module in the text classification model training device 100 according to the embodiment of the present invention adopts the same method as above when used. Figure 1 The training method of the text classification model described in

[15] is the same technical means and can produce the same technical effects, so I will not go into details here.

[0076] like Figure 3 , which is a structural diagram of an electronic device for implementing a training method for a text classification model according to the present invention.

[0077] The electronic device may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may further include a computer program stored in the memory 11 and executable on the processor 10, such as a training program for a text classification model.

[0078] Wherein, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 can also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Furthermore, the memory 11 can also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the training program of the text classification model, but can also be used to temporarily store data that has been output or is to be output.

[0079] In some embodiments, the processor 10 may be composed of an integrated circuit, such as a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines. It executes or executes programs or modules stored in the memory 11 (such as a training program for a text classification model) and calls data stored in the memory 11 to perform various functions of the electronic device and process data.

[0080] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. Such a bus may be classified as an address bus, a data bus, a control bus, etc. The communication bus 12 is configured to enable communication between the memory 11 and at least one processor 10. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0081] Figure 3 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 3 The structure shown does not limit the electronic device, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0082] For example, although not shown, the electronic device may further include a power source (such as a battery) for powering various components. Preferably, the power source may be logically connected to the at least one processor 10 via a power management device, thereby implementing functions such as charging management, discharging management, and power consumption management through the power management device. The power source may further include any components such as one or more DC or AC power sources, a recharging device, a power failure classification circuit, a power converter or inverter, a power status indicator, etc. The electronic device may further include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0083] Optionally, the communication interface 13 may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices.

[0084] Optionally, the communication interface 13 may further include a user interface, which may be a display or an input unit (such as a keyboard). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0085] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0086] The training program for the text classification model stored in the memory 11 of the electronic device is a combination of multiple computer programs. When running in the processor 10, it can achieve the following:

[0087] receiving a classification model training instruction, and acquiring training data from a database according to the classification model training instruction;

[0088] Inputting the training text into the encoder, extracting text information features of the training text to generate a text feature vector, and processing the text feature vector using the first linear layer to obtain a preset number of text labels and a probability distribution of predicted labels;

[0089] Calculating the vectors of the preset number of text labels and performing random initialization processing to obtain a plurality of random label feature vectors, inputting the obtained plurality of random label feature vectors into the first self-attention layer, and outputting a plurality of first correlation feature vectors;

[0090] Input the multiple first association feature vectors and the text feature vector into the second self-attention layer, and output a second label feature vector;

[0091] Performing dimension mapping on the second label feature vector using the second linear layer to obtain a mapped label feature vector, calculating the mapped label feature vector using a SoftMax algorithm to obtain a confused label distribution, and combining the confused label distribution with the true label distribution to obtain a simulated label distribution;

[0092] The loss value of the simulated label distribution and the probability distribution of the predicted label is calculated according to the loss function. If the loss value is greater than or equal to the preset loss threshold, the parameters of the text classification model are updated and the process returns to step S2. If the loss value is less than the preset loss threshold, the trained text classification model is obtained, and the trained text classification model is returned to the terminal of the classification model training instruction for the terminal to perform text classification.

[0093] Specifically, the specific implementation method of the processor 10 for the above computer program can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0094] Furthermore, if the module / unit integrated into the electronic device is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium can be non-volatile or volatile. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, or read-only memory (ROM).

[0095] An embodiment of the present invention may further provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor of an electronic device, the computer program may implement:

[0096] receiving a classification model training instruction, and acquiring training data from a database according to the classification model training instruction;

[0097] Inputting the training text into the encoder, extracting text information features of the training text to generate a text feature vector, and processing the text feature vector using the first linear layer to obtain a preset number of text labels and a probability distribution of predicted labels;

[0098] Calculating the vectors of the preset number of text labels and performing random initialization processing to obtain a plurality of random label feature vectors, inputting the obtained plurality of random label feature vectors into the first self-attention layer, and outputting a plurality of first correlation feature vectors;

[0099] Input the multiple first association feature vectors and the text feature vector into the second self-attention layer, and output a second label feature vector;

[0100] Performing dimension mapping on the second label feature vector using the second linear layer to obtain a mapped label feature vector, calculating the mapped label feature vector using a SoftMax algorithm to obtain a confused label distribution, and combining the confused label distribution with the true label distribution to obtain a simulated label distribution;

[0101] The loss value of the simulated label distribution and the probability distribution of the predicted label is calculated according to the loss function. If the loss value is greater than or equal to the preset loss threshold, the parameters of the text classification model are updated and the process returns to step S2. If the loss value is less than the preset loss threshold, the trained text classification model is obtained, and the trained text classification model is returned to the terminal of the classification model training instruction for the terminal to perform text classification.

[0102] Furthermore, the computer-usable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0103] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and other division methods may be used in actual implementation.

[0104] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0105] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0106] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0107] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0108] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0109] Blockchain, as used in this article, refers to a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each block contains information about a batch of online transactions, used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0110] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. Second-order terms are used to indicate names and do not imply any particular order.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A training method for a text classification model, the text classification model comprising an encoder, a first linear layer, a second linear layer, a first self-attention layer, a second self-attention layer, and a loss function, characterized in that: The method comprises: S1. Receive a classification model training instruction, and obtain training data from a database according to the classification model training instruction, wherein the training data includes a training text and a true label distribution corresponding to the training text; S2. Inputting the training text into the encoder, extracting text information features of the training text to generate a text feature vector, and processing the text feature vector using the first linear layer to obtain a preset number of text labels and a probability distribution of predicted labels; S3, calculating the vectors of the preset number of text labels and performing random initialization processing to obtain multiple random label feature vectors, inputting the obtained multiple random label feature vectors into the first self-attention layer, and outputting multiple first correlation feature vectors; S4, inputting the multiple first association feature vectors and the text feature vector into the second self-attention layer, and outputting a second label feature vector; S5. Perform dimension mapping on the second label feature vector using the second linear layer to obtain a mapped label feature vector, calculate the mapped label feature vector using the SoftMax algorithm to obtain a confused label distribution, and combine the confused label distribution with the true label distribution to obtain a simulated label distribution. S6. Calculate the loss value of the simulated label distribution and the probability distribution of the predicted label according to the loss function. If the loss value is greater than or equal to a preset loss threshold, update the parameters of the text classification model and return to step S2. If the loss value is less than the preset loss threshold, obtain the trained text classification model, and return the trained text classification model to the terminal of the classification model training instruction for the terminal to perform text classification.

2. The text classification model training method according to claim 1, wherein: The extracting text information features of the training text to generate a text feature vector includes: The formula for extracting the text information features of the training text to generate a text feature vector is: Hw=F(x)=F(x1,x2,...xn) Among them, Hw is the text feature vector, F(x) is the classification function of the encoder, and x1 to xn refer to multiple text input information of the model.

3. The text classification model training method according to claim 1, wherein: Processing the text feature vector using the first linear layer includes: Perform dimension mapping on the text feature vector using the first linear layer to obtain a preset number of text labels; The probability distribution of predicted labels is obtained by calculating the preset number of text labels using a SoftMax algorithm, wherein the predicted labels are used to represent the categories of the training texts.

4. The text classification model training method according to claim 3, wherein: The formula for calculating the probability distribution of the predicted labels by the SoftMax algorithm for the preset number of text labels is: Yp=softmax(H) Among them, Yp is the probability distribution of the predicted label, softmax is the softmax function used to calculate the probability distribution of the label, and H is the preset number of text labels.

5. The text classification model training method according to claim 1, wherein: The formula for inputting the multiple first associated feature vectors and the text feature vector into the second self-attention layer and outputting the second label feature vector is: Hc2=Attention(Hc1) Among them, Hc2 is the second label feature vector, Attention is the second self-attention layer function used to calculate the label feature vector, and Hc1 is the first association feature vector.

6. The text classification model training method according to claim 1, wherein: Combining the confusion label distribution with the true label distribution to obtain a simulated label distribution includes: The value of the confused label distribution is added to the value of the true label distribution, and the calculated value is normalized to obtain a simulated label distribution.

7. The text classification model training method according to claim 1, wherein: The loss function is a KL divergence algorithm, the input of which is the probability distribution of the predicted label and the sum of the confusion label distribution and the true label distribution, and the output is the loss value.

8. A text classification model training device, the text classification model comprising an encoder, a first linear layer, a second linear layer, a first self-attention layer, a second self-attention layer, and a loss function, characterized in that: The device comprises: A data receiving module is used to receive a classification model training instruction and obtain training data from a database according to the classification model training instruction, wherein the training data includes a training text and a true label distribution corresponding to the training text; a text feature extraction module, configured to input the training text into the encoder, extract text information features of the training text to generate a text feature vector, and process the text feature vector using the first linear layer to obtain a preset number of text labels and a probability distribution of predicted labels; A first self-attention layer module is configured to calculate vectors of the preset number of text labels and perform random initialization processing, input the obtained multiple random label feature vectors into the first self-attention layer, and output multiple first correlation feature vectors; A second self-attention layer module is configured to input the plurality of first associated feature vectors and the text feature vector into the second self-attention layer, and output a second label feature vector; a simulated label distribution module, configured to perform dimension mapping on the second label feature vector using the second linear layer to obtain a mapped label feature vector, then calculate the mapped label feature vector using a SoftMax algorithm to obtain a confused label distribution, and combine the confused label distribution with the true label distribution to obtain a simulated label distribution; The loss value calculation module is used to calculate the loss value of the probability distribution of the simulated label distribution and the predicted label according to the loss function. If the loss value is greater than or equal to the preset loss threshold, the parameters of the text classification model are updated and the text feature extraction module is returned to execute; if the loss value is less than the preset loss threshold, the trained text classification model is obtained, and the trained text classification model is returned to the terminal of the classification model training instruction for the terminal to perform text classification.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, the training method of the text classification model according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the text classification model training method according to any one of claims 1 to 7 is implemented.

Images