Text classification method, device, equipment and computer-readable storage medium

Abstract

The present invention discloses a text classification method, device, system and computer-readable storage medium, the method comprising: obtaining a text to be classified, inputting the text to be classified into a text classification model; wherein the text classification model at least comprises a pre-training model, and during the training process of the text classification model, the learning rate of each functional layer of the pre-training model decreases from the output layer to the input layer; and controlling the text classification model to process the text to be classified to obtain a text classification result of the text to be classified. The present invention enables the text classification model to learn the unique text semantic level and syntactic level information in a specific text classification task, and to give play to the universal effect of the pre-training model in mining semantic level information, syntactic level information and phrase information, thereby improving the accuracy of the text classification model in text classification, and making the result obtained by using the text classification model to classify the text to be classified more accurate.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing technology, and in particular to a text classification method, device, equipment and computer-readable storage medium. Background Art

[0002] A pre-trained model is a model created by predecessors to solve similar problems. When solving a problem, you do not need to train a new model from scratch, but can start with a model that has been trained in similar problems. Today, natural language processing technology can develop rapidly and be widely used, largely due to the use of pre-trained models in specific natural language processing applications. At present, in text classification processing, a text classification model trained based on a pre-trained model is also used. During the training of the text classification model, the learning rate used by each layer of the pre-trained model part of the text classification model is the same. However, since the learning rate of each layer of the pre-trained model part is the same, that is, the perturbation of the parameters of each layer by the training data during the training process is the same, the accuracy of text classification by the trained text classification model is not high, which makes the classification results obtained by the text classification model for text classification processing not accurate enough. Summary of the invention

[0003] The main purpose of the present invention is to provide a text classification method, device, system and computer-readable storage medium, aiming to solve the problem that the current method of training a text classification model based on a pre-trained model makes the trained text classification model have low accuracy, and thus the classification results obtained by performing text classification processing through the text classification model are not accurate enough.

[0004] To achieve the above object, the present invention provides a text classification method, which comprises the following steps:

[0005] Obtaining a text to be classified, and inputting the text to be classified into a text classification model;

[0006] The text classification model at least includes a pre-training model, and during the training process of the text classification model, the learning rate of each functional layer of the pre-training model decreases from the output layer to the input layer;

[0007] The text classification model is controlled to process the text to be classified to obtain a text classification result of the text to be classified.

[0008] Optionally, before the step of obtaining the text to be classified and inputting the text to be classified into the text classification model, the step further includes:

[0009] Obtaining the overall learning rate of the model to be trained, and determining the learning rate of each functional layer of the pre-trained model in the model to be trained according to the overall learning rate;

[0010] The model to be trained is trained according to the learning rate of each functional layer of the pre-trained model to obtain the text classification model.

[0011] Optionally, the step of determining the learning rate of each functional layer of the pre-trained model in the model to be trained according to the overall learning rate includes:

[0012] Using the overall learning rate as the learning rate of the functional layer of the model to be trained that is closest to the output layer of the pre-trained model;

[0013] Taking the learning rate of the functional layer closest to the output layer as a benchmark, with the output layer direction as the upward direction and the input layer direction as the downward direction, the learning rate of the previous functional layer corresponding to each functional layer is divided by a preset value, and the result is used as the learning rate of each functional layer, wherein the preset value is greater than one.

[0014] Optionally, before the step of determining the learning rate of each functional layer of the pre-trained model in the model to be trained according to the overall learning rate, the step further includes:

[0015] Performing functional analysis on each single layer of the pre-trained model in the model to be trained to determine the text analysis function corresponding to each single layer;

[0016] The single layers with the same text analysis function are divided into the same functional layer to obtain the functional layers of the pre-trained model.

[0017] Optionally, the step of training the model to be trained according to the learning rate of each functional layer of the pre-trained model to obtain the text classification model includes:

[0018] In each round of training, a preset training text set is input into the model to be trained, and the model to be trained is controlled to process the training text set to obtain a classification result corresponding to each text in the training text set;

[0019] Calculating the loss function of the model to be trained according to the classification labels corresponding to each text in the training text set and the classification results;

[0020] The gradient corresponding to each functional layer in the pre-trained model is calculated according to the loss function;

[0021] The gradient and learning rate of each functional layer in the pre-trained model are used to update each functional layer accordingly, so as to update the model to be trained;

[0022] After each round of training, the finally updated model to be trained is used as the text classification model.

[0023] Optionally, when the training round is less than a preset round, the overall learning rate increases as the training round increases; when the training round is greater than or equal to the preset round, the overall learning rate decreases as the training round increases.

[0024] Optionally, the step of obtaining the overall learning rate of the model to be trained includes:

[0025] For each training round of the model to be trained, each first round that is smaller than the preset round is substituted into a preset linear growth function to obtain a first learning rate corresponding to each first round;

[0026] Substitute each second round that is greater than or equal to the preset round into a preset exponential descent function to obtain a second learning rate corresponding to each second round, and use each first learning rate and each second learning rate as the overall learning rate of the model to be trained.

[0027] To achieve the above object, the present invention provides a text classification device, the device comprising:

[0028] An acquisition module is used to acquire the text to be classified and input the text to be classified into a text classification model; wherein the text classification model at least includes a pre-trained model, and during the training process of the text classification model, the learning rate of each functional layer of the pre-trained model decreases from the output layer to the input layer;

[0029] The processing module is used to control the text classification model to process the text to be classified to obtain a text classification result of the text to be classified.

[0030] To achieve the above-mentioned purpose, the present invention also provides a text classification device, which includes: a memory, a processor, and a text classification program stored in the memory and executable on the processor, wherein the text classification program implements the steps of the text classification method described above when executed by the processor.

[0031] In addition, to achieve the above-mentioned purpose, the present invention further proposes a computer-readable storage medium, on which a text classification program is stored, and when the text classification program is executed by a processor, the steps of the text classification method described above are implemented.

[0032] In the present invention, by obtaining the text to be classified, the text to be classified is input into a text classification model; wherein the text classification model at least includes a pre-training model, and during the training process of the text classification model, the learning rate of each functional layer of the pre-training model decreases from the output layer to the input layer; the text classification model is controlled to process the text to be classified to obtain a text classification result of the text to be classified. In the present invention, the pre-trained model is divided into a plurality of functional layers. According to the model characteristics of the pre-trained model, the functional layer close to the output layer is generally used to mine the semantic level information and the syntactic level information of the text, and the functional layer close to the input layer is generally used to mine the phrase level information of the text; in the text classification model training process, a larger learning rate is adopted in the functional layer close to the output layer of the pre-trained model, and the functional layer close to the output layer is changed more, so that the functional layer close to the output layer is learned more, thereby mining more text semantic level and syntactic level information in the specific text classification task in the training text set; and a smaller learning rate is adopted in the functional layer close to the input layer, and the functional layer close to the input layer is changed less, so that the functional layer close to the input layer will not be greatly disturbed by the training text set, so that the universality of the pre-trained model in phrase information mining can be more utilized, and the effect of the pre-trained model can be better exerted.

[0033] That is, in the present invention, by adopting a larger learning rate in the functional layer close to the output layer and a smaller learning rate in the functional layer close to the input layer, the trained text classification model can not only learn the unique text semantic level and syntactic level information in the specific text classification task, but also give play to the universal effect of the pre-trained model in mining semantic level information, syntactic level information and phrase information, thereby improving the accuracy of text classification by the text classification model, and making the results obtained by using the text classification model to classify the text to be classified more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention;

[0035] Figure 2 This is a flow chart of the first embodiment of the text classification method of the present invention;

[0036] Figure 3 A diagram showing a relationship between training rounds and learning rates involved in an embodiment of a text classification method of the present invention;

[0037] Figure 4 This is a functional schematic module diagram of a preferred embodiment of the text classification device of the present invention.

[0038] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0039] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0040] like Figure 1 As shown, Figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0041] It should be noted that the text classification device in the embodiment of the present invention can be a smart phone, a personal computer, a server and other devices, and no specific limitation is made here.

[0042] like Figure 1 As shown, the text classification device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0043] Those skilled in the art will understand that Figure 1 The device structure shown in the figure does not constitute a limitation on the text classification device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0044] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a text classification program. The operating system is a program that manages and controls the hardware and software resources of the device and supports the operation of the text classification program and other software or programs. Figure 1 In the device shown, the user interface 1003 is mainly used for data communication with the client; the network interface 1004 is mainly used for establishing a communication connection with the server; the processor 1001 can be used to call the text classification program stored in the memory 1005 and perform the following operations:

[0045] Obtaining a text to be classified, and inputting the text to be classified into a text classification model;

[0046] The text classification model at least includes a pre-training model, and during the training process of the text classification model, the learning rate of each functional layer of the pre-training model decreases from the output layer to the input layer;

[0047] The text classification model is controlled to process the text to be classified to obtain a text classification result of the text to be classified.

[0048] Furthermore, before the step of obtaining the text to be classified and inputting the text to be classified into the text classification model, the processor 1001 can be used to call the text classification program stored in the memory 1005, and further perform the following operations:

[0049] Obtaining the overall learning rate of the model to be trained, and determining the learning rate of each functional layer of the pre-trained model in the model to be trained according to the overall learning rate;

[0050] The model to be trained is trained according to the learning rate of each functional layer of the pre-trained model to obtain the text classification model.

[0051] Furthermore, the step of determining the learning rate of each functional layer of the pre-trained model in the model to be trained according to the overall learning rate includes:

[0052] Using the overall learning rate as the learning rate of the functional layer of the model to be trained that is closest to the output layer of the pre-trained model;

[0053] Taking the learning rate of the functional layer closest to the output layer as a benchmark, with the output layer direction as the upward direction and the input layer direction as the downward direction, the learning rate of the previous functional layer corresponding to each functional layer is divided by a preset value, and the result is used as the learning rate of each functional layer, wherein the preset value is greater than one.

[0054] Further, before the step of determining the learning rate of each functional layer of the pre-trained model in the model to be trained according to the overall learning rate, the processor 1001 can be used to call the text classification program stored in the memory 1005, and further perform the following operations:

[0055] Performing functional analysis on each single layer of the pre-trained model in the model to be trained to determine the text analysis function corresponding to each single layer;

[0056] The single layers with the same text analysis function are divided into the same functional layer to obtain the functional layers of the pre-trained model.

[0057] Furthermore, the step of training the model to be trained according to the learning rate of each functional layer of the pre-trained model to obtain the text classification model includes:

[0058] In each round of training, a preset training text set is input into the model to be trained, and the model to be trained is controlled to process the training text set to obtain a classification result corresponding to each text in the training text set;

[0059] Calculating the loss function of the model to be trained according to the classification labels corresponding to each text in the training text set and the classification results;

[0060] The gradient corresponding to each functional layer in the pre-trained model is calculated according to the loss function;

[0061] The gradient and learning rate of each functional layer in the pre-trained model are used to update each functional layer accordingly, so as to update the model to be trained;

[0062] After each round of training, the finally updated model to be trained is used as the text classification model.

[0063] Furthermore, when the training round is less than a preset round, the overall learning rate increases as the training round increases; when the training round is greater than or equal to the preset round, the overall learning rate decreases as the training round increases.

[0064] Furthermore, the step of obtaining the overall learning rate of the model to be trained includes:

[0065] For each training round of the model to be trained, each first round that is smaller than the preset round is substituted into a preset linear growth function to obtain a first learning rate corresponding to each first round;

[0066] Substitute each second round that is greater than or equal to the preset round into a preset exponential descent function to obtain a second learning rate corresponding to each second round, and use each first learning rate and each second learning rate as the overall learning rate of the model to be trained.

[0067] Based on the above structure, various embodiments of the text classification method are proposed.

[0068] Reference Figure 2 , Figure 2 This is a flowchart of the first embodiment of the text classification method of the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a different order than here. The execution subjects of each embodiment of the text classification method of the present invention can be devices such as smart phones, personal computers and servers. For ease of description, the execution subjects are omitted in the following embodiments. In this embodiment, the text classification method includes:

[0069] Step S10, obtaining a text to be classified, and inputting the text to be classified into a text classification model; wherein the text classification model includes a pre-trained model, and during the training process of the text classification model, the learning rate of each functional layer of the pre-trained model decreases from the output layer to the input layer;

[0070] In the present embodiment, a text classification model can be pre-trained, and the text classification model can at least include a pre-trained model. If the pre-trained model itself does not have a text classification layer, the text classification model can also include a subsequent task model for realizing a specific text classification task. Wherein, the pre-trained model can adopt any pre-trained model that can be used for a text classification task, without limitation, for example, Bert (Bidirectional Encoder Representation from Transformers, a bidirectional encoder representation from a transformer) or XLNet (an autoregressive language model) etc. can be adopted. The subsequent task model can adopt a neural network model, connected behind the pre-trained model, and the subsequent task model can be set according to a specific text classification task, for example, the number of output units of the output layer of the subsequent task model can be set according to the target category of the text classification, and the target category corresponding to each output unit. During the training process of the text classification model, the learning rates of each functional layer in the pre-trained model can be set differently. The learning rate of the functional layer closer to the output layer is larger, and the learning rate of the functional layer closer to the input layer is smaller. That is, the learning rate of each functional layer of the pre-trained model decreases from the output layer to the input layer. The input layer refers to the first layer of the text classification model, and the output layer refers to the last layer of the text classification model. That is, from the output layer to the input layer, the learning rate of each functional layer in the pre-trained model gradually decreases.

[0071] It should be noted that in the following embodiments, when the up and down directions are involved, the input layer direction is down and the output layer direction is up, and when the front and back directions are involved, the input layer direction is front and the output layer direction is back.

[0072] The pre-trained model generally includes multiple single layers connected in sequence. The functional layers of the pre-trained model can be divided according to the role of each single layer in text analysis. A functional layer includes at least one single layer. Since it is divided according to function, the single layers of each functional layer in the text analysis functional layer can be connected in sequence or spaced apart. The distance between the functional layer and the output layer can be based on the distance between the topmost single layer in the functional layer and the output layer, or the distance between the bottommost single layer in the functional layer and the output layer.

[0073] Specifically, in this embodiment, the upper layer in the pre-trained model generally contains more information at the semantic level, the middle layer generally contains more information at the syntactic level, and the lower layer generally contains more information on phrases. According to this specific situation, the pre-trained model can be divided into three functional layers: the upper layer, the middle layer and the lower layer, wherein the upper layer is the layer closer to the output layer. For example, the pre-trained model has 12 single layers, from bottom to top, the 1st to 4th single layers are divided into the lower layer, the first 5-8th are divided into the middle layer, and the 9th to 12th are divided into the upper layer. Then, the learning rate of the upper layer can be set to be larger than the learning rate of the middle layer, and the learning rate of the middle layer can be set to be larger than the lower layer. The difference between the learning rates of each functional layer can be set according to the specific application scenario.

[0074] For text that needs to be classified, the text is used as the text to be classified, and the text to be classified is input into the trained text classification model.

[0075] Step S20, controlling the text classification model to process the text to be classified to obtain a text classification result of the text to be classified.

[0076] The text classification model is controlled to process the text to be classified to obtain the text classification result of the text to be classified. Specifically, when the text classification model only includes a pre-training model, and the pre-training model includes a text classification layer, that is, when the pre-training model is a text classification model, each single layer of the pre-training model can be controlled to process the text to be classified to obtain the text classification result. When the text classification model also includes a subsequent task model, each single layer of the pre-training model can be controlled to process the text to be classified, and the high-dimensional feature vector extracted from the text to be classified is obtained through each layer. The high-dimensional feature vector is input into the subsequent task model, and the subsequent task model is controlled to classify the high-dimensional feature vector to obtain the text classification result. Among them, the text classification result can be the probability that the text to be classified belongs to each category, or it can be the category number to which the text to be classified belongs. The specific control of the text classification model to process the text to be classified can also refer to the existing process of using a machine learning model for data processing, which will not be described in detail here.

[0077] In this embodiment, through analysis, the pre-trained model is trained on a large number of data sets, so it has universality in mining the phrase level, syntactic level and semantic level information of the text. When the pre-trained model is applied to a specific text classification task, the syntactic level information to be mined will be somewhat different depending on the specific classification category, and the text semantic information to be mined will be even more different, while there is not much difference in the phrase level information.

[0078] Based on this, in order to make the pre-trained model more suitable for a specific text classification task when used for the specific text classification task, the information of the text semantic level and syntactic level unique to the specific text classification task can be mined; in the present embodiment, the pre-trained model is divided into multiple functional layers, the functional layer close to the output layer is generally used to mine the semantic level information and syntactic level information of the text, and the functional layer close to the input layer is generally used to mine the phrase level information of the text; by adopting a larger learning rate in the functional layer close to the output layer of the pre-trained model during the training process of the text classification model, the functional layer close to the output layer is changed more, so that the functional layer close to the output layer can be learned more, which can not only inherit the universal effect of the pre-trained model in mining the text semantic level and syntactic level information, but also can mine more text semantic levels and syntactic levels in specific text classification tasks in the training text set. hierarchical information; and by adopting a smaller learning rate in the functional stratification close to the input layer, the functional stratification close to the input layer is changed less, so that the functional stratification close to the input layer will not be greatly disturbed by the training text set, so that the universality of the pre-trained model in phrase information mining can be more utilized, and the effect of the pre-trained model can be better exerted; that is, in this embodiment, by adopting a larger learning rate in the functional stratification close to the output layer and a smaller learning rate in the functional stratification close to the input layer, the trained text classification model can not only learn the unique text semantic level and syntactic level information in the specific text classification task, but also exert the universal effect of the pre-trained model in the mining of semantic level information, syntactic level information and phrase information, thereby improving the accuracy of the text classification model in text classification, and making the result obtained by using the text classification model to classify the text to be classified more accurate.

[0079] Further, based on the above first embodiment, a second embodiment of the text classification method of the present invention is proposed. In this embodiment, before step S10, the method further includes:

[0080] Step S30, obtaining the overall learning rate of the model to be trained, and determining the learning rate of each functional layer of the pre-trained model in the model to be trained according to the overall learning rate;

[0081] Furthermore, a text classification model to be trained (hereinafter referred to as the model to be trained) can be constructed in advance. The model to be trained can include at least a pre-trained model and can also include a subsequent task model. The parameters of the subsequent task model can be parameters initialized based on experience or randomly initialized. The parameters in the pre-trained model are obtained by pre-training with a large number of data sets. When applied to a specific text classification task, the model to be trained needs to be trained. The training process mainly involves adjusting the parameters of the pre-trained model and the subsequent task model to adapt to the specific text classification task.

[0082] When training the model to be trained, it is necessary to determine the learning rate of each part of the model to be trained. In this embodiment, the overall learning rate of the model to be trained can be obtained, and the overall learning rate of the model to be trained can be used as a benchmark learning rate for each part of the model to be trained. At this benchmark learning rate, the learning rate of each part can be adjusted as needed. The overall learning rate can be set in advance based on experience. The training process can be carried out for multiple rounds of iterative training, and the overall learning rate of each round of training can be the same or different. Further, in a feasible real-time manner, in the process of training the model to be trained, the overall learning rate of the model to be trained can be adjusted by monitoring the convergence of the model to be trained; for example, when the model to be trained still does not converge after multiple rounds of training, the overall learning rate can be increased to find the convergence point as soon as possible; when the convergence speed of the model to be trained slows down, the overall learning rate can be reduced to find the best convergence point.

[0083] After obtaining the overall learning rate of the model to be trained, the learning rate of each functional layer of the pre-trained model can be determined according to the overall learning rate. It can be that the overall learning rate is used as the learning rate of any functional layer of the pre-trained model, and then the learning rate of other functional layers is determined based on the functional layer. Specifically, after determining the learning rate of a functional layer, each functional layer in front of the functional layer can subtract a number greater than zero from the learning rate of the functional layer, or divide it by a number greater than one. It can be understood that the farther the distance from the functional layer, the greater the value subtracted, or the greater the value of the divisor, so that the learning rate of each functional layer decreases successively as the functional layer moves forward; each functional layer behind the functional layer can add a number greater than zero or divide it by a number less than one based on the learning rate of the functional layer. It can be understood that the farther the distance from the functional layer, the greater the value added, or the smaller the divisor, so that the learning rate of each functional layer increases successively as the functional layer moves backward. Among them, each value can be set in advance according to specific needs.

[0084] The overall learning rate can be used as the learning rate of the subsequent task model. If the model to be trained also includes other parts besides the pre-trained model and the subsequent task model, the overall learning rate can also be used as the learning rate of other parts. For example, in front of the pre-trained model, an input layer that matches the specific text classification task is set, and the learning rate of the input layer can use the overall learning rate.

[0085] Step S40, training the model to be trained according to the learning rate of each functional layer of the pre-trained model to obtain the text classification model.

[0086] After determining the learning rates of each functional layer of the pre-trained model, the model to be trained can be trained according to the learning rates of each functional layer of the pre-trained model. Specifically, when updating each part of the model to be trained, each part is updated according to the learning rate of each part.

[0087] It should be noted that when the overall learning rate of each round of training is different, the learning rate of each part in each round of training also changes accordingly with the overall learning rate; the learning rate of each functional layer of the pre-trained model is determined in each round of training based on the overall learning rate of that round of training, and the learning rate of each functional layer in each round of training still conforms to the rule that the learning rate of the later functional layer is greater than that of the earlier functional layer.

[0088] Furthermore, the step of determining the learning rate of each functional layer of the pre-trained model in the model to be trained according to the overall learning rate in step S30 includes:

[0089] Step S301, using the overall learning rate as the learning rate of the functional layer that is closest to the output layer of the pre-trained model in the model to be trained;

[0090] Furthermore, after determining the overall learning rate of the model to be trained, the overall learning rate can be used as the learning rate of the functional layer closest to the output layer in the pre-trained model (the functional layer closest to the output layer), that is, as the learning rate of the last functional layer (also called the topmost functional layer) of the pre-trained model.

[0091] Step S302, taking the learning rate of the functional layer closest to the output layer as a reference, taking the output layer direction as the upward direction, and taking the input layer direction as the downward direction, the learning rate of the previous functional layer corresponding to each functional layer is divided by a preset value, and the result is used as the learning rate of each functional layer, wherein the preset value is greater than one.

[0092] After determining the learning rate of the functional layer closest to the output layer, the learning rate of the functional layer is used as a reference, the output layer direction is taken as the upper direction, and the input layer direction is taken as the lower direction. The learning rate of the upper functional layer corresponding to each functional layer is divided by the preset value, and the result is used as the learning rate of each functional layer. That is, counting from top to bottom, the learning rate of the first functional layer has been determined, the learning rate of the second functional layer is obtained by dividing the learning rate of the first layer by the preset value, the learning rate of the third functional layer is obtained by dividing the learning rate of the second layer by the preset value, and so on, until the learning rate of each functional layer is determined. It should be noted that the preset value is greater than one, so as to ensure that the learning rate of each functional layer is decreasing from top to bottom; the preset values ​​used in calculating each functional layer can be the same or different, and can be set according to needs.

[0093] In this embodiment, by obtaining the overall learning rate of the model to be trained, and then determining the learning rate of each functional layer of the pre-trained model according to the overall learning rate, the learning rate of each part of the model to be trained is based on the overall learning rate, which is conducive to the overall convergence of the model to be trained in a general direction, and is convenient for adjusting the convergence direction and convergence speed of the model to be trained; the learning rate of each part of the model to be trained is determined based on the overall learning rate, so that the learning rate of each part can be adjusted on the overall learning rate, which is conducive to the convergence direction and convergence speed of each part. The learning rate of each functional layer of the pre-trained model is based on the overall learning rate. Furthermore, the learning rate of the functional layer close to the output layer is greater than that of the functional layer close to the input layer, so that the functional layer close to the output layer can mine more text semantic level and syntactic level information in specific text classification tasks from the training data set through a larger learning rate, thereby improving the classification accuracy of the trained text classification model in specific text classification tasks; while the learning rate of the functional layer close to the input layer is smaller, so that it can inherit more phrase level information learned by the pre-trained model in a large number of data sets, thereby further improving the accuracy of the text classification model in mining phrase level information, that is, further improving the classification accuracy of the text classification model.

[0094] Furthermore, the text classification method also includes:

[0095] Step S50, performing functional analysis on each single layer of the pre-trained model in the model to be trained, and determining the text analysis function corresponding to each single layer;

[0096] Furthermore, in this embodiment, each single layer of the pre-trained model in the to-be-trained model may be divided in advance to obtain each functional layer. Specifically, each single layer may be functionally analyzed to determine the text analysis function corresponding to each single layer.

[0097] The method of performing functional analysis on each single layer can be: predetermine several text analysis functions, for example, it can be a phrase level information mining function, a syntactic level information mining function and a semantic level information mining function; then determine one from each text analysis function in turn as a target text analysis function (hereinafter referred to as the target function), for the target function, analyze whether each single layer can achieve the target function, if it can achieve the target function, then determine the text analysis function of the single layer as the target function.

[0098] Specifically, based on the target function, a test model connected to the pre-trained model can be set. The test model can adopt a neural network model. The test model corresponds to a specific text processing task. The accuracy of the processing result of the text processing task depends on the part of the pre-trained model that realizes the target function. For example, for the phrase level information mining function, the test model can be a text phrase segmentation model. The test model is connected to the pre-trained model. The test model obtains the phrase segmentation result of the text based on the high-dimensional feature vector of the text output by the pre-trained model. The accuracy of the phrase segmentation result mainly depends on the part of the pre-trained model that can mine the phrase level information. After setting the model structure of the pre-trained model connected to the test model, the model structure is trained using the training data set; during the training process, only one single layer of the pre-trained model and the test model are trained at a time, and the other single layers of the pre-trained model remain unchanged. After the training is completed, the accuracy of the output result of the model structure is detected (the accuracy calculation method can refer to the existing model output accuracy calculation method); in this way, the accuracy of the model output result when each single layer is trained separately is obtained. The higher the accuracy corresponding to the single layer, the better the effect of the single layer in achieving the target function. Therefore, one or more single layers with higher corresponding accuracy can be used as single layers that can achieve the target function, and then the text analysis function corresponding to the one or more single layers is determined to be the target function.

[0099] Step S60, dividing the single layers with the same text analysis function into the same functional layer to obtain the functional layers of the pre-trained model.

[0100] After determining the text analysis function corresponding to each single layer, the single layers with the same text analysis function are divided into the same functional layer. There are multiple categories of text analysis functions, and multiple functional layers are obtained accordingly.

[0101] It should be noted that, in this embodiment, the step of performing functional analysis on each single layer can be performed before S301, and the order of the steps before step S301 is not limited here.

[0102] In the pre-trained model, the single layer close to the output layer can generally mine more semantic and syntactic information, and the single layer close to the input layer can generally mine more phrase information. According to this rule, if the functional layers are divided equally according to the number of single layers in the pre-trained model, the functional layers divided to a certain extent conform to the above rule, that is, the functional layers close to the output layer are used to mine speech and syntactic information, and the functional layers close to the input layer are used to mine phrase information. However, due to the role played by each single layer in the pre-trained model and the functions that can be achieved are uncertain, the functional layers directly divided equally may not be accurate enough. For example, if the pre-trained model includes 12 single layers, if they are equally divided into 3 functional layers: 1-4, 5-8 and 9-12, in fact, the function of 1-2 layers may be to mine phrase information, the function of 3-6 layers may be to mine syntactic information, and the function of 7-12 layers may be to mine semantic information. In this embodiment, in order to divide the pre-trained model more accurately, the function of each single layer of the pre-trained model is analyzed in advance to determine the text analysis function of each single layer, and then the single layer with the same text analysis function is divided into a functional layer to obtain the functional layers of the pre-trained model. The functional layers obtained by the division are more accurate, thereby making the subsequent training process of each functional layer using different learning rates more precise, thereby improving the accuracy of text classification by the trained text classification model.

[0103] Further, based on the above first and second embodiments, a third embodiment of the text classification method of the present invention is proposed. In this embodiment, step S40 includes:

[0104] Step S401, in each round of training, inputting a preset training text set into the model to be trained, controlling the model to be trained to process the training text set, and obtaining a classification result corresponding to each text in the training text set;

[0105] Furthermore, the model to be trained can be trained for multiple rounds. In each round of training, a preset training text set is input into the model to be trained, and the model to be trained is controlled to process the training text set to obtain classification results corresponding to each text in the training text set. It should be noted that the basis of each round of training is the model to be trained after the previous round of training.

[0106] Step S402, calculating the loss function of the model to be trained according to the classification label corresponding to each text in the training text set and the classification result;

[0107] According to the classification labels and classification results corresponding to each text in the training text set, the loss function of the model to be trained is calculated. Specifically, the loss function of the model to be trained can be calculated with reference to the loss function commonly used in the existing machine learning model, which will not be described in detail here.

[0108] Step S403, calculating the gradient corresponding to each functional layer in the pre-trained model according to the loss function;

[0109] The gradients corresponding to each functional layer in the pre-trained model are calculated based on the loss function. If the model to be trained includes other parts, such as a subsequent task model, the gradients corresponding to the subsequent task model are calculated based on the loss function. Specifically, the gradients corresponding to each functional layer refer to the gradients corresponding to each parameter in each functional layer, and the gradients corresponding to the subsequent task model are also the gradients corresponding to each parameter in the subsequent task model.

[0110] Step S404, using the gradient and learning rate of each functional layer in the pre-trained model to update each functional layer accordingly, so as to update the model to be trained;

[0111] The gradient and learning rate of each functional layer in the pre-trained model are used to update each functional layer accordingly. That is, for each functional layer, the gradient and learning rate of the functional layer are used to update the parameters of the functional layer. Similarly, the gradient and learning rate of the subsequent task model can be used to update the parameters of the subsequent task model. Specifically, the method of updating the parameters by updating the gradient and learning rate of the parameters can refer to the parameter updating method of the existing machine learning model, which will not be described in detail here. After updating the model parameters of each part of the model to be trained, one round of training is completed.

[0112] Step S405: After each round of training, the finally updated model to be trained is used as the text classification model.

[0113] After multiple rounds of training for the model to be trained, the model to be trained that is finally updated is used as the text classification model. The stopping condition of the iterative training can be preset, for example, the training can be stopped when the number of training rounds reaches a preset maximum number of rounds, or when the loss function of the model to be trained is detected to converge, or when the training time reaches a preset maximum training time.

[0114] Furthermore, the overall learning rate of each round of training can be set differently. In the present embodiment, the relationship between the training round and the overall learning rate can be: when the training round is less than the preset round, the overall learning rate increases as the training round increases; when the training round is greater than or equal to the preset round, the overall learning rate decreases as the training round increases.

[0115] Specifically, the overall learning rate corresponding to each round can be pre-set, or the overall learning rate of the first training can be pre-set, that is, the initial learning rate, and the overall learning rate of each subsequent round of training can be calculated based on the initial learning rate. Among them, the learning rate of each round of training can conform to the following rules: when the training round is less than the preset round, the overall learning rate increases with the increase of the training round, and when the training round is greater than or equal to the preset round, the overall learning rate decreases with the increase of the training round. That is, the overall learning rate increases first and then decreases with the increase of the training round, and the preset round is the intermediate turning point. The preset round can be set according to specific needs.

[0116] Further, the initial learning rate and the maximum learning rate may be pre-set, and then a growth function may be set, the independent variable of the growth function is the training round, the dependent variable is the learning rate, and the growth function makes the learning rate be the initial learning rate when the training round is 1, and the learning rate also increases when the training round increases, and the learning rate reaches the maximum learning rate when the training round is the preset training. It should be noted that the growth function may be implemented using a commonly used growth function, such as an exponential growth function, a linear growth function, etc. When the training round is less than the preset round, the training round may be substituted into the growth function to obtain the overall learning rate corresponding to the training round. Correspondingly, a decline function may be set, the independent variable of the decline function is the training round, and the dependent variable is the learning rate, and the decline function makes the learning rate be the maximum learning rate when the training round is the preset round, and the learning rate gradually decreases when the training round increases. The decline function may also be implemented using a commonly used decline function, such as an exponential decline function and a linear decline function. When the training round is greater than or equal to the preset round, the training round can be substituted into the descent function to obtain the overall learning rate corresponding to the training round. In the above manner, the overall learning rate in each round of training can be obtained.

[0117] In this embodiment, by setting the overall learning rate in each round of training of the model to be trained, the overall learning rate increases first and then decreases as the number of training rounds increases; since the pre-trained model is obtained through training of a large number of data sets, it may have converged to a certain local convergence range, and in the training process of the text classification model, it is necessary to adjust the model parameters of the pre-trained model part so that the pre-trained model can be applied to a specific text classification task. Therefore, in order to avoid the convergence direction of the pre-trained model being limited to the local convergence range and causing it to be unable to converge to the global optimal convergence point, in this embodiment, by allowing the overall learning of the model to be trained to increase first, the pre-trained model part can jump out of the local convergence range, find the global optimal convergence point, and converge rapidly towards the global optimal convergence point based on a larger learning rate, that is, the pre-trained model can quickly converge to the appropriate area of ​​the parameter space at the beginning of training. And by increasing the overall learning of the model to be trained first and then decreasing, the parameters of each part of the model to be trained can be refined in the appropriate area of ​​the parameter space, so that it finally converges to the global optimal convergence point, and then obtains a text classification model with higher accuracy.

[0118] Furthermore, the step of obtaining the overall learning rate of the model to be trained in step S30 includes:

[0119] Step S303, for each training round of the model to be trained, each first round that is smaller than the preset round is substituted into a preset linear growth function to obtain a first learning rate corresponding to each first round;

[0120] Step S304, substitute each second round that is greater than or equal to the preset round into the preset exponential descent function, obtain the second learning rate corresponding to each second round, and use each first learning rate and each second learning rate as the overall learning rate of the model to be trained.

[0121] Furthermore, in the present embodiment, the above-mentioned growth function may adopt a preset linear growth function, that is, for each training round of the model to be trained, the training round that is less than the preset round is taken as the first round, and each first round is substituted into the preset linear growth function to obtain the first learning rate corresponding to each first round. The above-mentioned decrease function may adopt a preset exponential decrease function, that is, the training round that is greater than or equal to the preset round is taken as the second round, and each second round is substituted into the preset exponential decrease function to obtain the second learning rate corresponding to each second round. Each first learning rate and each second learning rate are taken as the overall learning rate of the model to be trained, that is, the overall learning rate corresponding to each round of training of the model to be trained is determined. The representation of the preset linear growth function and the preset exponential decrease function in the coordinate system can be referred to. Figure 3 .

[0122] After determining the overall learning rate of each round of training for the model to be trained, for each round of training, the overall learning rate of the training round is used to determine the learning rate of each functional level of the pre-trained model in the training round. That is, the overall learning rate of each round of training is different, so the learning rate of the same functional level is also different in each round of training.

[0123] In this embodiment, by using a linear growth function to calculate the overall learning rate, the overall learning rate of the model to be trained increases linearly at the beginning of training, so that the model to be trained can more smoothly explore the appropriate area of ​​the parameter space. By using an exponential decline function to calculate the overall learning rate, the overall learning rate of the model to be trained decreases from rapid to slow during the decline phase, first quickly converges to the vicinity of the global optimal convergence point, speeds up the training speed, and then slowly explores in the vicinity of the global optimal convergence point to avoid the model to be trained missing the global optimal convergence point.

[0124] In addition, an embodiment of the present invention further provides a text classification device, wherein the device is deployed on a coordination device participating in horizontal federated learning, referring to Figure 4 , the device comprises:

[0125] The acquisition module 10 is used to acquire the text to be classified and input the text to be classified into a text classification model; wherein the text classification model at least includes a pre-trained model, and during the training process of the text classification model, the learning rate of each functional layer of the pre-trained model decreases from the output layer to the input layer;

[0126] The processing module 20 is used to control the text classification model to process the text to be classified to obtain a text classification result of the text to be classified.

[0127] Furthermore, the acquisition module 10 is also used for:

[0128] Obtaining the overall learning rate of the model to be trained, and determining the learning rate of each functional layer of the pre-trained model in the model to be trained according to the overall learning rate;

[0129] The text classification device also includes:

[0130] The training module is used to train the model to be trained according to the learning rate of each functional layer of the pre-trained model to obtain the text classification model.

[0131] Furthermore, the acquisition module 10 includes:

[0132] A determination unit, configured to use the overall learning rate as the learning rate of the functional layer of the model to be trained whose pre-trained model is closest to the output layer;

[0133] The first calculation unit is used to take the learning rate of the functional layer closest to the output layer as a benchmark, take the output layer direction as the upper direction, and take the input layer direction as the lower direction, and divide the learning rate of the previous functional layer corresponding to each functional layer by a preset value to obtain the result as the learning rate of each functional layer, wherein the preset value is greater than one.

[0134] Furthermore, the text classification device also includes:

[0135] An analysis module, used for performing functional analysis on each single layer of the pre-trained model in the model to be trained, and determining the text analysis function corresponding to each single layer;

[0136] The division module is used to divide the single layers with the same text analysis function into the same functional layer to obtain the functional layers of the pre-trained model.

[0137] Furthermore, the training module includes:

[0138] An input unit, used to input a preset training text set into the model to be trained in each round of training, control the model to be trained to process the training text set, and obtain a classification result corresponding to each text in the training text set;

[0139] A second calculation unit, used to calculate the loss function of the model to be trained according to the classification label corresponding to each text in the training text set and the classification result;

[0140] A third calculation unit, used to calculate the gradient corresponding to each functional layer in the pre-trained model according to the loss function;

[0141] An updating unit, used to update each functional layer correspondingly using the gradient and learning rate of each functional layer in the pre-trained model, so as to update the model to be trained;

[0142] The determination unit is used to use the finally updated model to be trained as the text classification model after each round of training.

[0143] Furthermore, when the training round is less than a preset round, the overall learning rate increases as the training round increases; when the training round is greater than or equal to the preset round, the overall learning rate decreases as the training round increases.

[0144] Furthermore, the acquisition module 10 includes:

[0145] A fourth calculation unit is used to substitute each first round that is smaller than the preset round into a preset linear growth function for each training round of the model to be trained, so as to obtain a first learning rate corresponding to each first round;

[0146] The fifth calculation unit is used to substitute each second round greater than or equal to the preset round into the preset exponential descent function, obtain the second learning rate corresponding to each second round, and use each first learning rate and each second learning rate as the overall learning rate of the model to be trained.

[0147] The expanded contents of the specific implementation of the text classification device of the present invention are basically the same as the embodiments of the above-mentioned text classification method, and will not be elaborated here.

[0148] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a text classification program is stored. When the text classification program is executed by a processor, the steps of the text classification method described below are implemented.

[0149] The various embodiments of the text classification device and the computer-readable storage medium of the present invention may refer to the various embodiments of the text classification method of the present invention, and will not be described in detail here.

[0150] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0151] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the embodiments of the present invention.

[0153] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A text classification method, characterized in that: The method comprises the following steps: Obtaining a text to be classified, and inputting the text to be classified into a text classification model; The text classification model at least includes a pre-training model, and during the training process of the text classification model, the learning rate of each functional layer of the pre-training model decreases from the output layer to the input layer, the pre-training model includes a plurality of sequentially connected single layers, and the functional layers of the pre-training model are obtained by dividing according to the role of each single layer in text analysis; Controlling the text classification model to process the text to be classified to obtain a text classification result of the text to be classified; Before the step of obtaining the text to be classified and inputting the text to be classified into the text classification model, the method further includes: Obtaining the overall learning rate of the model to be trained, and determining the learning rate of each functional layer of the pre-trained model in the model to be trained according to the overall learning rate; The model to be trained is trained according to the learning rate of each functional layer of the pre-trained model to obtain the text classification model; The step of determining the learning rate of each functional layer of the pre-trained model in the model to be trained according to the overall learning rate comprises: Using the overall learning rate as the learning rate of the functional layer of the model to be trained that is closest to the output layer of the pre-trained model; Taking the learning rate of the functional layer closest to the output layer as a benchmark, with the output layer direction as the upward direction and the input layer direction as the downward direction, the learning rate of the previous functional layer corresponding to each functional layer is divided by a preset value, and the result is used as the learning rate of each functional layer, wherein the preset value is greater than one.

2. The text classification method according to claim 1, characterized in that: Before the step of determining the learning rate of each functional layer of the pre-trained model in the model to be trained according to the overall learning rate, the step further includes: Performing functional analysis on each single layer of the pre-trained model in the model to be trained to determine the text analysis function corresponding to each single layer; The single layers with the same text analysis function are divided into the same functional layer to obtain the functional layers of the pre-trained model.

3. The text classification method according to claim 1, characterized in that: The step of training the model to be trained according to the learning rate of each functional layer of the pre-trained model to obtain the text classification model comprises: In each round of training, a preset training text set is input into the model to be trained, and the model to be trained is controlled to process the training text set to obtain a classification result corresponding to each text in the training text set; Calculating the loss function of the model to be trained according to the classification labels corresponding to each text in the training text set and the classification results; The gradient corresponding to each functional layer in the pre-trained model is calculated according to the loss function; The gradient and learning rate of each functional layer in the pre-trained model are used to update each functional layer accordingly, so as to update the model to be trained; After each round of training, the finally updated model to be trained is used as the text classification model.

4. The text classification method according to any one of claims 1 to 3, characterized in that: When the training round is less than the preset round, the overall learning rate increases as the training round increases; when the training round is greater than or equal to the preset round, the overall learning rate decreases as the training round increases.

5. The text classification method according to claim 4, characterized in that: The step of obtaining the overall learning rate of the model to be trained includes: For each training round of the model to be trained, each first round that is smaller than the preset round is substituted into a preset linear growth function to obtain a first learning rate corresponding to each first round; Substitute each second round that is greater than or equal to the preset round into a preset exponential descent function to obtain a second learning rate corresponding to each second round, and use each first learning rate and each second learning rate as the overall learning rate of the model to be trained.

6. A text classification device, characterized in that: The device comprises: An acquisition module is used to acquire a text to be classified and input the text to be classified into a text classification model; wherein the text classification model at least includes a pre-training model, and during the training process of the text classification model, the learning rate of each functional layer of the pre-training model decreases from the output layer to the input layer, the pre-training model includes a plurality of sequentially connected single layers, and the functional layers of the pre-training model are obtained by dividing according to the role played by each single layer in text analysis; A processing module, used for controlling the text classification model to process the text to be classified to obtain a text classification result of the text to be classified; The acquisition module is also used for: Obtaining the overall learning rate of the model to be trained, and determining the learning rate of each functional layer of the pre-trained model in the model to be trained according to the overall learning rate; The text classification device also includes: A training module, used for training the model to be trained according to the learning rate of each functional layer of the pre-trained model to obtain the text classification model; The acquisition module comprises: A determination unit, configured to use the overall learning rate as the learning rate of the functional layer of the model to be trained whose pre-trained model is closest to the output layer; The first calculation unit is used to take the learning rate of the functional layer closest to the output layer as a benchmark, take the output layer direction as the upper direction, and take the input layer direction as the lower direction, and divide the learning rate of the previous functional layer corresponding to each functional layer by a preset value to obtain the result as the learning rate of each functional layer, wherein the preset value is greater than one.

7. A text classification device, characterized in that: The text classification device comprises: a memory, a processor, and a text classification program stored in the memory and executable on the processor. When the text classification program is executed by the processor, the steps of the text classification method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a text classification program, and when the text classification program is executed by a processor, the steps of the text classification method according to any one of claims 1 to 5 are implemented.

Images