Label classification method and device for long text, equipment, storage medium and product

Abstract

The invention discloses a label classification method and device for a long text, equipment, a storage medium and a product, relates to the technical field of computers, and discloses a label classification method for the long text, which comprises the following steps: obtaining a to-be-classified text; and based on the to-be-classified text, performing label classification processing through a preset large language model to obtain a label classification result. According to the method, the to-be-classified texts are converted into the text sequence vectors by the large language model through position coding, and then correlation among the text sequence vectors is calculated in parallel by utilizing a multi-head self-attention mechanism, that is, the large language model can process the text sequence vectors in parallel, so that the video memory utilization rate of the large language model is improved, and the classification efficiency is improved. Due to the improvement of the utilization rate of the video memory, the large language model can process the long text with more characters, namely, the large language model does not need to segment the long text, so that semantic loss caused by segmentation is avoided, and the tag classification accuracy of the long text is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a label classification method, device, equipment, storage medium and product for long texts. Background Art

[0002] Natural language processing (NLP) is an important branch of computer science and artificial intelligence. NLP enables computers to understand and process human language. In current NLP applications, text classification is one of the basic and key tasks. With the increase in user demand, label classification of long texts often requires classification into multiple categories. For example, a news article covers multiple topics such as politics, economy, and society. Compared with traditional single-label classification, multi-label classification requires the model to be able to identify and process multiple related topics in the text.

[0003] In the related art, the autoregressive language pre-training model XLNet is usually used for text multi-label classification. However, the XLNet model has a character limit of 512 characters, that is, when the XLNet model is used to process long texts exceeding the character limit, the long text needs to be segmented into multiple independent texts, and text segmentation will lead to semantic loss, which in turn leads to low label classification accuracy of long texts.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a label classification method for long texts, aiming to solve the technical problem of low accuracy of label classification of long texts.

[0006] To achieve the above purpose, the present application proposes a label classification method for a long text, and the label classification method for a long text includes:

[0007] Get the text to be classified;

[0008] Based on the text to be classified, label classification processing is performed through a preset large language model to obtain a label classification result, wherein the model architecture of the large language model includes a multi-head self-attention mechanism and position encoding, the position encoding is used to encode text position information into a corresponding position vector, and construct a text sequence vector based on the position vector, and the multi-head self-attention mechanism is to calculate the correlation between text sequence vectors in parallel through multiple self-attention layers.

[0009] Optionally, before the step of performing label classification processing based on the text to be classified by using a preset large language model to obtain a label classification result, the method includes:

[0010] Obtaining text training samples and classification labels of the text training samples;

[0011] Performing vector conversion on the text training sample to obtain a text vector, performing position encoding on the text vector to obtain a position vector, and generating an encoded text vector based on the position vector;

[0012] Based on a plurality of preset self-attention layers, performing attention weighted calculation on the encoded text vector in parallel to obtain a weighted text vector;

[0013] Based on the weighted text vector and the classification label, the preset model to be trained is iteratively trained to obtain a large language model that meets the accuracy requirements.

[0014] Optionally, the step of performing attention weighted calculation on the encoded text vector in parallel based on the preset multiple self-attention layers to obtain a weighted text vector includes:

[0015] Get the preset initial weight matrix of each attention layer;

[0016] Based on the first low-rank parameter matrix under the respective attention layers, low-dimensional mapping is performed on the encoded text vector to obtain a text vector after dimensionality reduction;

[0017] Based on the second low-rank parameter matrix under the respective attention layers, high-dimensional mapping is performed on the reduced-dimensional text vector to obtain an increased-dimensional text vector, wherein the dimension of the increased-dimensional text vector is the same as the dimension of the encoded text vector;

[0018] Based on the initial weight matrix and the dimension-upgraded text vector, the target text vectors of the respective attention layers are calculated, and the target text vectors of the respective attention layers are fused to obtain a weighted text vector.

[0019] Optionally, the step of iteratively training a preset model to be trained based on the weighted text vector and the classification label to obtain a large language model that meets the accuracy condition includes:

[0020] Based on the weighted text vector, label classification processing is performed through a preset model to be trained to obtain a predicted classification result;

[0021] Performing error calculation on the predicted classification result and the classification label to obtain an error result;

[0022] Determining whether the error result meets the error standard indicated by a preset error threshold range;

[0023] If the error result does not meet the error standard indicated by the preset error threshold range, the target weight matrix is ​​calculated based on the preset gradient descent algorithm, and the model parameters are updated based on the target weight matrix, and then based on the updated model to be trained, the preset multiple self-attention layers are returned, and the encoded text vector is subjected to attention weighted calculation in parallel to obtain the weighted text vector step, and the training is stopped after the error result meets the error standard indicated by the preset error threshold range to obtain a large language model, wherein different learning rate parameters are set for the first low-rank parameter matrix and the second low-rank parameter matrix in the gradient descent algorithm respectively.

[0024] Optionally, the step of iteratively training a preset model to be trained based on the weighted text vector and the classification label to obtain a large language model that meets the accuracy condition includes:

[0025] Determining a target weight matrix for each iteration cycle based on the weighted text vector and the classification label;

[0026] Performing low-precision quantization on the target weight matrix of each iteration cycle to obtain a quantized weight matrix;

[0027] Based on the quantized weight matrix, the model to be trained in each iteration cycle is updated until a large language model that meets the accuracy requirements is obtained.

[0028] Optionally, the step of obtaining the text to be classified includes:

[0029] Get the initial text and the number of characters in the initial text;

[0030] Determining whether the number of characters is greater than a preset character threshold;

[0031] If the number of characters is greater than the character threshold, the initial text is segmented to obtain segmented texts, and semantic feature values ​​of the segmented texts are extracted;

[0032] Based on the semantic feature values, the semantic difference values ​​between adjacent segmented texts are calculated, and based on the semantic difference values, the initial text is segmented to obtain the text to be classified.

[0033] In addition, to achieve the above purpose, the present application also proposes a label classification device for long texts, the label classification device for long texts comprising:

[0034] An acquisition module is used to obtain the text to be classified;

[0035] The classification module is used to perform label classification processing based on the text to be classified through a preset large language model to obtain a label classification result, wherein the model architecture of the large language model includes a multi-head self-attention mechanism and position encoding, the position encoding is used to encode the text position information into a corresponding position vector, and construct a text sequence vector based on the position vector, and the multi-head self-attention mechanism is to calculate the correlation between text sequence vectors in parallel through multiple self-attention layers.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes a label classification device for long texts, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the label classification method for long texts as described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the long text label classification method described above are implemented.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the long text label classification method described above are implemented.

[0039] One or more technical solutions proposed in this application have at least the following technical effects:

[0040] In the related art, the autoregressive language pre-training model XLNet is usually used for text multi-label classification. However, the XLNet model has a character limit of 512 characters, that is, when processing a long text exceeding the character limit through the XLNet model, the long text needs to be segmented into multiple independent texts, and text segmentation will lead to semantic loss, which in turn leads to low label classification accuracy of the long text. In comparison, the present application obtains the text to be classified; based on the text to be classified, label classification processing is performed through a preset large language model to obtain a label classification result, wherein the model architecture of the large language model includes a multi-head self-attention mechanism and position encoding, the position encoding is used to encode the text position information into a corresponding position vector, and construct a text sequence vector based on the position vector, and the multi-head self-attention mechanism is to calculate the correlation between text sequence vectors in parallel through multiple self-attention layers. It is understandable that the large language model in the present application converts the text to be classified into a text sequence vector through position encoding, and then uses the multi-head self-attention mechanism to parallelly calculate the correlation between the text sequence vectors, that is, the large language model can process the text sequence vectors in parallel, so as to improve the memory utilization of the large language model. The improvement in memory utilization can enable the large language model to process long texts with more characters, that is, the large language model does not need to segment the long text, thereby avoiding the semantic loss caused by segmentation, and thus improving the label classification accuracy of the long text. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A flowchart diagram of the first embodiment of the tag classification method for long texts of the present application;

[0044] Figure 2 A schematic diagram of the overall process of large language model training and reasoning for the label classification method of long text in this application;

[0045] Figure 3 A flowchart diagram of the second embodiment of the label classification method for long texts of the present application;

[0046] Figure 4 A schematic diagram of the model fine-tuning process for the label classification method of long text in this application;

[0047] Figure 5 A flowchart diagram of the third embodiment of the label classification method for long texts of the present application;

[0048] Figure 6 This is a schematic diagram of the module structure of the label classification device for long texts in an embodiment of the present application;

[0049] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the tag classification method for long text in the embodiment of the present application.

[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solution of the embodiment of the present application is: obtaining the text to be classified; based on the text to be classified, performing label classification processing through a preset large language model to obtain a label classification result, wherein the model architecture of the large language model includes a multi-head self-attention mechanism and position encoding, the position encoding is used to encode the text position information into a corresponding position vector, and construct a text sequence vector based on the position vector, and the multi-head self-attention mechanism is to calculate the correlation between text sequence vectors in parallel through multiple self-attention layers.

[0054] In this embodiment, the long text label classification device is used as the execution body. For the convenience of description, it is hereinafter referred to as "device" for explanation.

[0055] In the related art, the autoregressive language pre-training model XLNet is usually used for text multi-label classification. However, the XLNet model has a character limit of 512 characters, that is, when the XLNet model is used to process a long text exceeding the character limit, the long text needs to be segmented into multiple independent texts, and text segmentation will lead to semantic loss, which in turn leads to low label classification accuracy of the long text.

[0056] This application provides a solution to achieve label classification of long texts and improve the accuracy of label classification of long texts.

[0057] It can be seen from the above embodiments that the large language model in the present application converts the text to be classified into a text sequence vector through position encoding, and then uses a multi-head self-attention mechanism to parallelly calculate the correlation between the text sequence vectors, that is, the large language model can process the text sequence vectors in parallel, thereby improving the memory utilization of the large language model. The improvement in memory utilization enables the large language model to process long texts with more characters, that is, the large language model does not need to segment the long text, thereby avoiding the semantic loss caused by segmentation, and thereby improving the label classification accuracy of the long text.

[0058] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a terminal system, etc. that can realize the above functions. The following takes the label classification device of a long text as an example to illustrate this embodiment and the following embodiments.

[0059] Based on this, the present application embodiment provides a label classification method for long texts, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the tag classification method for long texts of the present application.

[0060] In this embodiment, the tag classification method of the long text includes steps S100 to S200:

[0061] Step S100, obtaining the text to be classified;

[0062] It should be noted that the text to be classified refers to text data that has not yet been assigned a category label. They need to use a classification model to determine their category or label. In natural language processing (NLP) and machine learning, the text to be classified is the input object of the classification task.

[0063] In a specific implementation, the device may obtain the text to be classified by extracting it from different article sources under the premise of obtaining user authorization, or by receiving the text to be classified uploaded by the user, which is not specifically limited here.

[0064] Step S200, based on the text to be classified, label classification processing is performed through a preset large language model to obtain a label classification result, wherein the model architecture of the large language model includes a multi-head self-attention mechanism and position encoding, the position encoding is used to encode the text position information into a corresponding position vector, and construct a text sequence vector based on the position vector, and the multi-head self-attention mechanism is to calculate the correlation between text sequence vectors in parallel through multiple self-attention layers.

[0065] It should be noted that, refer to Figure 2, the large language model refers to a language model containing a large number of parameters, which are trained on a large amount of text data, such as models GPT-3, PaLM, Galactica and LLaMA. Preferably, the large language model is an LLM model. Specifically, LLM is built on the Transformer architecture, in which multi-head attention layers are stacked in a very deep neural network. Specifically, LLM mainly adopts a model architecture (i.e., Transformer) and pre-training objectives (i.e., language modeling) similar to those of the small language model. LLM greatly expands the model size, pre-training data, and total computational effort (expansion multiples), so that natural language can be better understood and high-quality text can be generated based on a given context (e.g., prompt).

[0066] In the specific implementation, the model architecture of the large language model includes a multi-head self-attention mechanism and position encoding. Specifically, LLM is mainly composed of a Transformer architecture. The structure of this model includes a multi-layer Transformer decoder, each layer of which consists of a self-attention mechanism and a feed-forward neural network. In each layer of the Transformer model, the self-attention mechanism first quantitatively evaluates the interaction between each word in the input sequence and other words in the sequence. Subsequently, based on these evaluation results, corresponding weights are applied to each word in the input sequence to enhance the large language model's ability to recognize and capture long-distance dependency features within the text. Next, the feed-forward neural network performs further nonlinear transformations on the output of the self-attention. In addition, LLM can also use position encoding to process the word order information in the sequence, because the Transformer itself cannot process this order information. Position encoding encodes the information of each position into a vector and adds it to the word vector of the corresponding position. Finally, LLM can also use masked self-attention, which allows the large language model to only use the words before the word when predicting the next word, but not the words after the word, thereby enabling the large model to simulate the natural process of humans generating text word by word from left to right.

[0067] In the specific implementation, the present application also proposes to use vLLM to accelerate reasoning in the process of label classification processing through a preset large language model. The core mechanism of vLLM is PagedAttention, which is a technology for paging management of key-value cache (KV cache) space. PagedAttention technology solves this problem by dividing the KV cache of each sequence into multiple fixed-length blocks. Each block contains a certain number of tokens, and these blocks can be efficiently located and accessed when calculating self-attention. It can be understood that each fixed-length block can be likened to a "page" in virtual memory, and each token is similar to a byte, and the sequence can be regarded as a "process" in the operating system. By using a block table, continuous logical blocks can be mapped to non-continuous physical blocks. These physical blocks can be allocated on demand according to the newly generated tags, thereby effectively replacing the continuous video memory allocation and improving the utilization efficiency of the video memory.

[0068] In the related art, the autoregressive language pre-training model XLNet is usually used for text multi-label classification. However, the XLNet model has a character limit of 512 characters, that is, when processing a long text exceeding the character limit through the XLNet model, the long text needs to be segmented into multiple independent texts, and text segmentation will lead to semantic loss, which in turn leads to low label classification accuracy of the long text. In comparison, the present application obtains the text to be classified; based on the text to be classified, label classification processing is performed through a preset large language model to obtain a label classification result, wherein the model architecture of the large language model includes a multi-head self-attention mechanism and position encoding, the position encoding is used to encode the text position information into a corresponding position vector, and construct a text sequence vector based on the position vector, and the multi-head self-attention mechanism is to calculate the correlation between text sequence vectors in parallel through multiple self-attention layers. It is understandable that the large language model in the present application converts the text to be classified into a text sequence vector through position encoding, and then uses the multi-head self-attention mechanism to parallelly calculate the correlation between the text sequence vectors, that is, the large language model can process the text sequence vectors in parallel, so as to improve the memory utilization of the large language model. The improvement in memory utilization can enable the large language model to process long texts with more characters, that is, the large language model does not need to segment the long text, thereby avoiding the semantic loss caused by segmentation, and thus improving the label classification accuracy of the long text.

[0069] Based on the above first embodiment, the present application also proposes another embodiment, referring to Figure 3 , the label classification method of the long text includes:

[0070] In a specific implementation, before the device performs label classification processing based on the text to be classified by using a preset large language model to obtain a label classification result, the method includes:

[0071] Step A100, obtaining text training samples and classification labels of the text training samples;

[0072] It should be noted that the text training samples are the samples that the model is going to learn and understand, and the classification labels are the categories to which these texts belong, which are used to guide the model to learn how to classify new, unseen texts.

[0073] In a specific implementation, the purpose of the device acquiring text training samples and classification labels of the text training samples is to collect data for training a model. Specifically, the device acquires multiple groups of text data and their corresponding classification labels.

[0074] Step A200, performing vector conversion on the text training sample to obtain a text vector, performing position encoding on the text vector to obtain a position vector, and generating an encoded text vector based on the position vector;

[0075] In a specific implementation, the device converts text training samples into text vectors through an embedding layer. Specifically, the process usually involves converting words or characters into pre-trained vectors or vectors learned by the model. Secondly, in order to enable the model to understand the order of words in the text, the device adds position encoding to the text vector to generate a position vector. Specifically, the device combines the position information to generate the final encoded text vector, which will be input into the model. That is, the above process is to convert the text into a numerical form that the model can process.

[0076] Step A300, based on a plurality of preset self-attention layers, performing attention weighted calculation on the encoded text vector in parallel to obtain a weighted text vector;

[0077] In the specific implementation, the device processes the encoded text vectors in parallel through multiple self-attention layers. Each layer calculates the correlation between text sequence vectors to generate weighted text vectors. Specifically, the self-attention mechanism allows the model to dynamically focus on the importance of different parts of the text. These weighted text vectors contain rich contextual information.

[0078] In a specific implementation, the device performs attention weighted calculation on the encoded text vector in parallel based on a plurality of preset self-attention layers to obtain a step of weighted text vector, including:

[0079] Obtain the preset initial weight matrix of each attention layer; based on the first low-rank parameter matrix under the respective attention layer, perform low-dimensional mapping on the encoded text vector to obtain a reduced-dimensional text vector; based on the second low-rank parameter matrix under the respective attention layer, perform high-dimensional mapping on the reduced-dimensional text vector to obtain an increased-dimensional text vector, wherein the dimension of the increased-dimensional text vector is the same as the dimension of the encoded text vector; based on the initial weight matrix and the increased-dimensional text vector, calculate the target text vector of each attention layer, and fuse the target text vectors of the respective attention layers to obtain a weighted text vector.

[0080] It should be noted that the initial weight matrix is ​​a weight matrix randomly set when the model is initialized. In the self-attention layer, each layer has a preset initial weight matrix; a low-rank parameter matrix refers to a situation in mathematics and machine learning where the rank of a matrix is ​​less than the number of its rows and columns. The rank is the maximum number of linearly independent rows or columns in a matrix, which measures the "amount of information" or "complexity" of the matrix. If the rank of a matrix is ​​much smaller than its dimension, it means that there is a lot of redundant information in the matrix, and the original matrix can be approximated by a lower-dimensional representation, thereby reducing the complexity of the data.

[0081] In a specific implementation, in a deep learning model, a low-rank parameter matrix can reduce the number of parameters and computational complexity of the model while maintaining the generalization ability of the model. Specifically, the device uses the first low-rank parameter matrix under each attention layer (which can be regarded as a dimensionality reduction matrix) to perform a linear transformation on the encoded text vector and map it to a lower dimensional space. This process is called dimensionality reduction, and its purpose is to reduce the complexity of the data while trying to retain the most important information. Secondly, the second low-rank parameter matrix under each attention layer (which can be regarded as a dimensionality increase matrix) is used to map the reduced dimensionality text vector back to the original high-dimensional space. In this way, the dimensionality increased text vector has the same dimension as the encoded text vector.

[0082] Furthermore, the initial weight matrix and the dimension-enhanced text vector are combined, and the target text vector of each attention layer is obtained through the calculation of the self-attention layer. This process involves the calculation of the query (Q), key (K), and value (V), as well as the attention score and weighted sum based on these calculations. Finally, the device fuses the target text vectors of all self-attention layers. The fusion can be a simple concatenation or a more complex operation, such as weighted summation or using additional layers to integrate information.

[0083] In a specific implementation, the device performs low-dimensional mapping on the encoded text vector based on the first low-rank parameter matrix under the respective attention layers to obtain a reduced-dimensional text vector; and performs high-dimensional mapping on the reduced-dimensional text vector based on the second low-rank parameter matrix under the respective attention layers to obtain an increased-dimensional text vector, wherein the dimension of the increased-dimensional text vector is the same as the dimension of the encoded text vector. It is understandable that the above process describes the fine-tuning process of the model training process, specifically, the model is fine-tuned by adding a low-rank parameter matrix to the last layer of the model.

[0084] Further, refer to Figure 4 In the model training phase, the device inputs the vectorized text into the pre-trained model. Based on the pre-trained large model, the device freezes the pre-trained model weights (that is, the initial weight matrix), and injects the trainable rank decomposition matrix into each weight of the Transformer layer, which greatly reduces the number of trainable parameters for downstream tasks. That is, the device first uses a Linear layer A to reduce the data from d dimensions to r, and then uses a second Linear layer B to change the data from r back to d dimensions. Finally, the results of the left and right parts are added and fused to obtain the output hidden_state. Specifically, based on the initial weight matrix and the text vector after dimensionality increase, the device calculates the target text vector of each attention layer according to the following formula:

[0085] h=W0x+ΔWx=W0x+BAx

[0086] In the specific implementation, refer to Figure 4 The left side shows the weights of the pre-trained model, with input and output dimensions of d, which are frozen during training and do not accept gradient updates. The right side uses a random Gaussian initialization for A, B is zero at the beginning of training, r is the rank, and α / r is scaled for △Wx.

[0087] Step A400, based on the weighted text vector and the classification label, iteratively train the preset model to be trained to obtain a large language model that meets the accuracy requirements.

[0088] In a specific implementation, the device uses the weighted text vectors and the corresponding classification labels to iteratively train the preset model to be trained. The above process involves adjusting the weights of the model to reduce the prediction error. Secondly, the device iteratively trains until the performance of the model reaches a preset accuracy condition, such as a certain accuracy rate or performance on a validation set.

[0089] In a specific implementation, the device iteratively trains a preset model to be trained based on the weighted text vector and the classification label to obtain a large language model that meets the accuracy condition, including:

[0090] Based on the weighted text vector, label classification processing is performed through a preset model to be trained to obtain a predicted classification result; the predicted classification result and the classification label are subjected to error calculation to obtain an error result; it is determined whether the error result meets the error standard indicated by a preset error threshold range; if the error result does not meet the error standard indicated by the preset error threshold range, a target weight matrix is ​​calculated based on a preset gradient descent algorithm, and model parameters are updated based on the target weight matrix, and then based on the updated model to be trained, the step of returning to the preset multiple self-attention layers, and performing attention weighted calculation on the encoded text vector in parallel to obtain a weighted text vector is performed, and training is stopped until the error result meets the error standard indicated by the preset error threshold range to obtain a large language model, wherein different learning rate parameters are set for the first low-rank parameter matrix and the second low-rank parameter matrix in the gradient descent algorithm respectively.

[0091] In the specific implementation, the device first uses the weighted text vector obtained by the self-attention layer as input, and performs classification prediction through the model to be trained to obtain the predicted classification result. Secondly, the device compares the predicted classification result of the model with the actual classification label and calculates the error result. This is usually done through a loss function (such as cross entropy loss), which measures the difference between the model prediction and the actual label.

[0092] Further, the device checks whether the calculated error result is within a preset error threshold range. If the error is too large, that is, it does not meet the acceptable error standard, the model needs to be further trained. If the error result does not meet the preset error standard, the gradient descent algorithm is used to adjust the weights of the model. Specifically, this process involves calculating the gradient of the loss function with respect to the model parameters, and then updating the model parameters based on these gradients.

[0093] In the specific implementation, during the gradient descent process, two low-rank parameter matrices (the first low-rank parameter matrix and the second low-rank parameter matrix) are used to update the weights. These two matrices may represent different parts of the model (such as different self-attention layers) and different learning rate parameters can be set.

[0094] It should be emphasized that in order to control the model training process more finely, different learning rate parameters can be set for the first low-rank parameter matrix and the second low-rank parameter matrix. This means that in each iteration, the two matrices can be updated at different speeds according to their respective learning rates. Specifically, since the same learning rate will lead to poor feature learning effects, it is essentially because different features require different adjustment speeds during the gradient descent process. Usually, the operation of the A matrix is ​​more like a feature extraction or compression, and its gradient changes are relatively stable. The B matrix involves expanding low-dimensional information back to high-dimensional space, and the gradient changes are more complex and sensitive. Setting different learning rates, where B is set to θ times A (usually 2), can better adapt to this difference, so that both matrices can be updated with appropriate step sizes.

[0095] For example, the gradient descent algorithm before improvement: B = B-α×G B , A=A-α×G A ; Improved gradient descent algorithm: B = B-θα×G B , A=A-α×G A . Where: θ>>1, G B Represents the gradient.

[0096] In a specific implementation, the device iteratively trains a preset model to be trained based on the weighted text vector and the classification label to obtain a large language model that meets the accuracy condition, including:

[0097] Based on the weighted text vector and the classification label, determine the target weight matrix of each iteration cycle; perform low-precision quantization on the target weight matrix of each iteration cycle to obtain a quantized weight matrix; based on the quantized weight matrix, update the model to be trained in each iteration cycle until a large language model that meets the accuracy requirements is obtained.

[0098] In the specific implementation, in each iteration cycle (or training round) of model training, the loss function is calculated based on the weighted text vector (i.e., the text representation processed by the self-attention layer) and the true classification label, and the target weight matrix at the end of each iteration cycle is determined. The target weight matrix refers to the target value that the model parameters (weights) should be updated to in the current iteration cycle. This value is calculated based on the gradient of the loss function and the optimization algorithm (such as gradient descent).

[0099] In the specific implementation, the device performs low-precision quantization processing on the above target weight matrix, that is, converting the weight from a higher precision (such as a 32-bit floating point number) to a lower precision (such as an 8-bit integer). The purpose of quantization is to reduce the memory usage of the model and accelerate the reasoning process of the model while maintaining the model performance as much as possible.

[0100] Furthermore, the quantized weight matrix is ​​used to update the model to be trained in the current iteration cycle. This means that the quantized weights are applied to the model, replacing the original weights, and then training continues for the next iteration cycle. The above process is repeated, that is, the target weight matrix is ​​determined, quantized, and the model is updated in each iteration cycle until the performance of the model meets the preset accuracy conditions. These accuracy conditions include the accuracy of the model on the validation set, the loss value or other performance indicators reaching a certain threshold.

[0101] In the specific implementation, the above process describes the training and optimization process of a large language model, which involves the quantization and iterative update of weights until the model achieves satisfactory performance. This process combines the basic steps of model training and quantization technology, aiming to maintain model performance while reducing model resource consumption and improving model deployment efficiency.

[0102] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the label classification method of the long text of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0103] Based on the above first embodiment, the present application also proposes another embodiment, referring to Figure 5 , the label classification method of the long text includes:

[0104] In a specific implementation, the step of the device obtaining the text to be classified includes:

[0105] Step B100, obtaining an initial text and the number of characters in the initial text;

[0106] It should be noted that the initial text refers to the complete long text before label classification, and the number of characters is the number of characters in the initial text.

[0107] In a specific implementation, the device extracts text from a data source and counts the number of characters (including letters, numbers, spaces, etc.) contained in the text, that is, obtains original text data and calculates the number of characters.

[0108] Step B200, determining whether the number of characters is greater than a preset character threshold;

[0109] In a specific implementation, the device compares the number of characters in the initial text with a preset threshold to determine whether the text exceeds the character number range of the processing or classification model.

[0110] Step B300, if the number of characters is greater than the character threshold, segmenting the initial text to obtain segmented texts, and extracting semantic feature values ​​of the segmented texts;

[0111] In a specific implementation, if the number of characters is greater than the character threshold, the initial text is segmented to obtain segmented texts, wherein the text segmentation is specifically divided according to the paragraphs of the text, that is, the initial text is segmented into paragraphs.

[0112] In a specific implementation, the device extracts semantic feature values ​​for the segmented text of each paragraph. These feature values ​​can represent the main content or semantic information of the text segment, and can be obtained through word embedding, sentence embedding or a specific semantic extraction model.

[0113] Step B400, based on the semantic feature value, calculate the semantic difference between adjacent segmented texts, and based on the semantic difference, segment the initial text to obtain the text to be classified.

[0114] In a specific implementation, the device calculates the semantic difference between adjacent text segments by comparing their semantic feature values, for example, using a measurement method such as cosine similarity.

[0115] In a specific implementation, the device determines the optimal segmentation point based on the calculated semantic difference value to maximize the semantic difference between adjacent text segments, thereby obtaining semantically relatively independent text to be classified, wherein the semantic difference value measures the semantic similarity or difference between two adjacent text segments.

[0116] Specifically, the device segments the initial text based on the semantic difference value to obtain the step of obtaining the text to be classified, including:

[0117] Determine whether the semantic difference value is greater than a preset semantic difference value threshold; if the semantic difference value is greater than the preset semantic difference value threshold, segment the text corresponding to the semantic difference value into paragraphs to obtain at least two texts to be classified;

[0118] In the specific implementation, in order to decide whether the text needs to be further segmented, a preset semantic difference threshold is set. This threshold is a predefined standard used to determine whether the semantic difference between two text segments is large enough. If the calculated semantic difference is greater than this preset threshold, it means that the two text segments are semantically different enough to be considered independent parts. In this case, the text segments with larger semantic difference values ​​will be segmented, that is, a large text segment will be divided into at least two smaller text segments. Each smaller text segment after segmentation is regarded as an independent text to be classified, and these texts will be used for subsequent classification tasks.

[0119] It should be noted that the above description shows how to decide whether to segment a text based on semantic differences and preset thresholds, and how to obtain smaller text segments suitable for separate classification from the original long text. This method helps improve classification accuracy because it allows the model to focus on relatively semantically consistent text paragraphs instead of processing long texts containing multiple different topics or semantic information.

[0120] In a specific implementation, the step of segmenting the text corresponding to the semantic difference value to obtain at least two texts to be classified includes:

[0121] A character limit value of a preset large language model is obtained; based on the character limit value, the text corresponding to the semantic difference value is segmented into paragraphs to obtain at least two texts to be classified.

[0122] It should be noted that each large language model has a maximum input limit, which determines the maximum number of characters or tokens that the model can accept when processing text. For example, the maximum input limit of GPT-3 is 4096 tokens.

[0123] In the specific implementation, if the text to be processed exceeds the character limit of the model, it is necessary to split the text into multiple smaller parts, each of which does not exceed the processing capacity of the model. This is to ensure that the model can process the entire text, not just a part of the text.

[0124] In a specific implementation, the present application proposes that when performing paragraph segmentation, the character limit value of the large language model also needs to be considered. For example, the number of characters in the initial text exceeds the character threshold, and there are three paragraphs A, B, and C after paragraph division. The semantic difference A1 between AB and the semantic difference A2 between BC are both greater than the semantic difference threshold. If the paragraph between AB is segmented, two texts A and BC are obtained. In this case, if BC still exceeds the character limit value, BC needs to be segmented. If the paragraph between BC is segmented, two texts AB and C are obtained. If both texts meet the character limit value, the solution for segmenting the paragraph between BC is selected as the optimal text segmentation strategy.

[0125] This application also provides a long text label classification device, please refer to Figure 6 , the long text label classification device includes:

[0126] An acquisition module 10 is used to acquire the text to be classified;

[0127] The classification module 20 is used to perform label classification processing based on the text to be classified through a preset large language model to obtain a label classification result, wherein the model architecture of the large language model includes a multi-head self-attention mechanism and position encoding, the position encoding is used to encode the text position information into a corresponding position vector, and construct a text sequence vector based on the position vector, and the multi-head self-attention mechanism is to calculate the correlation between text sequence vectors in parallel through multiple self-attention layers.

[0128] Optionally, the long text label classification device further includes:

[0129] A sample acquisition module, used to acquire text training samples and classification labels of the text training samples;

[0130] An encoding module, used for performing vector conversion on the text training sample to obtain a text vector, performing position encoding on the text vector to obtain a position vector, and generating an encoded text vector based on the position vector;

[0131] A weighting module, configured to perform attention weighted calculation on the encoded text vector in parallel based on a plurality of preset self-attention layers to obtain a weighted text vector;

[0132] The training module is used to iteratively train the preset model to be trained based on the weighted text vector and the classification label to obtain a large language model that meets the accuracy requirements.

[0133] Optionally, the weighting module includes:

[0134] A weight matrix acquisition module is used to obtain the preset initial weight matrix of each attention layer;

[0135] A dimensionality reduction module, used for performing low-dimensional mapping on the encoded text vector based on the first low-rank parameter matrix under the respective attention layers to obtain a text vector after dimensionality reduction;

[0136] A dimension-increasing module, configured to perform high-dimensional mapping on the text vector after dimension reduction based on the second low-rank parameter matrix under the respective attention layers to obtain a text vector after dimension increase, wherein the dimension of the text vector after dimension increase is the same as the dimension of the encoded text vector;

[0137] A fusion module is used to calculate the target text vectors of the respective attention layers based on the initial weight matrix and the dimensionally upgraded text vector, and to fuse the target text vectors of the respective attention layers to obtain a weighted text vector.

[0138] Optionally, the training module includes:

[0139] A prediction module is used to perform label classification processing based on the weighted text vector through a preset model to be trained to obtain a prediction classification result;

[0140] An error calculation module is used to perform error calculation on the predicted classification result and the classification label to obtain an error result;

[0141] A judging module, used to judge whether the error result meets the error standard indicated by a preset error threshold range;

[0142] An iterative training module is used to calculate the target weight matrix based on the preset gradient descent algorithm if the error result does not meet the error standard indicated by the preset error threshold range, and update the model parameters based on the target weight matrix, and then return to the preset multiple self-attention layers based on the updated model to be trained, and perform attention weighted calculation on the encoded text vector in parallel to obtain the weighted text vector step, until the error result meets the error standard indicated by the preset error threshold range, and stop training to obtain a large language model, wherein different learning rate parameters are set for the first low-rank parameter matrix and the second low-rank parameter matrix in the gradient descent algorithm respectively.

[0143] Optionally, the training module further includes:

[0144] A determination module, used for determining a target weight matrix for each iteration cycle based on the weighted text vector and the classification label;

[0145] A quantization module, used for performing low-precision quantization on the target weight matrix of each iteration cycle to obtain a quantized weight matrix;

[0146] The iterative update module is used to update the model to be trained in each iteration cycle based on the quantized weight matrix until a large language model that meets the accuracy requirements is obtained.

[0147] Optionally, the acquisition module 10 includes:

[0148] A text acquisition module, used to acquire an initial text and the number of characters in the initial text;

[0149] A character number determination module, used to determine whether the character number is greater than a preset character threshold;

[0150] A segmentation module, configured to segment the initial text to obtain segmented texts if the number of characters is greater than the character threshold, and extract semantic feature values ​​of the segmented texts;

[0151] The segmentation module is used to calculate the semantic difference between adjacent segmented texts based on the semantic feature value, and segment the initial text based on the semantic difference to obtain the text to be classified.

[0152] The long text label classification device provided by the present application adopts the long text label classification method in the above embodiment, which can solve the technical problem of long text label classification. Compared with the prior art, the long text label classification device provided by the present application has the same beneficial effects as the long text label classification method provided by the above embodiment, and the other technical features in the long text label classification device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0153] The present application provides a tag classification device for long texts, the tag classification device for long texts comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the tag classification method for long texts in the above-mentioned embodiment 1.

[0154] Reference below Figure 7 , which shows a schematic diagram of the structure of a long text label classification device suitable for implementing the embodiment of the present application. The long text label classification device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The tag classification device for long text shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0155] like Figure 7As shown, the long text label classification device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the long text label classification device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the label classification device for long texts to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a label classification device for long texts with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0156] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0157] The long text label classification device provided by the present application adopts the long text label classification method in the above embodiment, which can solve the technical problem of long text label classification. Compared with the prior art, the long text label classification device provided by the present application has the same beneficial effects as the long text label classification method provided by the above embodiment, and the other technical features in the long text label classification device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0158] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0159] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0160] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the tag classification method for long texts in the above-mentioned embodiment.

[0161] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0162] The computer-readable storage medium may be included in the long text label classification device; or may exist independently without being assembled into the long text label classification device.

[0163] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the long text tag classification device, the long text tag classification device can classify the long text tags.

[0164] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0165] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0166] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0167] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned label classification method for long texts, and can solve the technical problem of label classification for long texts. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the label classification method for long texts provided by the above-mentioned embodiments, and will not be elaborated here.

[0168] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned long text tag classification method when executed by a processor.

[0169] The computer program product provided by this application can solve the technical problem of label classification of long texts. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the label classification method of long texts provided by the above embodiment, which will not be repeated here.

[0170] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A label classification method for long text, characterized in that: The method for label classification of the long text includes: Get the text to be classified; Based on the text to be classified, label classification processing is performed through a preset large language model to obtain a label classification result, wherein the model architecture of the large language model includes a multi-head self-attention mechanism and position encoding, the position encoding is used to encode text position information into a corresponding position vector, and construct a text sequence vector based on the position vector, and the multi-head self-attention mechanism is to calculate the correlation between text sequence vectors in parallel through multiple self-attention layers.

2. The tag classification method for long text according to claim 1, characterized in that: Before the step of performing label classification processing based on the text to be classified by using a preset large language model to obtain a label classification result, the method includes: Obtaining text training samples and classification labels of the text training samples; Performing vector conversion on the text training sample to obtain a text vector, performing position encoding on the text vector to obtain a position vector, and generating an encoded text vector based on the position vector; Based on a plurality of preset self-attention layers, performing attention weighted calculation on the encoded text vector in parallel to obtain a weighted text vector; Based on the weighted text vector and the classification label, the preset model to be trained is iteratively trained to obtain a large language model that meets the accuracy requirements.

3. The tag classification method for long text according to claim 2, characterized in that: The step of performing attention weighted calculation on the encoded text vector in parallel based on the preset multiple self-attention layers to obtain a weighted text vector includes: Get the preset initial weight matrix of each attention layer; Based on the first low-rank parameter matrix under the respective attention layers, low-dimensional mapping is performed on the encoded text vector to obtain a text vector after dimensionality reduction; Based on the second low-rank parameter matrix under the respective attention layers, high-dimensional mapping is performed on the reduced-dimensional text vector to obtain an increased-dimensional text vector, wherein the dimension of the increased-dimensional text vector is the same as the dimension of the encoded text vector; Based on the initial weight matrix and the dimension-upgraded text vector, the target text vectors of the respective attention layers are calculated, and the target text vectors of the respective attention layers are fused to obtain a weighted text vector.

4. The tag classification method for long text according to claim 3, characterized in that: The step of iteratively training the preset model to be trained based on the weighted text vector and the classification label to obtain a large language model that meets the accuracy condition includes: Based on the weighted text vector, label classification processing is performed through a preset model to be trained to obtain a predicted classification result; Performing error calculation on the predicted classification result and the classification label to obtain an error result; Determining whether the error result meets the error standard indicated by a preset error threshold range; If the error result does not meet the error standard indicated by the preset error threshold range, the target weight matrix is ​​calculated based on the preset gradient descent algorithm, and the model parameters are updated based on the target weight matrix, and then based on the updated model to be trained, the preset multiple self-attention layers are returned, and the encoded text vector is subjected to attention weighted calculation in parallel to obtain the weighted text vector step, and the training is stopped after the error result meets the error standard indicated by the preset error threshold range to obtain a large language model, wherein different learning rate parameters are set for the first low-rank parameter matrix and the second low-rank parameter matrix in the gradient descent algorithm respectively.

5. The tag classification method for long text according to claim 2, characterized in that: The step of iteratively training the preset model to be trained based on the weighted text vector and the classification label to obtain a large language model that meets the accuracy condition includes: Determining a target weight matrix for each iteration cycle based on the weighted text vector and the classification label; Performing low-precision quantization on the target weight matrix of each iteration cycle to obtain a quantized weight matrix; Based on the quantized weight matrix, the model to be trained in each iteration cycle is updated until a large language model that meets the accuracy requirements is obtained.

6. The tag classification method for long text according to claim 1, characterized in that: The step of obtaining the text to be classified includes: Get the initial text and the number of characters in the initial text; Determining whether the number of characters is greater than a preset character threshold; If the number of characters is greater than the character threshold, the initial text is segmented to obtain segmented texts, and semantic feature values ​​of the segmented texts are extracted; Based on the semantic feature values, the semantic difference values ​​between adjacent segmented texts are calculated, and based on the semantic difference values, the initial text is segmented to obtain the text to be classified.

7. A label classification device for long text, characterized in that: The device comprises: An acquisition module is used to obtain the text to be classified; The classification module is used to perform label classification processing based on the text to be classified through a preset large language model to obtain a label classification result, wherein the model architecture of the large language model includes a multi-head self-attention mechanism and position encoding, the position encoding is used to encode the text position information into a corresponding position vector, and construct a text sequence vector based on the position vector, and the multi-head self-attention mechanism is to calculate the correlation between text sequence vectors in parallel through multiple self-attention layers.

8. A long text label classification device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the long text tag classification method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the long text tag classification method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the long text tag classification method according to any one of claims 1 to 6 are implemented.

Images