Long text label classification method and device, equipment, storage medium and product

Abstract

The application discloses a long text label classification method and device, equipment, a storage medium and a product, and relates to the technical field of computers. The long text label classification method is disclosed, and the method comprises the following steps: acquiring a text to be classified; and performing label classification processing on the text to be classified by using a preset large language model to obtain a label classification result. In the application, the large language model converts the text to be classified into a text sequence vector by using position coding, and then uses a multi-head self-attention mechanism to perform parallel calculation on the correlation between the text sequence vectors, that is, the large language model can process the text sequence vectors in parallel, so as to improve the utilization rate of the display memory of the large language model. The improvement of the utilization rate of the display memory enables the large language model to process more characters of the long text, that is, the large language model does not need to segment the long text, so that the loss of semantics caused by segmentation is avoided, and the label classification accuracy of the long text is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a long text label classification method and device, equipment, a storage medium and a product. BACKGROUND

[0002] Natural language processing (NLP) is an important branch of computer science and artificial intelligence, which enables computers to understand and process human language. In current NLP applications, text classification is a basic and key task, and with the increasing demand of users, the label classification of long text often needs to be classified in multiple categories, for example, a news article involves multiple topics such as politics, economy and society. Compared with traditional single-label classification, multi-label classification requires the model to identify and process multiple related topics in the text.

[0003] In the related art, the self-recurrent language pre-training model XLNet is usually used for text multi-label classification. However, the XLNet model has a character limit of 512 characters, that is, when processing long text exceeding the character limit through the XLNet model, the long text needs to be segmented into multiple independent texts, and the text segmentation will cause semantic loss, thereby reducing the accuracy of label classification of long text.

[0004] The above content is only used to assist in understanding the technical solutions of the present application, and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0005] The main purpose of the present application is to provide a long text label classification method, which aims to solve the technical problem of low accuracy of long text label classification.

[0006] To achieve the above purpose, the present application provides a long text label classification method, which comprises:

[0007] obtaining a text to be classified;

[0008] based on the text to be classified, performing label classification processing through a preset large language model to obtain a label classification result, wherein the model architecture of the large language model comprises a multi-head self-attention mechanism and a position encoding, the position encoding is used to encode the text position information into a corresponding position vector, and a text sequence vector is constructed based on the position vector, and the multi-head self-attention mechanism is used to calculate the correlation between the text sequence vectors through multiple self-attention layers in parallel.

[0009] Optionally, before the step of performing label classification processing through a preset large language model based on the text to be classified to obtain a label classification result, the method comprises:

[0010] obtain a text training sample and a classification label of the text training sample;

[0011] perform vector conversion on the text training sample to obtain a text vector, perform position coding on the text vector to obtain a position vector, and generate an encoded text vector based on the position vector;

[0012] perform attention weighting calculation on the encoded text vector in parallel based on a plurality of preset self-attention layers to obtain a weighted text vector;

[0013] perform iterative training on a preset to-be-trained model based on the weighted text vector and the classification label to obtain a large language model that meets an accuracy condition.

[0014] Optionally, the step of performing attention weighting calculation on the encoded text vector in parallel based on a plurality of preset self-attention layers to obtain a weighted text vector comprises:

[0015] obtain an initial weight matrix of each self-attention layer;

[0016] perform low-dimensional mapping on the encoded text vector based on a first low-rank parameter matrix under each self-attention layer to obtain a reduced-dimensional text vector;

[0017] perform high-dimensional mapping on the reduced-dimensional text vector based on a second low-rank parameter matrix under each self-attention layer to obtain an increased-dimensional text vector, wherein the dimension of the increased-dimensional text vector is the same as that of the encoded text vector;

[0018] calculate a target text vector of each self-attention layer based on the initial weight matrix and the increased-dimensional text vector, and fuse the target text vectors of the self-attention layers to obtain the weighted text vector.

[0019] Optionally, the step of performing iterative training on a preset to-be-trained model based on the weighted text vector and the classification label to obtain a large language model that meets an accuracy condition comprises:

[0020] perform label classification processing on the weighted text vector through the preset to-be-trained model to obtain a predicted classification result;

[0021] perform error calculation on the predicted classification result and the classification label to obtain an error result;

[0022] determine whether the error result meets an error standard indicated by a preset error threshold range;

[0023] If the error result does not satisfy the error standard indicated by the preset error threshold range, a target weight matrix is calculated based on a preset gradient descent algorithm, the model parameters are updated based on the target weight matrix, and based on the updated to-be-trained model, the step of performing attention weighting calculation on the encoded text vector in parallel based on the preset multiple self-attention layers is returned until the error result satisfies the error standard indicated by the preset error threshold range, the training is stopped, and a large language model is obtained. In the gradient descent algorithm, the first low-rank parameter matrix and the second low-rank parameter matrix are respectively set with different learning rate parameters.

[0024] Optionally, the step of iteratively training a preset to-be-trained model based on the weighted text vector and the classification label to obtain a large language model satisfying an accuracy condition comprises:

[0025] Based on the weighted text vector and the classification label, a target weight matrix of each iteration period is determined.

[0026] The target weight matrix of each iteration period is quantized at low precision to obtain a quantized weight matrix.

[0027] Based on the quantized weight matrix, the to-be-trained model of each iteration period is updated until a large language model satisfying an accuracy condition is obtained.

[0028] Optionally, the step of obtaining the text to be classified comprises:

[0029] Obtaining an initial text and a character number of the initial text.

[0030] Determining whether the character number is greater than a preset character threshold.

[0031] If the character number is greater than the character threshold, the initial text is segmented to obtain segmented texts, and semantic feature values of the segmented texts are extracted.

[0032] Based on the semantic feature values, a semantic difference value between adjacent segmented texts is calculated, and the initial text is segmented based on the semantic difference value to obtain the text to be classified.

[0033] In addition, to achieve the above-mentioned purpose, the application further provides a long text label classification device, which comprises:

[0034] An obtaining module is configured to obtain text to be classified.

[0035] The classification module is configured to perform label classification processing on the text to be classified by using a preset large language model, so as to obtain a label classification result, wherein a model architecture of the large language model comprises a multi-head self-attention mechanism and position encoding, the position encoding is configured to encode text position information into a corresponding position vector, and construct a text sequence vector based on the position vector, and the multi-head self-attention mechanism is configured to calculate the correlation between the text sequence vectors in parallel through multiple self-attention layers.

[0036] In addition, to achieve the above object, the application further provides a long text label classification device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the long text label classification method as described above.

[0037] In addition, to achieve the above object, the application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the long text label classification method as described above.

[0038] In addition, to achieve the above object, the application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the long text label classification method as described above.

[0039] The one or more technical solutions provided by the application have at least the following technical effects:

[0040] Compared with the related art, a long text is usually classified by using an autoregressive language pre-training model XLNet. However, the XLNet model has a character limit of 512 characters, that is, when the XLNet model processes a long text exceeding the character limit, the long text needs to be segmented into multiple independent texts, and the text segmentation will cause semantic loss, thereby reducing the label classification accuracy of the long text. Compared with the related art, the long text is obtained, and based on the long text, a label classification processing is performed by using a preset large language model to obtain a label classification result. The model architecture of the large language model includes a multi-head self-attention mechanism and position encoding. The position encoding is used to encode the text position information into a corresponding position vector, and a text sequence vector is constructed based on the position vector. The multi-head self-attention mechanism is used to calculate the correlation between the text sequence vectors in parallel. It can be understood that the large language model in the present application converts the long text to be classified into a text sequence vector by using the position encoding, and then calculates the correlation between the text sequence vectors in parallel by using the multi-head self-attention mechanism, that is, the large language model can process the text sequence vectors in parallel, thereby improving the memory utilization of the large language model. The improvement of the memory utilization can enable the large language model to process more characters of the long text, that is, the large language model does not need to segment the long text, thereby avoiding the semantic loss caused by the segmentation, and improving the label classification accuracy of the long text. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate an embodiment consistent with the present application and, together with the description, serve to explain the principles of the application.

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, those skilled in the art can obtain other drawings from these drawings without any creative labor.

[0043] Figure 1 The flowchart provided for the first embodiment of the long text label classification method of the present application;

[0044] Figure 2 The overall flowchart of the large language model training and inference of the long text label classification method of the present application;

[0045] Figure 3 The flowchart provided for the second embodiment of the long text label classification method of the present application;

[0046] Figure 4 The model fine-tuning flowchart of the long text label classification method of the present application;

[0047] Figure 5 A flowchart is provided for the third embodiment of the label classification method of the long text of the present application;

[0048] Figure 6 A module structure diagram is provided for the label classification device of the long text of the embodiment of the present application;

[0049] Figure 7 A device structure diagram of the hardware running environment involved in the label classification method of the long text in the embodiment of the present application.

[0050] The purpose of the present application, the functional characteristics and advantages will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and do not limit the present application.

[0052] In order to better understand the technical solutions of the present application, the following will be described in detail in conjunction with the drawings and specific embodiments of the specification.

[0053] The main solution of the embodiment of the present application is: obtaining a text to be classified; based on the text to be classified, a label classification processing is performed through a pre-set large language model to obtain a label classification result, wherein the model architecture of the large language model includes a multi-head self-attention mechanism and a position encoding, the position encoding is used to encode the text position information into a corresponding position vector, and a text sequence vector is constructed based on the position vector, and the multi-head self-attention mechanism is used to calculate the correlation between the text sequence vectors through multiple self-attention layers in parallel.

[0054] In the present embodiment, the label classification device of the long text is taken as the execution subject, and for the convenience of description, it is referred to as "device" in the following.

[0055] In the related art, the self-recurrent language pre-training model XLNet is usually used for text multi-label classification. However, the XLNet model has a character limit of 512 characters, that is, when processing long text exceeding the character limit through the XLNet model, the long text needs to be segmented into multiple independent texts, and the text segmentation will cause semantic loss, thereby reducing the accuracy of the label classification of the long text.

[0056] The present application provides a solution to realize the label classification of long text and improve the accuracy of the label classification of long text.

[0057] From the above embodiments, the large language model in the present application converts the text to be classified into a text sequence vector through position encoding, and then uses the multi-head self-attention mechanism to calculate the correlation between the text sequence vectors in parallel, that is, the large language model can process the text sequence vector in parallel, thereby improving the memory utilization rate of the large language model. The improvement of the memory utilization rate can enable the large language model to process more characters of long text, that is, the large language model does not need to segment the long text, thereby avoiding the loss of semantics caused by segmentation, and further improving the label classification accuracy of the long text.

[0058] It should be noted that the execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a terminal system, etc. capable of realizing the above functions. The following will take the long text label classification device as an example to illustrate the present embodiment and each of the following embodiments.

[0059] Based on this, the present application provides a long text label classification method, which is described with reference to Figure 1 , Figure 1 The flowchart of the first embodiment of the long text label classification method of the present application is shown in the figure.

[0060] In the present embodiment, the long text label classification method comprises steps S100-S200:

[0061] Step S100, obtaining text to be classified;

[0062] It should be noted that the text to be classified refers to text data that has not been assigned a category label, which needs to be determined by a classification model which category or label it belongs to. In natural language processing (NLP) and machine learning, the text to be classified is the input object of the classification task.

[0063] In specific implementation, the device can extract from different article sources under the premise of obtaining user authorization, or receive user-uploaded text to be classified, without specific limitation here.

[0064] Step S200, based on the text to be classified, performing label classification processing through a preset large language model to obtain a label classification result, wherein the model architecture of the large language model comprises a multi-head self-attention mechanism and position encoding, the position encoding is used to encode text position information into a corresponding position vector, and a text sequence vector is constructed based on the position vector, and the multi-head self-attention mechanism is used to calculate the correlation between the text sequence vectors in parallel through multiple self-attention layers.

[0065] It should be noted that the large language model in the present application is described with reference to Figure 2, the large language model refers to a language model comprising a large number of parameters, which are trained on a large amount of text data, for example, the model GPT-3, PaLM, Galactica and LLaMA. Preferably, the large language model is an LLM model, specifically, the LLM is built on the Transformer architecture, in which multi-head attention layers are stacked in a very deep neural network. Specifically, the LLM mainly adopts a similar model architecture (i.e., Transformer) and pre-training target (i.e., language modeling) as the small language model. The LLM greatly expands the model size, pre-training data and total computation (scale-up factor), which can better understand natural language and generate high-quality text according to the given context (e.g., prompt).

[0066] In a specific implementation, the model architecture of the large language model includes a multi-head self-attention mechanism and position encoding. Specifically, the LLM is mainly composed of a Transformer architecture, and the structure of this model includes a multi-layer Transformer decoder, each layer of which is composed of a self-attention mechanism and a feed-forward neural network. In each level of the Transformer model, the self-attention mechanism first quantitatively evaluates the interaction between each word in the input sequence and other words in the sequence. Then, according to these evaluation results, the appropriate weight is applied to each word in the input sequence, thereby enhancing the ability of the large language model to recognize and capture long-distance dependency features within the text. Next, the feed-forward neural network further nonlinearly transforms the output of the self-attention. In addition, the LLM can also use position encoding to process the order information of the words in the sequence, because the Transformer itself cannot process such order information. The position encoding encodes the information of each position into a vector and adds it to the word vector at the corresponding position. Finally, the LLM can also use masked self-attention, which allows the large language model to only use the words before the next word when predicting the next word, but not the words after the next word, thereby enabling the large model to simulate the natural process of human word-by-word generation of text from left to right.

[0067] In a specific implementation, the application also proposes to use vLLM to accelerate inference in the process of label classification processing by a preset large language model. The core mechanism of vLLM is PagedAttention, which is a paging management technology for key-value cache (KV cache) space. PagedAttention solves this problem by dividing the KV cache of each sequence into multiple fixed-length blocks. Each block contains a certain number of tokens, and when calculating self-attention, these blocks can be efficiently located and accessed. Understandably, each fixed-length block can be compared to a "page" in virtual memory, and each token is similar to a byte, and the sequence can be regarded as a "process" in the operating system. By using a block table, continuous logical blocks can be mapped to non-continuous physical blocks. These physical blocks can be allocated on demand according to newly generated tokens, effectively replacing continuous video memory allocation and improving the utilization efficiency of video memory.

[0068] In related technologies, an autoregressive language pre-training model XLNet is usually used for text multi-label classification. However, the XLNet model has a character limit of 512 characters, that is, when processing long text exceeding the character limit through the XLNet model, the long text needs to be divided into multiple independent texts, and text segmentation will cause semantic loss, thereby reducing the label classification accuracy of long text. Compared with the related art, the application obtains a text to be classified; based on the text to be classified, a label classification processing is performed through a preset large language model to obtain a label classification result, wherein the model architecture of the large language model includes a multi-head self-attention mechanism and a position encoding, the position encoding is used to encode text position information into a corresponding position vector, and a text sequence vector is constructed based on the position vector, and the multi-head self-attention mechanism is used to calculate the correlation between text sequence vectors in parallel. Understandably, in the application, the large language model converts the text to be classified into a text sequence vector through the position encoding, and then utilizes the multi-head self-attention mechanism to calculate the correlation between the text sequence vectors in parallel, that is, the large language model can process the text sequence vectors in parallel, thereby improving the utilization rate of the video memory of the large language model, and the improvement of the utilization rate of the video memory enables the large language model to process more characters of long text, that is, the large language model does not need to segment the long text, thereby avoiding the semantic loss caused by segmentation, and thereby improving the label classification accuracy of long text.

[0069] Based on the above first embodiment, the application also proposes another embodiment, referring to Figure 3 , the label classification method of the long text comprises:

[0070] In a specific implementation, before the step of classifying the text based on the text to be classified by a preset large language model to obtain a label classification result, the method comprises:

[0071] Step A100, obtaining a text training sample and a classification label of the text training sample;

[0072] It should be noted that the text training sample is a sample to be learned and understood by the model, and the classification label is the category to which the text belongs, which is used to guide the model to learn how to classify new and unseen texts.

[0073] In a specific implementation, the purpose of the device obtaining the text training sample and the classification label of the text training sample is to collect data for training the model. Specifically, the device obtains multiple sets of text data and their corresponding classification labels.

[0074] Step A200, converting the text training sample into a vector to obtain a text vector, performing position encoding on the text vector to obtain a position vector, and generating an encoded text vector based on the position vector;

[0075] In a specific implementation, the device converts the text training sample into a text vector through an embedding layer. Specifically, this process usually involves converting words or characters into pre-trained vectors or vectors learned by the model. Secondly, in order to enable the model to understand the order of words in the text, the device adds position encoding to the text vector to generate a position vector. Specifically, the device generates the final encoded text vector in combination with the position information, which will be input into the model. That is, the above process is to convert the text into a numerical form that can be processed by the model.

[0076] Step A300, performing attention weighting calculation on the encoded text vector in parallel based on a plurality of preset self-attention layers to obtain a weighted text vector;

[0077] In a specific implementation, the device processes the encoded text vector in parallel through a plurality of self-attention layers, each layer calculating the correlation between text sequence vectors to generate a weighted text vector. Specifically, the self-attention mechanism allows the model to dynamically focus on the importance of different parts of the text, and these weighted text vectors contain rich context information.

[0078] In a specific implementation, the step of performing attention weighting calculation on the encoded text vector in parallel based on a plurality of preset self-attention layers to obtain a weighted text vector comprises:

[0079] obtain a dimension-reduced text vector by performing low-dimensional mapping on the encoded text vector based on a first low-rank parameter matrix under the respective self-attention layer; obtain a dimension-increased text vector by performing high-dimensional mapping on the dimension-reduced text vector based on a second low-rank parameter matrix under the respective self-attention layer, wherein a dimension of the dimension-increased text vector is the same as a dimension of the encoded text vector; calculate a target text vector of the respective self-attention layer based on the initial weight matrix and the dimension-increased text vector, and fuse the target text vectors of the respective self-attention layers to obtain a weighted text vector.

[0080] It should be noted that the initial weight matrix is a weight matrix randomly set when the model is initialized, and each layer in the self-attention layer has a preset initial weight matrix; the low-rank parameter matrix refers to a case where the rank of a matrix is less than the number of rows and columns in the mathematical and machine learning fields. The rank is the maximum number of linearly independent rows or columns in the matrix, which measures the "amount of information" or "complexity" of the matrix. If the rank of a matrix is much smaller than its dimension, it means that there is a lot of redundant information in the matrix, which can be approximated by a lower-dimensional representation, thereby reducing the complexity of the data.

[0081] In a specific implementation, in a deep learning model, the low-rank parameter matrix can reduce the number of parameters and the computational complexity of the model while maintaining the generalization ability of the model. Specifically, the device uses the first low-rank parameter matrix (which can be regarded as a dimension-reduction matrix) under the respective self-attention layer to perform linear transformation on the encoded text vector, and map it to a lower-dimensional space. This process is called dimension reduction, and the purpose is to reduce the complexity of the data while trying to preserve the most important information. Secondly, the second low-rank parameter matrix (which can be regarded as a dimension-increase matrix) under the respective self-attention layer is used to map the dimension-reduced text vector back to the original high-dimensional space. In this way, the dimension-increased text vector has the same dimension as the encoded text vector.

[0082] Further, in combination with the initial weight matrix and the dimension-increased text vector, the target text vector of the respective self-attention layer is obtained through the calculation of the self-attention layer. This process involves the calculation of query (Q), key (K) and value (V), as well as the attention score and weighted sum based on these calculations. Finally, the device fuses the target text vectors of all self-attention layers. The fusion can be a simple concatenation, or a more complex operation such as weighted sum or using additional layers to integrate information.

[0083] In a specific implementation, the apparatus performs low-dimensional mapping on the encoded text vector based on the first low-rank parameter matrix under each attention layer to obtain a reduced dimension text vector; and performs high-dimensional mapping on the reduced dimension text vector based on the second low-rank parameter matrix under each attention layer to obtain a dimension-increased text vector, wherein the dimension of the dimension-increased text vector is the same as that of the encoded text vector. It can be understood that the above process describes a fine-tuning process of the model training process. Specifically, the model is fine-tuned by adding a low-rank parameter matrix to the last layer of the model.

[0084] Further, with reference to Figure 4 In the model training stage, the apparatus inputs the vectorized text into the pre-trained model that has been trained, and based on the pre-trained large model, the apparatus freezes the pre-trained model weight (i.e., the initial weight matrix) and injects the trainable rank decomposition matrix into each weight of the Transformer layer, greatly reducing the number of trainable parameters of the downstream task. That is, the apparatus first uses a Linear layer A to reduce the data from d dimensions to r, and then uses a second Linear layer B to change the data from r back to d dimensions. Finally, the results of the left and right parts are added and fused to obtain the output hidden_state. Specifically, the apparatus calculates the target text vector of each attention layer based on the initial weight matrix and the dimension-increased text vector according to the following formula:

[0085] h = W0x + AWx = W0x + BAx

[0086] In a specific implementation, with reference to Figure 4 The left is the weight of the pre-trained model, and the input and output dimensions are both d, which is frozen during training and does not accept gradient updates. The right part uses random Gaussian initialization for A, and B is zero at the beginning of training, r is the rank, which will scale AWx by a / r.

[0087] Step A400, based on the weighted text vector and the classification label, the pre-set to-be-trained model is iteratively trained to obtain a large language model that meets the accuracy condition.

[0088] In a specific implementation, the apparatus uses the weighted text vector and the corresponding classification label to iteratively train the pre-set to-be-trained model. The above process involves adjusting the weights of the model to reduce the prediction error. Secondly, the apparatus iteratively trains until the performance of the model reaches the pre-set accuracy condition, such as a certain accuracy or performance on the validation set.

[0089] In a specific implementation, the apparatus iteratively trains the pre-set to-be-trained model based on the weighted text vector and the classification label to obtain a large language model that meets the accuracy condition, including:

[0090] Based on the weighted text vector, a label classification process is performed through a preset to-be-trained model to obtain a predicted classification result; an error calculation is performed on the predicted classification result and the classification label to obtain an error result; it is judged whether the error result meets an error standard indicated by a preset error threshold range; if the error result does not meet the error standard indicated by the preset error threshold range, a target weight matrix is calculated based on a preset gradient descent algorithm, model parameters are updated based on the target weight matrix, and the step of performing attention weighting calculation on the encoded text vector based on the preset multiple self-attention layers in parallel to obtain a weighted text vector is returned until the error result meets the error standard indicated by the preset error threshold range to stop training, thereby obtaining a large language model, wherein different learning rate parameters are set for the first low-rank parameter matrix and the second low-rank parameter matrix in the gradient descent algorithm.

[0091] In a specific implementation, first, the device uses the weighted text vector obtained through the self-attention layer processing as input, performs classification prediction through the to-be-trained model, and obtains a predicted classification result. Second, the device compares the predicted classification result of the model with the real classification label, and calculates an error result. This is usually done through a loss function (such as cross-entropy loss), which measures the difference between the model prediction and the actual label.

[0092] Further, the device checks whether the calculated error result is within a preset error threshold range. If the error is too large, i.e., it does not meet the acceptable error standard, the model needs to be further trained. If the error result does not meet the preset error standard, a gradient descent algorithm is used to adjust the weights of the model. Specifically, this process involves calculating the gradient of the loss function with respect to the model parameters, and then updating the model parameters according to these gradients.

[0093] In a specific implementation, in the process of gradient descent, two low-rank parameter matrices (the first low-rank parameter matrix and the second low-rank parameter matrix) are used to update the weights. These two matrices may represent different parts of the model (such as different self-attention layers), and different learning rate parameters can be set for them.

[0094] It is emphasized that, in order to more finely control the model training process, the first low-rank parameter matrix and the second low-rank parameter matrix can be set with different learning rate parameters. This means that in each iteration, the two matrices can be updated at different speeds according to the respective learning rates. Specifically, since the same learning rate will lead to poor feature learning results, essentially because in the gradient descent process, different features need different adjustment speeds, and generally, the operation of the A matrix is more like feature extraction or compression, and its gradient change is relatively stable. The B matrix involves expanding low-dimensional information back to high-dimensional space, and the gradient change is more complex and sensitive. Setting different learning rates, where B is set to θ times (usually 2) of A, can better adapt to this difference, allowing both matrices to be updated at appropriate steps.

[0095] For example, the gradient descent algorithm before improvement: B = B - a x G B , A = A - a x G A ; the improved gradient descent algorithm: B = B - θa x G B , A = A - a x G A . Where: θ >> 1, G B represents the gradient.

[0096] In a specific implementation, the step of the device performing iterative training on the preset to-be-trained model based on the weighted text vector and the classification label to obtain a large language model meeting an accuracy condition comprises:

[0097] Based on the weighted text vector and the classification label, determine the target weight matrix of each iteration period; quantize the target weight matrix of each iteration period at low precision to obtain a quantized weight matrix; update the to-be-trained model of each iteration period based on the quantized weight matrix until a large language model meeting an accuracy condition is obtained.

[0098] In a specific implementation, in each iteration period (or training round) of model training, based on the weighted text vector (i.e., the text representation processed by the self-attention layer) and the real classification label, the loss function is calculated, and the target weight matrix at the end of each iteration period is determined. The target weight matrix refers to the target value to which the model parameters (weights) should be updated in the current iteration period, which is calculated according to the gradient of the loss function and the optimization algorithm (such as gradient descent).

[0099] In a specific implementation, the device performs low-precision quantization processing on the above target weight matrix, i.e., converts the weights from a higher precision (such as 32-bit floating-point numbers) to a lower precision (such as 8-bit integers). The purpose of quantization is to reduce the memory occupancy of the model and speed up the inference process of the model, while trying to maintain the performance of the model.

[0100] Further, the quantized weight matrix is used to update the model to be trained in the current iteration cycle. This means that the quantized weights are applied to the model, replacing the original weights, and then the training of the next iteration cycle is continued. The above process is repeated, i.e., the target weight matrix is determined in each iteration cycle, quantization is performed, and the model is updated, until the performance of the model meets the preset accuracy conditions. These accuracy conditions include that the accuracy, loss value or other performance indicators of the model on the validation set reach a certain threshold.

[0101] In a specific implementation, the above process describes the training and optimization process of a large language model, which involves weight quantization and iterative updating until the model reaches satisfactory performance. This process combines the basic steps of model training and quantization techniques to maintain model performance while reducing resource consumption and improving model deployment efficiency.

[0102] It should be noted that the above examples are only for understanding the present application and do not constitute a limitation on the long text tag classification method of the present application. Further simple transformations based on this technical concept are within the scope of protection of the present application.

[0103] Based on the above first embodiment, the present application further proposes another embodiment, which refers to Figure 5 The long text tag classification method comprises:

[0104] In a specific implementation, the step of obtaining the text to be classified by the device comprises:

[0105] Step B100, obtaining an initial text and the number of characters of the initial text;

[0106] It should be noted that the initial text refers to the complete long text before label classification, and the number of characters refers to the number of characters of the initial text.

[0107] In a specific implementation, the device extracts the text from the data source and counts the number of characters (including letters, numbers, spaces, etc.) contained in the text, i.e., obtains the original text data and calculates the number of characters.

[0108] Step B200, determining whether the number of characters is greater than a preset character threshold;

[0109] In a specific implementation, the device compares the number of characters of the initial text with the threshold set in advance to determine whether the text exceeds the character number range of the processing or classification model.

[0110] Step B300, if the number of characters is greater than the character threshold, segmenting the initial text to obtain segmented texts, and extracting semantic feature values of the segmented texts;

[0111] In a specific implementation, if the number of characters is greater than the character threshold, the initial text is segmented to obtain segmented texts, where the text segmentation is specifically performed according to the paragraphs of the text, that is, the initial text is segmented according to the paragraphs.

[0112] In a specific implementation, the device extracts semantic feature values for each paragraph of the segmented text, which can represent the main content or semantic information of the text paragraph, and can be obtained through word embedding, sentence embedding or a specific semantic extraction model.

[0113] Step B400, based on the semantic feature values, calculate the semantic difference between adjacent segmented texts, and based on the semantic difference, segment the initial text to obtain the text to be classified.

[0114] In a specific implementation, the device calculates the semantic difference between adjacent text paragraphs, which is specifically implemented by comparing their semantic feature values, for example, using a cosine similarity or other measurement method.

[0115] In a specific implementation, the device determines the optimal segmentation point according to the calculated semantic difference, so that the semantic difference between adjacent text paragraphs is maximized, thereby obtaining the text to be classified which is relatively independent in semantics, where the semantic difference measures the similarity or difference degree of the semantic difference between two adjacent text segments.

[0116] Specifically, the step of segmenting the initial text based on the semantic difference to obtain the text to be classified includes:

[0117] determining whether the semantic difference is greater than a preset semantic difference threshold; if the semantic difference is greater than the preset semantic difference threshold, the text corresponding to the semantic difference is segmented by paragraphs to obtain at least two texts to be classified;

[0118] In a specific implementation, in order to decide whether the text needs to be further segmented, a preset semantic difference threshold is set. This threshold is a predefined standard for judging whether the semantic difference between two text segments is large enough. If the calculated semantic difference is greater than this preset threshold, it means that the two text segments have enough semantic difference and can be considered as independent parts. In this case, the text segment with a larger semantic difference is segmented, that is, a large text segment is divided into at least two smaller text segments. Each smaller text segment after segmentation is regarded as an independent text to be classified, which will be used for subsequent classification tasks.

[0119] It should be noted that the above description illustrates how to determine whether to perform paragraph segmentation on the text based on the semantic difference value and the preset threshold value, and how to obtain smaller text segments suitable for individual classification from the original long text. This method helps improve the accuracy of classification, as it allows the model to focus on text passages that are relatively consistent in semantics, rather than processing long texts that contain multiple different topics or semantic information.

[0120] In a specific implementation, the step of performing paragraph segmentation on the text corresponding to the semantic difference value to obtain at least two texts to be classified includes:

[0121] obtaining a character limit value of a preset large language model; and performing paragraph segmentation on the text corresponding to the semantic difference value based on the character limit value to obtain at least two texts to be classified.

[0122] It should be noted that each large language model has a maximum input limit, which determines the maximum number of characters or tokens that the model can accept when processing text. For example, the maximum input limit of GPT-3 is 4096 tokens.

[0123] In a specific implementation, if the text to be processed exceeds the character limit value of the model, the text needs to be segmented into multiple smaller parts, each of which does not exceed the processing capacity of the model. This is to ensure that the model can process the entire text, rather than only processing part of the text.

[0124] In a specific implementation, the present application proposes that when performing paragraph segmentation, the character limit value of the large language model also needs to be considered. For example, if the number of characters in the initial text exceeds the character threshold value, and there are three paragraphs A, B, and C after paragraph segmentation, the semantic difference value A1 between AB and the semantic difference value A2 between BC are both greater than the semantic difference value threshold value. If the paragraph between AB is segmented, i.e., A and BC are obtained, and BC still exceeds the character limit value, BC needs to be segmented. If the paragraph between BC is segmented, i.e., AB and C are obtained, and both of them satisfy the character limit value, then the scheme of segmenting the paragraph between BC is selected as the optimal text segmentation strategy.

[0125] The present application also provides a label classification device for long text, which is described in detail in Figure 6 The label classification device for long text includes:

[0126] The obtaining module 10 is configured to obtain a text to be classified.

[0127] The classification module 20 is configured to perform label classification processing on the text to be classified based on a preset large language model to obtain a label classification result. The model architecture of the large language model includes a multi-head self-attention mechanism and position encoding. The position encoding is used to encode text position information into a corresponding position vector, and a text sequence vector is constructed based on the position vector. The multi-head self-attention mechanism is used to calculate the correlation between text sequence vectors in parallel through multiple self-attention layers.

[0128] Optionally, the label classification device for long text further includes:

[0129] The sample acquisition module is configured to acquire a text training sample and a classification label of the text training sample.

[0130] The encoding module is configured to perform vector conversion on the text training sample to obtain a text vector, perform position encoding on the text vector to obtain a position vector, and generate an encoded text vector based on the position vector.

[0131] The weighting module is configured to perform attention weighting calculation on the encoded text vector in parallel based on a preset plurality of self-attention layers to obtain a weighted text vector.

[0132] The training module is configured to perform iterative training on a preset to-be-trained model based on the weighted text vector and the classification label to obtain a large language model that meets an accuracy condition.

[0133] Optionally, the weighting module includes:

[0134] The weight matrix acquisition module is configured to acquire an initial weight matrix of each self-attention layer.

[0135] The dimension reduction module is configured to perform low-dimensional mapping on the encoded text vector based on a first low-rank parameter matrix under each self-attention layer to obtain a dimension-reduced text vector.

[0136] The dimension increasing module is configured to perform high-dimensional mapping on the dimension-reduced text vector based on a second low-rank parameter matrix under each self-attention layer to obtain a dimension-increased text vector. The dimension of the dimension-increased text vector is the same as that of the encoded text vector.

[0137] The fusion module is configured to calculate a target text vector of each self-attention layer based on the initial weight matrix and the dimension-increased text vector, and fuse the target text vectors of the self-attention layers to obtain the weighted text vector.

[0138] Optionally, the training module includes:

[0139] a prediction module configured to perform label classification processing on the weighted text vector based on a preset to-be-trained model to obtain a prediction classification result;

[0140] an error calculation module configured to perform error calculation on the prediction classification result and the classification label to obtain an error result;

[0141] a judgment module configured to judge whether the error result meets an error standard indicated by a preset error threshold range;

[0142] an iterative training module configured to, if the error result does not meet the error standard indicated by the preset error threshold range, calculate a target weight matrix based on a preset gradient descent algorithm, update model parameters based on the target weight matrix, and return the step of performing attention weighting calculation on the encoded text vector in parallel based on the updated to-be-trained model based on the preset plurality of self-attention layers to obtain a weighted text vector until the error result meets the error standard indicated by the preset error threshold range to stop training and obtain a large language model, wherein the first low-rank parameter matrix and the second low-rank parameter matrix in the gradient descent algorithm are respectively set with different learning rate parameters.

[0143] Optionally, the training module further comprises:

[0144] a determination module configured to determine a target weight matrix of each iteration period based on the weighted text vector and the classification label;

[0145] a quantization module configured to perform low-precision quantization on the target weight matrix of each iteration period to obtain a quantized weight matrix;

[0146] an iterative update module configured to update the to-be-trained model of each iteration period based on the quantized weight matrix until a large language model meeting an accuracy condition is obtained.

[0147] Optionally, the obtaining module 10 comprises:

[0148] a text obtaining module configured to obtain an initial text and a character number of the initial text;

[0149] a character number judgment module configured to judge whether the character number is greater than a preset character threshold;

[0150] a segmentation module configured to, if the character number is greater than the character threshold, segment the initial text to obtain segmented texts and extract semantic feature values of the segmented texts;

[0151] The segmentation module is configured to calculate semantic difference values between adjacent segmented texts based on the semantic feature values, and segment the initial text based on the semantic difference values to obtain a text to be classified.

[0152] The long text label classification device provided in the present application adopts the long text label classification method in the above embodiments, and can solve the technical problem of long text label classification. Compared with the prior art, the long text label classification device provided in the present application has the same beneficial effects as the long text label classification method provided in the above embodiments, and other technical features in the long text label classification device are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0153] The present application provides a long text label classification device, which comprises at least one processor and a memory connected with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the long text label classification method in the above embodiment one.

[0154] Reference will now be made to the drawings, in which Figure 7 which shows a structural schematic diagram of a long text label classification device suitable for implementing the embodiments of the present application. The long text label classification device in the embodiments of the present application can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 7 The long text label classification device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0155] As Figure 7As shown, the long text label classification device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the long text label classification device to operate are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the long text label classification device to communicate with other devices wirelessly or by wire to exchange data. Although the long text label classification device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.

[0156] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0157] The long text label classification device provided by the present disclosure adopts the long text label classification method in the above-mentioned embodiments, and can solve the technical problem of long text label classification. Compared with the prior art, the long text label classification device provided by the present disclosure has the same beneficial effects as the long text label classification method provided by the above-mentioned embodiments, and other technical features in the long text label classification device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0158] It should be understood that portions of the application disclosed can be implemented in hardware, software, firmware, or combinations thereof. In the description of the embodiments above, specific features, structures, materials or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0159] The above description is merely illustrative of the application and is not intended to limit the scope of the application. Any changes or modifications that can be made to the application in accordance with the principles of the application would be readily apparent to persons skilled in the art and the generic principles of the application defined herein are intended to include all such changes and modifications. The scope of the application is therefore intended to be defined only by the scope of the claims presented and equivalents thereof.

[0160] The application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e., a computer program) for performing the long text label classification method in the above-described embodiments.

[0161] The computer readable storage medium provided by the application may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more conductive wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted in any suitable medium, including but not limited to electrical wire, optical cable, RF (Radio Frequency), etc., or any suitable combination thereof.

[0162] The above computer readable storage medium can be included in the long text label classification device; or can exist separately and not be assembled into the long text label classification device.

[0163] The above computer readable storage medium carries one or more programs, which, when executed by the long text label classification device, cause the long text label classification device to perform the long text label classification.

[0164] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0165] The flow diagrams and the block diagrams in the drawings are meant as methodological and functional description of implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0166] The modules involved in the embodiments of the present application can be implemented in software or hardware. In some cases, the names of the modules do not constitute a limitation on the modules themselves.

[0167] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the long text label classification method described above, and can solve the technical problem of long text label classification. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the long text label classification method provided by the above embodiments, which will not be repeated here.

[0168] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the long text label classification method as described above.

[0169] The computer program product provided by the application can solve the technical problem of long text label classification. Compared with the prior art, the beneficial effects of the computer program product provided by the application are the same as those of the long text label classification method provided by the above-mentioned embodiments, and are not described here.

[0170] The above only describes some embodiments of the application, and does not limit the patent scope of the application. Any equivalent structural transformation, direct / indirect application in other related technical fields, or direct / indirect application in other related technical fields based on the technical concept of the application and the content of the specification and drawings are included in the patent protection scope of the application.

Claims

1. A method of label classification of long text, characterized by, The method for label classification of long text comprises: acquiring a text to be classified; based on the text to be classified, performing label classification processing through a preset large language model to obtain a label classification result, wherein the model architecture of the large language model comprises a multi-head self-attention mechanism and position encoding, the position encoding is used to encode text position information into a corresponding position vector, and a text sequence vector is constructed based on the position vector, and the multi-head self-attention mechanism is used to calculate the correlation between text sequence vectors in parallel through multiple self-attention layers.

2. The method of claim 1, wherein, Before the step of performing label classification processing through a preset large language model based on the text to be classified to obtain a label classification result, the method comprises: acquiring a text training sample and a classification label of the text training sample; performing vector conversion on the text training sample to obtain a text vector, performing position encoding on the text vector to obtain a position vector, and generating an encoded text vector based on the position vector; performing attention weighting calculation on the encoded text vector in parallel based on a preset plurality of self-attention layers to obtain a weighted text vector; based on the weighted text vector and the classification label, iteratively training a preset to-be-trained model to obtain a large language model meeting an accuracy condition.

3. The method of claim 2, wherein the long text is classified by the tag, and The step of performing attention weighting calculation on the encoded text vector in parallel based on a preset plurality of self-attention layers to obtain a weighted text vector comprises: acquiring an initial weight matrix of each self-attention layer; performing low-dimensional mapping on the encoded text vector based on a first low-rank parameter matrix under each self-attention layer to obtain a reduced-dimension text vector; performing high-dimensional mapping on the reduced-dimension text vector based on a second low-rank parameter matrix under each self-attention layer to obtain an increased-dimension text vector, wherein the dimension of the increased-dimension text vector is the same as that of the encoded text vector; based on the initial weight matrix and the increased-dimension text vector, calculating a target text vector of each self-attention layer, and fusing the target text vectors of the self-attention layers to obtain a weighted text vector.

4. The method of claim 3, wherein the long text is classified by the tag. The step of iteratively training a preset to-be-trained model based on the weighted text vector and the classification label to obtain a large language model meeting an accuracy condition comprises: based on the weighted text vector, performing label classification processing through a preset to-be-trained model to obtain a predicted classification result; performing error calculation on the predicted classification result and the classification label to obtain an error result; determining whether the error result meets an error standard indicated by a preset error threshold range; If the error result does not satisfy the error standard indicated by the preset error threshold range, a target weight matrix is calculated based on a preset gradient descent algorithm, the model parameters are updated based on the target weight matrix, and based on the updated to-be-trained model, the step of performing attention weighting calculation on the encoded text vector in parallel based on the preset multiple self-attention layers is returned until the error result satisfies the error standard indicated by the preset error threshold range, and the training is stopped to obtain the large language model. In the gradient descent algorithm, the first low-rank parameter matrix and the second low-rank parameter matrix are respectively set with different learning rate parameters.

5. The method of claim 2, wherein the long text is tagged by a category. The step of iteratively training a preset to-be-trained model based on the weighted text vector and the classification label to obtain a large language model satisfying an accuracy condition comprises: determining a target weight matrix of each iteration period based on the weighted text vector and the classification label; performing low-precision quantization on the target weight matrix of each iteration period to obtain a quantized weight matrix; updating the to-be-trained model of each iteration period based on the quantized weight matrix until a large language model satisfying an accuracy condition is obtained.

6. The method of claim 1, wherein, The step of obtaining the to-be-classified text comprises: obtaining an initial text and the number of characters of the initial text; determining whether the number of characters is greater than a preset character threshold; if the number of characters is greater than the character threshold, segmenting the initial text to obtain segmented texts, and extracting semantic feature values of the segmented texts; based on the semantic feature values, calculating the semantic difference between adjacent segmented texts, and based on the semantic difference, segmenting the initial text to obtain the to-be-classified text.

7. A label classification apparatus for long text, characterized by comprising: The device comprises: an acquisition module for acquiring to-be-classified text; a classification module for performing label classification processing on the to-be-classified text based on a preset large language model to obtain a label classification result, wherein the model architecture of the large language model comprises a multi-head self-attention mechanism and a position encoding, the position encoding is used to encode text position information into a corresponding position vector, and the position vector is used to construct a text sequence vector, and the multi-head self-attention mechanism is used to calculate the correlation between text sequence vectors in parallel through multiple self-attention layers.

8. A long text label classification apparatus characterized by comprising: The device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the label classification method for long text according to any one of claims 1 to 6.

9. A storage medium, characterized by The storage medium is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the label classification method for long text according to any one of claims 1 to 6.

10. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is executed by the processor to implement the steps of the label classification method for long text according to any one of claims 1 to 6.

Images