Long text classification model training method, long text classification method and device

Abstract

The invention provides a long text classification model training method and device, and a long text classification method and device, and the method comprises the steps: carrying out the vectorization of long text data, carrying out the partitioning of a target vector according to a plurality of preset partitioning sizes, and obtaining the local attention based on the local attention of each partitioning; and carrying out convolution on the global attention parameter of the target vector by utilizing a preset convolution layer, obtaining global attention based on the convolved global attention parameter, and training the initial long text classification model based on the local attention and the global attention. According to the method, the target vector is partitioned by adopting various preset partition sizes, so that the semantic information lost by the partition of one partition size is complemented by the partition of another size, the data calculation amount is reduced, the semantic information loss caused by the partition is reduced, and the model accuracy is improved. Moreover, a convolutional layer is used when the global attention is calculated, so that interaction between local information and global information is realized, and the accuracy of the model is further improved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a long text classification model training method, a long text classification method and a long text classification device. Background Art

[0002] Currently, Transformer-based language models (such as BERT, RoBERTa, etc.) are widely used in text classification tasks. However, since the computational complexity of attention in these models is quadratic with the length of the text, and the maximum length of text input supported by most models is 512, these models are not suitable for long text processing tasks.

[0003] In order to cope with the task of processing long texts, models such as Longformer and FLASH have been proposed. These models use linear attention, local attention + global attention, so that the attention calculation complexity is linearly related to the text length, and can accept longer text input lengths to expand the model's modeling capabilities on long texts. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a long text classification model training method, a long text classification method and a device to reduce model information loss and thereby improve model output accuracy.

[0005] According to one aspect of the present invention, a long text classification model training method is provided, the method comprising:

[0006] Input each piece of training data into the initial long text classification model, wherein the training data is long text data, and the label of the training data is a preset classification label;

[0007] Vectorizing the long text data using the initial long text classification model to obtain a target vector corresponding to the long text data;

[0008] Divide the target vector into blocks according to a plurality of preset block sizes to obtain blocks, and obtain the local attention of the long text data based on the local attention of each block;

[0009] Convolving the global attention parameters of the target vector using a preset convolution layer, and obtaining the global attention of the long text data based on the convolved global attention parameters, wherein the global attention parameters include a global query matrix and a global key matrix;

[0010] Outputting a target classification result of the long text data based on the local attention and the global attention;

[0011] Based on the target classification result and the loss between the preset classification labels of the training data, the initial long text classification model is trained until the loss converges to obtain a target long text classification model.

[0012] In a possible embodiment, the using the initial long text classification model to vectorize the long text data to obtain a target vector corresponding to the long text data includes:

[0013] When the length of the long text data is greater than the maximum input length of the initial long text classification model, truncating the long text data according to the maximum input length to obtain target text data;

[0014] When the length of the long text data is less than the maximum input length, padding the long text data according to the maximum input length to obtain target text data;

[0015] Obtaining embedding vectors, word vectors, position vectors, and paragraph vectors of the target text data;

[0016] The embedding vector, word vector, position vector and paragraph vector are concatenated to obtain a target vector corresponding to the target text data.

[0017] In a possible embodiment, the target vector is divided into blocks according to a plurality of preset block sizes to obtain the blocks, and based on the local attention of each of the blocks, the local attention of the long text data is obtained, including:

[0018] Performing affine transformation and linear transformation on the target vector according to a plurality of preset linear transformation matrices to obtain a local query matrix, a local key matrix, and a value matrix corresponding to the target vector;

[0019] Divide the target vector into blocks according to a first preset block size to obtain first blocks;

[0020] Divide the target vector into blocks according to a second preset block size to obtain second blocks; the second preset block size is different from the first preset block size;

[0021] For each of the first blocks, based on the position of the first block in the target vector, obtaining a first query matrix, a first key matrix and a first value matrix corresponding to the first block from the local query matrix, the local key matrix and the value matrix;

[0022] Calculate a first local attention of the first block based on the first query matrix, the first key matrix, and the first value matrix;

[0023] For each of the second blocks, based on the position of the second block in the target vector, obtaining a second query matrix, a second key matrix and a second value matrix corresponding to the second block from the local query matrix, the local key matrix and the value matrix;

[0024] Calculate a second local attention of the second block based on the second query matrix, the second key matrix, and the second value matrix;

[0025] The first local attention and the second local attention are fused to obtain the local attention of the long text data.

[0026] In a possible embodiment, the method further includes:

[0027] Performing affine transformation and linear transformation on the target vector according to a plurality of preset linear transformation matrices to obtain a global query matrix, a global key matrix, and a value matrix corresponding to the target vector;

[0028] The method of convolving the global attention parameter of the target vector using a preset convolution layer and obtaining the global attention of the long text data based on the convolved global attention parameter includes:

[0029] Using a preset convolution layer to convolve the global query matrix and the global key matrix respectively, to obtain a convolution query matrix and a convolution key matrix;

[0030] Based on the convolution query matrix, the convolution key matrix and the global value matrix, the global attention of the long text data is obtained.

[0031] In a possible embodiment, outputting the target classification result of the long text data based on the local attention and the global attention includes:

[0032] Based on the local attention and the global attention, the attention layer output result is obtained according to the following formula:

[0033] O=(U⊙(A lin +αA quad ))W o

[0034] Among them, O is the output of the attention layer, U is the result obtained by linearly mapping the target vector, and A lin is the global attention, A quad is the local attention, α is the preset parameter, W o are the parameters of the model to be trained;

[0035] The target classification result of the long text data is output based on the attention layer output result.

[0036] According to another aspect of the present invention, a long text classification method is provided, the method comprising:

[0037] Get the target long text data;

[0038] The target long text data is input into a preset long text classification model so that the preset long text classification model vectorizes the target long text data to obtain a target vector corresponding to the target long text data; the target vector is divided into blocks according to a plurality of preset block sizes to obtain blocks, and based on the local attention of each block, the local attention of the target long text data is obtained; the global attention parameters of the target vector are convolved using a preset convolutional layer, and the global attention of the target long text data is obtained based on the convolved global attention parameters, wherein the global attention parameters include a global query matrix and a global key matrix; the target classification result of the target long text data is output based on the local attention and the global attention, wherein the preset long text classification model is pre-trained by any of the long text classification model training methods described above.

[0039] According to another aspect of the present invention, a long text classification model training device is provided, the device comprising:

[0040] An input module, used to input each piece of training data into the initial long text classification model, wherein the training data is long text data, and the label of the training data is a preset classification label;

[0041] A vectorization module, used to vectorize the long text data using the initial long text classification model to obtain a target vector corresponding to the long text data;

[0042] A local attention calculation module, used for dividing the target vector into blocks according to a plurality of preset block sizes to obtain each block, and obtaining the local attention of the long text data based on the local attention of each block;

[0043] A global attention calculation module, used to convolve the global attention parameters of the target vector using a preset convolution layer, and obtain the global attention of the long text data based on the convolved global attention parameters, wherein the global attention parameters include a global query matrix and a global key matrix;

[0044] An output module, configured to output a target classification result of the long text data based on the local attention and the global attention;

[0045] A training module is used to train the initial long text classification model based on the target classification result and the loss between the preset classification labels of the training data until the loss converges to obtain a target long text classification model.

[0046] According to another aspect of the present invention, a long text classification device is provided, the device comprising:

[0047] An acquisition module is used to acquire target long text data;

[0048] A computing module is used to input the target long text data into a preset long text classification model so that the preset long text classification model vectorizes the target long text data to obtain a target vector corresponding to the target long text data; divide the target vector into blocks according to a plurality of preset block sizes to obtain each block, and obtain the local attention of the target long text data based on the local attention of each block; use a preset convolutional layer to convolve the global attention parameters of the target vector, and obtain the global attention of the target long text data based on the convolved global attention parameters, wherein the global attention parameters include a global query matrix and a global key matrix; output a target classification result of the target long text data based on the local attention and the global attention, wherein the preset long text classification model is pre-trained by any of the long text classification model training methods described above.

[0049] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0050] Processor; and

[0051] Memory for storing programs,

[0052] Wherein, the program includes instructions, and when the instructions are executed by the processor, the processor executes any of the above-mentioned long text classification model training methods and / or long text classification methods.

[0053] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any of the long text classification model training methods and / or long text classification methods described above.

[0054] One or more technical solutions provided in the embodiments of the present invention use a plurality of preset block sizes to block the target vector, calculate the local attention of each block, and obtain the local attention of the long text data based on the local attention of each block. In this way, the semantic information lost by using one block size for block division may be supplemented by the semantic information of the block obtained by using another block size for block division, thereby reducing the amount of data calculation based on block calculation while reducing the loss of semantic information caused by block division, thereby improving the accuracy of subsequent model output results. Furthermore, by using a convolutional layer when calculating global attention, since the convolutional layer uses a sliding window to obtain features, the features contained in the sliding window are usually partial features in the global attention, that is, a local attention. In this way, by interacting the local attention and the global attention in the process of calculating the global attention, the model learns as many training data features as possible, further improving the accuracy of the model output results. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0056] Figure 1 A schematic diagram of a flow chart of a long text classification model training method provided by an embodiment of the present invention;

[0057] Figure 2 A schematic diagram of performing target vector segmentation in an embodiment of the present invention;

[0058] Figure 3 A schematic diagram of a flow chart for calculating global attention in an embodiment of the present invention;

[0059] Figure 4 A schematic diagram of the structure of an initial long text classification model in an embodiment of the present invention;

[0060] Figure 5 A schematic diagram of a flow chart of a long text classification method provided by an embodiment of the present invention;

[0061] Figure 6 A schematic diagram of the structure of a long text classification model training device provided by an embodiment of the present invention;

[0062] Figure 7 A schematic diagram of the structure of a long text classification device provided by an embodiment of the present invention;

[0063] Figure 8 A block diagram of an exemplary electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0064] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0065] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0066] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0067] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0068] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0069] The FLSH model adopts a local + global attention calculation method. Standard attention is used within local blocks, and fast linear attention is used between global blocks. The final attention is obtained by adding local attention and global attention. Although this calculation method can reduce the computational complexity, due to the lack of interaction between global information and local information during the calculation of local and global attention, the model does not fully learn the semantic features of the text, resulting in some information loss, which in turn affects the accuracy of the model output results.

[0070] In order to reduce the information loss of the long text classification model and thus improve the accuracy of the model output results, the embodiments of the present invention provide a long text classification model training method, a long text classification method and a device. The text classification model training method provided by the embodiments of the present invention can be applied to any electronic device with a model training function, which can be a server, a computer or a mobile terminal, etc. The scheme of the present invention is described below with reference to the accompanying drawings:

[0071] Figure 1 A flowchart of a long text classification model training method provided by an embodiment of the present invention may include the following steps:

[0072] S101, inputting each piece of training data into an initial long text classification model, wherein the training data is long text data, and the label of the training data is a preset classification label;

[0073] S102, vectorizing the long text data using the initial long text classification model to obtain a target vector corresponding to the long text data;

[0074] S103, dividing the target vector into blocks according to a plurality of preset block sizes to obtain blocks, and obtaining the local attention of the long text data based on the local attention of each block;

[0075] S104, convolving the global attention parameters of the target vector using a preset convolution layer, and obtaining the global attention of the long text data based on the convolved global attention parameters, wherein the global attention parameters include a global query matrix and a global key matrix;

[0076] S105, outputting a target classification result of the long text data based on the local attention and the global attention;

[0077] S106: Based on the target classification result and the loss between the preset classification labels of the training data, the initial long text classification model is trained until the loss converges to obtain a target long text classification model.

[0078] By applying the embodiment of the present invention, the target vector is divided into blocks by using a plurality of preset block sizes, and the local attention of each block is calculated. Based on the local attention of each block, the local attention of the long text data is obtained. In this way, the semantic information lost when dividing the blocks using one block size may be supplemented by the semantic information of the blocks obtained by dividing the blocks using another block size. Thus, while reducing the amount of data calculation based on the block calculation, the loss of semantic information caused by the block calculation is reduced, thereby improving the accuracy of the subsequent model output results. Furthermore, by using the convolution layer when calculating the global attention, since the convolution layer uses a sliding window to obtain features, the features contained in the sliding window are usually partial features in the global attention, that is, a kind of local attention. In this way, by interacting the local attention and the global attention in the process of calculating the global attention, the model learns as many training data features as possible, further improving the accuracy of the model output results.

[0079] The following is an exemplary description of S101-S106:

[0080] In S101, the initial long text classification model can be a FLASH model, a Longformer model, etc. The initial long text classification model includes an attention layer, which can calculate attention for input data, and the attention includes global attention and global attention. The above training data can come from an open source data set or a self-built data set. Each piece of training data is long text data, wherein long text data refers to data whose length exceeds a preset length threshold, which can be 1000 words, 1500 words, etc. The above long text data can be news reports, novels, papers or reviews, etc. Each training data has a preset classification label, which can be a tendency label (such as "positive", "negative"), a subject label (such as "biology", "chemistry", etc.) or a subject label (such as "male frequency", "female frequency", etc.), etc., which can be set according to the actual application scenario.

[0081] The initial long text classification model can vectorize the input long text data, that is, convert the long text data into a machine-understandable vector form. In a possible embodiment, the long text data can be vectorized according to a preset vocabulary, which includes the correspondence between preset words and codes. Specifically, the long text data can be segmented according to the preset vocabulary, and the code corresponding to each segmented word can be obtained according to the correspondence. The codes can be spliced ​​to obtain the word vector corresponding to the long text data.

[0082] In a possible embodiment, the position vector and paragraph vector of each of the above-mentioned word segments can also be obtained, wherein the position vector can identify the sequential position of the word segmentation in all word segmentation results, and the paragraph vector can identify the paragraph position of the word segmentation in the long text data.

[0083] In a possible embodiment, the initial text classification model includes an embedding layer, which can embed and vectorize the input data, wherein embedding vectorization refers to mapping the input data from a high-dimensional space to a low-dimensional space, and the embedding vector can express the characteristics of the input data. Accordingly, after the long text data is converted into a vector form, the embedding layer can be used to embed and vectorize the word vector to obtain the embedding vector of the long text data. The word vector, position vector, paragraph vector and embedding vector of the long text data are concatenated to obtain the target vector.

[0084] In a possible embodiment, when the length of the long text data is greater than the maximum input length of the initial long text classification model, the long text data can be truncated according to the maximum input length to obtain target text data; when the length of the long text data is less than the maximum input length, the long text data is padded according to the maximum input length to obtain target text data; the embedding vector, word vector, position vector and paragraph vector of the target text data are obtained; the embedding vector, word vector, position vector and paragraph vector are concatenated to obtain a target vector corresponding to the target text data.

[0085] The above maximum input length can be flexibly set according to the actual application scenario. For example, the maximum output length can be 8192 tokens, where the token refers to each word segment in the preset word list. For example, for long text data with a length greater than the maximum input length, the first 8192 tokens can be retained. For text data with a length less than the maximum input length, a preset value can be added to the end of the input text, such as 0 or 1, so that the token length reaches the maximum input text length.

[0086] For the target vector, an affine transformation and a linear transformation can be performed on the target vector according to a plurality of preset transformation matrices. Affine transformation refers to a linear transformation of a vector space followed by a translation in geometry to transform it into another vector space. In the model scenario, the affine transformation can be obtained by a preset transformation matrix and a preset vector. The dimensions of the preset transformation matrix and the preset vector can be set according to the actual application scenario. By performing an affine transformation on the target vector, the local query matrix Q used for local attention calculation can be obtained. quad , local bond matrix K quad , the global query matrix Q for global attention calculation lin, global key matrix K lin By performing a linear transformation on the target vector, the value matrix V can be obtained.

[0087] Among them, attention calculation refers to calculating the correlation between each token in the target vector and other tokens in the target vector to determine the degree of attention of the model to the input data. Attention calculation is usually based on three matrices: Q (query), K (key), and V (value).

[0088] In an embodiment of the present invention, the target vector may be divided into blocks according to a preset size, and local attention may be calculated for each obtained block, so that the amount of data calculation may be reduced, thereby reducing the model calculation pressure. In a possible embodiment, the target vector may be divided into blocks according to a plurality of preset sizes to obtain blocks of a plurality of different sizes. Specifically, the local attention may be calculated according to the following steps:

[0089] S201, dividing the target vector into blocks according to a first preset block size to obtain first blocks;

[0090] S202, dividing the target vector into blocks according to a second preset block size to obtain second blocks; the second preset block size is different from the first preset block size;

[0091] In a possible embodiment, in order to minimize the semantic loss of long text data caused by local attention calculated based on block, the breakpoints caused by block division according to the first preset block size can be completely present in the second preset block, that is, the least common multiple of the first preset block size and the second preset block size is maximized, thereby reducing information loss between blocks.

[0092] Exemplarily, for long text data with a length of 6, the first preset block size can be set to 2 and the second preset block size can be set to 3. In this way, the blocks obtained according to the first preset block size are [1,2], [3,4], [5,6], and the blocks obtained according to the second preset block size are [1,2,3], [4,5,6]. It can be seen that the block breakpoints obtained according to the first preset block size are between 2 and 3 and between 4 and 5, and in the blocks obtained according to the second preset block size, 2 and 3 belong to the same block, and 4 and 5 belong to the same block, so that the semantic loss between 2 and 3 and between 4 and 5 is compensated in the second block.

[0093] S203: for each of the first blocks, based on the position of the first block in the target vector, obtaining a first query matrix, a first key matrix and a first value matrix corresponding to the first block from the local query matrix, the local key matrix and the value matrix;

[0094] The block is a part of the data in the target vector. Therefore, in this step, the values ​​of the corresponding positions in the QKV matrix can be determined according to the position of the first block in the target vector to form the QKV matrix corresponding to the first block. For example, if the target vector is a 6*5 dimensional matrix and the first block is [1,2], then the QKV matrix corresponding to the first block is a matrix composed of the first 2*5 dimensional data in the local QKV.

[0095] S204, calculating a first local attention of the first block based on the first query matrix, the first key matrix and the first value matrix;

[0096] In this step, the first local attention of the first block can be calculated according to the following formula:

[0097]

[0098] in, is the local attention when the block size is c1, relu 2 () is the activation function, which can normalize the data. is the block query matrix when the block size is c1, is the block key matrix when the block size is c1, V c1 is the block value matrix when the block size is c1, and b1 is the trainable model parameter.

[0099] In a possible embodiment, the first local attention of each first block can be calculated according to the above formula, and the first local attentions can be fused according to the number of first blocks to obtain the first local attention corresponding to the first preset block size. As a possible implementation method, one-dimensional data can be added to the matrix composed of the first local attentions of each first block. The dimensional data is used to represent the rank position of the first block in each first block, and then the high-dimensional data is compressed to obtain the first local attention corresponding to the first preset block size.

[0100] Exemplarily, the first local attention of each first block is a 6*5 dimensional plane matrix, the number of first blocks is 3, and the first local attention of each block can constitute a 6*5*3 dimensional three-dimensional space matrix, which is then compressed into a plane matrix, and the plane matrix can be used as the first local attention corresponding to the first preset block size. In the embodiment of the present invention, any feasible matrix compression method can be used to compress the three-dimensional space matrix, and the present invention does not specifically limit this.

[0101] S205: for each of the second blocks, based on the position of the second block in the target vector, obtain a second query matrix, a second key matrix and a second value matrix corresponding to the second block from the local query matrix, the local key matrix and the value matrix;

[0102] S206, calculating a second local attention of the second block based on the second query matrix, the second key matrix and the second value matrix;

[0103] In this step, the first local attention of the first block can be calculated according to the following formula:

[0104]

[0105] in, is the local attention when the block size is c2, relu 2 () is the activation function, which can normalize the data. is the block query matrix when the block size is c2, is the block key matrix when the block size is c2, V c2 is the block value matrix when the block size is c2, and b2 is the trainable model parameter.

[0106] S207: Fuse the first local attention and the second local attention to obtain the local attention of the long text data.

[0107] In this step, the local attention A of the long text data can be obtained according to the following formula quad :

[0108]

[0109] like Figure 2 As shown, Figure 2 This is a schematic diagram of calculating local attention in an embodiment of the present invention. Specifically, the target vector is divided into blocks according to the first block size and the second block size to obtain the first block and the second block, wherein the breakpoint of the first block is in the second block, and the breakpoint of the second block is in the first block. While reducing the amount of calculation by dividing the blocks, the information loss caused by the blocks is reduced. Then, the local attention is calculated for the first block and the second block respectively, and the local attention of the two blocks is added to obtain the local attention of the long text data.

[0110] The FLASH model uses a linear attention calculation method to calculate global attention to reduce the complexity of attention calculation. Specifically, the FLASH model uses the following formula to calculate global attention:

[0111]

[0112] Among them, A lin′ is the global attention, Q lin is the global query matrix, K lin is the global key matrix, V is the value matrix. This calculation method only learns the weights of the global tokens between inputs, and does not consider the dependencies between tokens within the input. That is, when calculating the global attention, it does not interact with the local attention, that is, the global information does not interact with the local information, which makes the semantic information in the long text data not fully utilized.

[0113] In a possible embodiment, in order to enhance the interaction between global information and local information, a convolution layer can be added to the model. Since the principle of convolution is to use a sliding window of a preset size (convolution kernel size) to sample each data in the input and fuse it to obtain the convolution layer output, that is, the sampling and fusion of the local information of the target vector is realized during the convolution process. Therefore, the global query matrix and the global key matrix are convolved through the convolution layer, so that local information is introduced in the calculation process of global attention, thereby enhancing the interaction between global information and local information. The configuration of the above-mentioned convolution layer and the number of convolution layers can be flexibly set according to actual needs. The following takes two consecutive one-dimensional convolutions as an example to exemplify the process of convolving the global query matrix and the global key matrix in an embodiment of the present invention:

[0114] like Figure 3 As shown, after performing affine transformation and linear transformation on the input vector (i.e., the target vector mentioned above) to obtain the global query matrix, the global key matrix and the global value matrix, the global query matrix and the global key matrix are convolved twice through convolution layer 1 and convolution layer 2 to obtain the convolution query matrix and the convolution key matrix. The global attention of the long text data is obtained by calculating the output layer based on the convolution query matrix, the convolution key matrix and the global value matrix.

[0115] In one possible embodiment, the global attention can be calculated by the following formula:

[0116] Q lin_conv =conv2(conv1(Q lin ))

[0117] K lin_conv =conv2(conv1(K lin ))

[0118]

[0119] Among them, Q lin is the global query matrix, K linis the global key matrix, conv1() and conv2() represent two one-dimensional convolutional layers, Q lin_conv Q lin The convolution query matrix obtained after two one-dimensional convolutions, K lin_conv K lin The convolution key matrix A is obtained after two one-dimensional convolutions. lin Global attention for long text data.

[0120] The above attention-related calculations are all performed by the attention layer in the initial long text classification model, and then the output of the attention layer can be obtained based on the above local attention and global attention. Specifically, the output of the attention layer can be obtained according to the following formula:

[0121] O=(U⊙(A lin +αA quad ))W o

[0122] Among them, O is the output of the attention layer, U is the result obtained by linearly mapping the target vector, and A lin is the global attention, A quad is the local attention, α is the preset parameter, W o are the model parameters to be trained.

[0123] In a possible embodiment, Figure 4 As shown, the above-mentioned initial long text classification model can be composed of an embedding layer, a FLASH improved model and a classification layer, wherein the FLASH improved model refers to the FLASH model after the attention layer is improved according to the above-mentioned attention calculation method, and the classification layer is used to classify according to the output of the FLASH improved model, and the output of the FLASH model improved model can be a text feature. The input long text data is embedded vectorized by the embedding layer to obtain a target vector, and the target vector is input to the FLASH improved model. The FLASH improved model performs attention calculation based on the target vector, and obtains text features based on the attention calculation result. The classification layer outputs a classification result based on the text feature, and the classification result can be the probability that the long text data belongs to each preset classification, or a specific classification.

[0124] Afterwards, the loss can be calculated based on the classification result and the classification label of the long text data, and the loss can be a cross entropy loss, a semantic loss, etc. Afterwards, the embedding layer and the FLASH improved model can be trained based on the loss until the loss converges to obtain the target long text classification model. In the embodiment of the present invention, various feasible parameter adjustment methods such as the gradient descent method can be used to train the initial long text classification model based on the above loss to obtain the target long text classification model.

[0125] By applying the embodiments of the present invention, a new local attention calculation method is introduced and the interaction between global attention and local attention is enhanced, so as to reduce the information loss caused by local block calculation and the information loss caused by insufficient interaction between global information and local information. At the same time, too many new trainable parameters are not introduced into the model. Under the premise of not significantly increasing the model training time and inference time, the performance and generalization ability of the model in the long text modeling task are improved.

[0126] Based on the same inventive concept, the embodiment of the present invention also provides a long text classification method, such as Figure 5 As shown, the method may include the following steps:

[0127] S501, obtaining target long text data;

[0128] S502. Input the target long text data into a preset long text classification model so that the preset long text classification model vectorizes the target long text data to obtain a target vector corresponding to the target long text data; divide the target vector into blocks according to a plurality of preset block sizes to obtain blocks, and obtain the local attention of the target long text data based on the local attention of each block; use a preset convolutional layer to convolve the global attention parameters of the target vector, and obtain the global attention of the target long text data based on the convolved global attention parameters, wherein the global attention parameters include a global query matrix and a global key matrix; output a target classification result of the target long text data based on the local attention and the global attention, wherein the preset long text classification model is pre-trained by any of the long text classification model training methods described above.

[0129] According to another aspect of the present invention, a long text classification model training device is provided. Figure 6 As shown, the apparatus 600 may include:

[0130] An input module 601 is used to input each piece of training data into an initial long text classification model, wherein the training data is long text data and the label of the training data is a preset classification label;

[0131] A vectorization module 602 is used to vectorize the long text data using the initial long text classification model to obtain a target vector corresponding to the long text data;

[0132] A local attention calculation module 603 is used to divide the target vector into blocks according to a plurality of preset block sizes to obtain each block, and obtain the local attention of the long text data based on the local attention of each block;

[0133] A global attention calculation module 604 is used to convolve the global attention parameters of the target vector using a preset convolution layer, and obtain the global attention of the long text data based on the convolved global attention parameters, wherein the global attention parameters include a global query matrix and a global key matrix;

[0134] An output module 605 is used to output a target classification result of the long text data based on the local attention and the global attention;

[0135] The training module 606 is used to train the initial long text classification model based on the target classification result and the loss between the preset classification labels of the training data until the loss converges to obtain the target long text classification model.

[0136] According to another aspect of the present invention, a long text classification device is provided. Figure 7 As shown, the apparatus 700 may include:

[0137] An acquisition module 701 is used to acquire target long text data;

[0138] The calculation module 702 is used to input the target long text data into a preset long text classification model so that the preset long text classification model vectorizes the target long text data to obtain a target vector corresponding to the target long text data; divides the target vector into blocks according to a plurality of preset block sizes to obtain each block, and obtains the local attention of the target long text data based on the local attention of each block; uses a preset convolution layer to convolve the global attention parameters of the target vector, and obtains the global attention of the target long text data based on the convolved global attention parameters, wherein the global attention parameters include a global query matrix and a global key matrix; outputs a target classification result of the target long text data based on the local attention and the global attention, wherein the preset long text classification model is pre-trained by any of the long text classification model training methods described above.

[0139] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0140] The exemplary embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present invention when executed by the at least one processor.

[0141] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0142] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.

[0143] refer to Figure 8 , a block diagram of an electronic device 800 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0144] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0145] A plurality of components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 may be any type of device capable of inputting information to the electronic device 800, and the input unit 806 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 807 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 808 may include, but is not limited to, a disk, an optical disk. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0146] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above. For example, in some embodiments, the long text classification model training method and the long text classification method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. In some embodiments, the computing unit 801 may be configured to perform the long text classification model training method and the long text classification method by any other appropriate means (e.g., by means of firmware).

[0147] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0148] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0149] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0150] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0151] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0152] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A long text classification model training method, It is characterized in that The method comprises: Input each piece of training data into the initial long text classification model, wherein the training data is long text data, and the label of the training data is a preset classification label; Vectorizing the long text data using the initial long text classification model to obtain a target vector corresponding to the long text data; Divide the target vector into blocks according to a plurality of preset block sizes to obtain blocks, and obtain the local attention of the long text data based on the local attention of each block; Convolving the global attention parameters of the target vector using a preset convolution layer, and obtaining the global attention of the long text data based on the convolved global attention parameters, wherein the global attention parameters include a global query matrix and a global key matrix; Outputting a target classification result of the long text data based on the local attention and the global attention; Based on the target classification result and the loss between the preset classification labels of the training data, the initial long text classification model is trained until the loss converges to obtain a target long text classification model.

2. The method according to claim 1, It is characterized in that The using the initial long text classification model to vectorize the long text data to obtain a target vector corresponding to the long text data includes: When the length of the long text data is greater than the maximum input length of the initial long text classification model, truncating the long text data according to the maximum input length to obtain target text data; When the length of the long text data is less than the maximum input length, padding the long text data according to the maximum input length to obtain target text data; Obtaining embedding vectors, word vectors, position vectors, and paragraph vectors of the target text data; The embedding vector, word vector, position vector and paragraph vector are concatenated to obtain a target vector corresponding to the target text data.

3. The method according to claim 1, It is characterized in that The target vector is divided into blocks according to a plurality of preset block sizes to obtain each block, and based on the local attention of each block, the local attention of the long text data is obtained, including: Performing affine transformation and linear transformation on the target vector according to a plurality of preset linear transformation matrices to obtain a local query matrix, a local key matrix, and a value matrix corresponding to the target vector; Divide the target vector into blocks according to a first preset block size to obtain first blocks; Divide the target vector into blocks according to a second preset block size to obtain second blocks; the second preset block size is different from the first preset block size; For each of the first blocks, based on the position of the first block in the target vector, obtaining a first query matrix, a first key matrix and a first value matrix corresponding to the first block from the local query matrix, the local key matrix and the value matrix; Calculate a first local attention of the first block based on the first query matrix, the first key matrix, and the first value matrix; For each of the second blocks, based on the position of the second block in the target vector, obtaining a second query matrix, a second key matrix and a second value matrix corresponding to the second block from the local query matrix, the local key matrix and the value matrix; Calculate a second local attention of the second block based on the second query matrix, the second key matrix, and the second value matrix; The first local attention and the second local attention are fused to obtain the local attention of the long text data.

4. The method according to claim 1, It is characterized in that The method further comprises: Performing affine transformation and linear transformation on the target vector according to a plurality of preset linear transformation matrices to obtain a global query matrix, a global key matrix, and a value matrix corresponding to the target vector; The method of convolving the global attention parameter of the target vector using a preset convolution layer, and obtaining the global attention of the long text data based on the convolved global attention parameter, includes: Using a preset convolution layer to convolve the global query matrix and the global key matrix respectively, to obtain a convolution query matrix and a convolution key matrix; Based on the convolution query matrix, the convolution key matrix and the global value matrix, the global attention of the long text data is obtained.

5. The method according to claim 1, It is characterized in that The outputting the target classification result of the long text data based on the local attention and the global attention includes: Based on the local attention and the global attention, the attention layer output result is obtained according to the following formula: O=(U⊙(A lin +αA quad ))W o Among them, O is the output of the attention layer, U is the result obtained by linearly mapping the target vector, and A lin is the global attention, A quad is the local attention, α is the preset parameter, W o are the parameters of the model to be trained; The target classification result of the long text data is output based on the attention layer output result.

6. A long text classification method, It is characterized in that The method comprises: Get the target long text data; Inputting the target long text data into a preset long text classification model so that the preset long text classification model vectorizes the target long text data to obtain a target vector corresponding to the target long text data; The target vector is divided into blocks according to a plurality of preset block sizes to obtain blocks, and based on the local attention of each block, the local attention of the target long text data is obtained; Convolving the global attention parameters of the target vector using a preset convolution layer, and obtaining the global attention of the target long text data based on the convolved global attention parameters, wherein the global attention parameters include a global query matrix and a global key matrix; The target classification result of the target long text data is output based on the local attention and the global attention, wherein the preset long text classification model is pre-trained by the long text classification model training method according to any one of claims 1-5.

7. A long text classification model training device, It is characterized in that The device comprises: An input module, used to input each piece of training data into the initial long text classification model, wherein the training data is long text data, and the label of the training data is a preset classification label; A vectorization module, used to vectorize the long text data using the initial long text classification model to obtain a target vector corresponding to the long text data; A local attention calculation module, used for dividing the target vector into blocks according to a plurality of preset block sizes to obtain each block, and obtaining the local attention of the long text data based on the local attention of each block; A global attention calculation module, used to convolve the global attention parameters of the target vector using a preset convolution layer, and obtain the global attention of the long text data based on the convolved global attention parameters, wherein the global attention parameters include a global query matrix and a global key matrix; An output module, configured to output a target classification result of the long text data based on the local attention and the global attention; A training module is used to train the initial long text classification model based on the target classification result and the loss between the preset classification labels of the training data until the loss converges to obtain a target long text classification model.

8. A long text classification device, It is characterized in that The device comprises: An acquisition module is used to acquire target long text data; A computing module, used for inputting the target long text data into a preset long text classification model so that the preset long text classification model vectorizes the target long text data to obtain a target vector corresponding to the target long text data; dividing the target vector into blocks according to a plurality of preset block sizes to obtain each block, and obtaining the local attention of the target long text data based on the local attention of each block; convolving the global attention parameters of the target vector using a preset convolutional layer, and obtaining the global attention of the target long text data based on the convolved global attention parameters, wherein the global attention parameters include a global query matrix and a global key matrix; outputting a target classification result of the target long text data based on the local attention and the global attention, wherein the preset long text classification model is pre-trained by the long text classification model training method as described in any one of claims 1-5.

9. An electronic device, include: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1-5 or 6.

10. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to make a computer execute the method according to any one of claims 1-5 or 6.

Images