A long text processing method, device, equipment and storage medium

Abstract

The embodiment of the present invention provides a method, device, equipment and storage medium for processing a long text, the method comprising: obtaining a long text to be processed, and dividing the long text into a plurality of text blocks, the length of each text block being determined according to a preset text processing model; encoding each text block to obtain a feature vector corresponding to the text block; obtaining a candidate feature vector, the candidate feature vector being obtained by encoding a target text; calculating a weight value corresponding to each text block based on the feature vector corresponding to each text block and the candidate feature vector; obtaining a feature vector of the long text based on the feature vector corresponding to each text block and the weight value corresponding to each text block, and using the feature vector of the long text as a processing result of the long text. The embodiment of the present invention can realize the processing of a long text.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method, device, equipment and storage medium for processing long texts. Background Art

[0002] With the development of artificial intelligence, natural language processing (NLP) technology has been widely used in many scenarios, such as sentiment analysis, text similarity calculation, comment extraction, text classification, lexical analysis, etc. In these natural language processing scenarios, text processing is required. At present, most text processing models are trained or adjusted based on pre-trained models. Specifically, based on the pre-trained model, the parameters of the pre-trained model are fine-tuned to train downstream tasks, and text processing models in different scenarios are implemented to process text. The downstream tasks can be, for example, text classification, key information extraction, sequence labeling, etc.

[0003] Bidirectional Encoder Representations from Transformers (BERT) is a language representation model, in which Transformer is a feature extractor based on self-attention. However, the current Transformer-based pre-training model has a limit on the length of text that can be input into the model at a time. For example, the length of text that can be processed at a time by a 12-layer or 24-layer Transformer structure does not exceed 512 words. Therefore, for some NLP tasks that need to process long texts, the length of 512 words cannot meet the requirements.

[0004] Therefore, how to process long texts based on the current pre-training model has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a method, device, equipment and storage medium for processing long texts, so as to realize the processing of long texts. The specific technical scheme is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a method for processing a long text, the method comprising:

[0007] Obtaining a long text to be processed, and dividing the long text into a plurality of text blocks, wherein the length of each text block is determined according to a preset text processing model;

[0008] For each text block, the text block is encoded to obtain a feature vector corresponding to the text block to obtain a candidate feature vector, wherein the candidate feature vector is obtained by encoding the target text;

[0009] Based on the feature vector corresponding to each of the text blocks and the candidate feature vectors, calculating the weight value corresponding to each of the text blocks;

[0010] Based on the feature vectors corresponding to each text block and the weight values ​​corresponding to each text block, the feature vector of the long text is obtained, and the feature vector of the long text is used as the processing result of the long text.

[0011] Optionally, the calculating the weight value corresponding to each text block based on the feature vector corresponding to each text block and the candidate feature vector includes:

[0012] For each feature vector corresponding to the text block, the feature vector corresponding to the text block is interactively calculated with the candidate feature vector, and a weight value corresponding to the text block is obtained based on the interactive result.

[0013] Optionally, obtaining a weight value corresponding to the text block based on the interaction result includes:

[0014] The interaction result is normalized, and the normalized result is determined as the weight value corresponding to the text block.

[0015] Optionally, obtaining the feature vector of the long text based on the feature vector corresponding to each text block and the weight value corresponding to each text block includes:

[0016] Calculate the product of the feature vector corresponding to each text block and the weight value corresponding to the text block to obtain the target feature vector corresponding to the text block;

[0017] The target feature vectors corresponding to the text blocks are summed, and the sum result is determined as the feature vector of the long text.

[0018] Optionally, encoding each text block to obtain a feature vector corresponding to the text block includes:

[0019] For each text block, the pre-trained model is used to encode the text block to obtain a feature vector corresponding to the text block.

[0020] Optionally, the target text is: text in a target application scenario, or the long text; and obtaining a candidate feature vector includes:

[0021] In the case where the text in the target application scenario is a short text, encoding the text in the target application scenario to obtain the candidate feature vector;

[0022] Alternatively, when the text in the target application scenario is a long text, the text in the target application scenario is divided into blocks to obtain a plurality of target text blocks; for each target text block, the target text block is encoded to obtain a feature vector corresponding to the target text block; the feature vectors corresponding to each of the target text blocks are aggregated to obtain the candidate feature vector;

[0023] Alternatively, the long text is divided into blocks to obtain a plurality of text blocks; for each text block, the text block is encoded to obtain a feature vector corresponding to the text block; the feature vectors corresponding to each of the text blocks are aggregated to obtain the candidate feature vector.

[0024] Optionally, the aggregating the feature vectors corresponding to the target text blocks to obtain the candidate feature vectors includes:

[0025] Concatenating the feature vectors corresponding to the target text blocks to obtain the candidate feature vectors;

[0026] Alternatively, the candidate feature vector is determined by accumulating and summing the feature vectors corresponding to each target text block;

[0027] Alternatively, the candidate feature vector is determined by taking the weighted sum vector of the feature vectors corresponding to each target text block.

[0028] In a second aspect, an embodiment of the present invention provides a long text processing device, the device comprising:

[0029] A segmentation module, used for obtaining a long text to be processed and segmenting the long text into a plurality of text blocks, wherein the length of each text block is determined according to a preset text processing model;

[0030] The encoding module is used to encode each text block to obtain a feature vector corresponding to the text block;

[0031] An acquisition module, used for acquiring a candidate feature vector, wherein the candidate feature vector is obtained by encoding the target text;

[0032] A calculation module, used for calculating a weight value corresponding to each of the text blocks based on the feature vector corresponding to each of the text blocks and the candidate feature vectors;

[0033] The determination module is used to obtain the feature vector of the long text based on the feature vector corresponding to each text block and the weight value corresponding to each text block, and use the feature vector of the long text as the processing result of the long text.

[0034] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0035] Memory, used to store computer programs;

[0036] The processor is used to implement the steps of the long text processing method described in the first aspect when executing the program stored in the memory.

[0037] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a long text processing method described in the first aspect are implemented.

[0038] An embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the steps of the above-mentioned method for processing a long text.

[0039] Beneficial effects of the embodiments of the present invention:

[0040] The embodiment of the present invention provides a long text processing method, device, equipment and storage medium. Based on a preset text processing model, the long text to be processed is divided into multiple text blocks of specific lengths, that is, the long text to be processed is converted into short texts that can be processed by the preset text processing model for processing, and can adapt to the processing of long texts by text processing models with limited input length. Based on the feature vector corresponding to each text block and the candidate feature vector, the semantic contribution of each text block to the long text is calculated, and the semantic interaction between each text block and the target text is realized, and the needs of downstream tasks are taken into consideration. Further, based on the feature vector corresponding to each text block and the weight value corresponding to each text block, the feature vector of the long text is obtained, and the feature vector of the long text is used as the processing result of the long text, thereby realizing the processing of the long text.

[0041] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0043] Figure 1 A flowchart of a method for processing a long text provided by an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of a flow chart of an implementation method of determining a long text feature vector provided by an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of a process for obtaining a candidate feature vector according to an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of an architecture for determining a long text feature vector implementation method provided by an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of the structure of a long text processing device provided by an embodiment of the present invention;

[0048] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field based on this application belong to the scope of protection of the present invention.

[0050] In order to process a long text, an embodiment of the present invention provides a method, an apparatus, a device and a storage medium for processing a long text.

[0051] Among them, an embodiment of the present invention provides a long text processing method, including:

[0052] Obtain a long text to be processed, and divide the long text into multiple text blocks, where the length of each text block is determined according to a preset text processing model;

[0053] For each text block, the text block is encoded to obtain a feature vector corresponding to the text block to obtain a candidate feature vector, where the candidate feature vector is obtained by encoding the target text;

[0054] Based on the feature vector corresponding to each text block and the candidate feature vectors, a weight value corresponding to each text block is calculated;

[0055] Based on the feature vectors corresponding to each text block and the weight values ​​corresponding to each text block, a feature vector of the long text is obtained, and the feature vector of the long text is used as a processing result of the long text.

[0056] The embodiment of the present invention provides a method for processing a long text. Based on a preset text processing model, the long text to be processed is divided into a plurality of text blocks of a specific length, that is, the long text to be processed is converted into a short text that can be processed by the preset text processing model for processing. The method can adapt to the processing of the long text by the text processing model with a limited input length. Based on the feature vector corresponding to each text block and the candidate feature vector, the semantic contribution of each text block to the long text is calculated, and the semantic interaction between each text block and the target text is realized, and the needs of downstream tasks are taken into consideration. Furthermore, based on the feature vector corresponding to each text block and the weight value corresponding to each text block, the feature vector of the long text is obtained, and the feature vector of the long text is used as the processing result of the long text, thereby realizing the processing of the long text.

[0057] The following is a detailed introduction to a long text processing method provided by an embodiment of the present invention:

[0058] A long text processing method provided by an embodiment of the present invention can be applied to an electronic device, which may be a client device, a server device, or the like.

[0059] like Figure 1 As shown, an embodiment of the present invention provides a method for processing a long text, which may include the following steps:

[0060] S101, obtaining a long text to be processed, and dividing the long text into a plurality of text blocks, wherein the length of each text block is determined according to a preset text processing model.

[0061] In an embodiment of the present invention, a long text to be processed is obtained from an electronic device. The long text to be processed may be a text having a length greater than a preset length. The preset length may be determined based on the maximum text length that can be processed by a preset text processing model, wherein the text length is determined based on the number of words and / or characters contained in the text. In the field of natural language processing, some text processing models have certain limitations on the length of text that can be processed. For long texts that exceed the maximum text length that can be processed by the text processing model, in an embodiment of the present invention, the long text is segmented into multiple text blocks, and the length of each text block can be determined based on the preset text processing model.

[0062] Among them, the preset text processing model can be a text processing model determined according to actual needs, which can realize text processing tasks in different scenarios. The text processing model has certain restrictions on the length of the input text. Exemplarily, the maximum length of the text that can be processed by the text processing model based on the Transformer structure is 512 words at a time. Then, for a long text with a length greater than 512 words, the long text can be divided into multiple text blocks, and the length of each text block is not greater than 512 words. For texts in different languages, the text length is judged differently. For example, Chinese text can judge the number of words contained in the text, and English text can judge the number of characters contained in the text.

[0063] As an optional implementation of the embodiment of the present invention, the process of dividing a long text into a plurality of text blocks may include: dividing the long text into a plurality of mutually non-intersecting text blocks.

[0064] In an embodiment of the present invention, a long text to be processed can be divided into multiple non-intersecting text blocks. In one implementation, the length of the long text to be processed can be determined first, and then the long text to be processed can be divided into multiple non-intersecting text blocks with a set length in a sequential order according to the set length, or the long text to be processed can be divided into multiple non-intersecting text blocks with a length not exceeding the set length in a sequential order according to specified characters. The set length can be any value not exceeding the above-mentioned preset length. Specifically, the length is determined according to the number of words and / or characters contained in the text, and those skilled in the art can set it according to actual needs.

[0065] Taking the text processing model based on the Transformer structure as an example, the maximum length of the text that can be processed at a time is 512 words, so the long text can be divided into multiple text blocks with a length not exceeding 512 words. Exemplarily, taking the length of the text block after segmentation as 512 words or 256 words as an example, the long text to be processed is divided into multiple non-intersecting text blocks, and the words in the long text are sorted in positive order. The first text block contains the 1st to 512th words of the long text, and the second text block contains the 513th to 1024th words of the long text, and so on, or the first text block contains the 1st to 256th words of the long text, and the second text block contains the 257th to 512th words of the long text, and so on. It should be noted that the length judgment method is different for texts in different languages. For example, Chinese text can judge the number of words contained in the text, and English text can judge the number of characters contained in the text, etc.

[0066] The long text to be processed is divided into multiple non-intersecting text blocks so that there is no text overlap between the text blocks, avoiding the problem of repeated encoding of some words and semantic overlap between different text blocks during the subsequent encoding of the text blocks.

[0067] As an optional implementation of the embodiment of the present invention, the process of implementing the segmentation of a long text into a plurality of text blocks may include: segmenting the long text into a plurality of text blocks in a sliding window manner.

[0068] In one implementation, the sliding step size can be set according to actual needs, and the long text is divided into multiple text blocks of length not exceeding the preset length in a sliding window manner, and the multiple text blocks are intersecting. Exemplarily, the sliding step size is 128 words, the preset length is 512 words, and the long text to be processed is divided into multiple text blocks of length not exceeding the preset length in a sliding window manner, and the words in the long text are sorted in positive order, and the first text block contains the 1st to 512th words of the long text, and the second text block contains the 129th to 640th words of the long text, and so on, or the first text block contains the 1st to 256th words of the long text, and the second text block contains the 129th to 384th words of the long text, and so on.

[0069] The long text to be processed is divided into multiple intersecting text blocks in a sliding window manner, so that the context between each text block can be taken into account, so as to better extract the semantic features of the long text.

[0070] See also Figure 1 , S102, for each text block, encode the text block to obtain a feature vector corresponding to the text block.

[0071] After the long text is divided into a plurality of text blocks, each text block may be encoded to obtain a feature vector corresponding to the text block, wherein the feature vector is a vector representation of the encoded text block.

[0072] As an optional implementation of the embodiment of the present invention, for each text block, encoding the text block to obtain a feature vector corresponding to the text block may include:

[0073] For each text block, the pre-trained model is used to encode the text block to obtain the feature vector corresponding to the text block.

[0074] In an embodiment of the present invention, for each text block obtained by segmentation, the text block can be input into a pre-trained model for encoding to obtain a feature vector corresponding to the text block. The pre-trained model can be a model using a Transformer structure, for example, a pre-trained model based on a bidirectional encoder representation from Transformers (BERT), a RoBERTa pre-trained model, a RoBERTa-large Chinese pre-trained model, and the like.

[0075] For example, the BERT pre-trained model is used as a pre-trained model for illustration. The BERT pre-trained model is a typical bidirectional encoding model. The BERT model extracts text features based on Transformer, which is more efficient than CNN / RNN and can capture true bidirectional context information and longer-distance dependencies. Specifically, for each text block, the text block can be input into the Transformer-based pre-trained model for encoding to obtain the feature vector corresponding to the text block.

[0076] As an optional implementation of the embodiment of the present invention, the pre-trained model may also be a model obtained by adopting a Transformer structure and fine-tuning each text block contained in the long text.

[0077] In an embodiment of the present invention, a pre-trained model of a Transformer structure may be used to encode text blocks, or a pre-trained model obtained by fine-tuning each text block contained in a long text to be processed may be used to encode each text block.

[0078] A pre-trained model with a Transformer structure is used to encode text blocks. For example, a BERT pre-trained model based on a Transformer bidirectional encoder representation is used to encode text blocks. The BERT pre-trained model is an open source pre-trained language model that can pre-process most texts in the field of natural language processing. In order to improve the accuracy of encoding text blocks, the BERT pre-trained model can be further fine-tuned using the long text to be processed and specific training tasks, so that the BERT pre-trained model obtained after fine-tuning can extract more accurate feature information for the long text to be processed. The above-mentioned specific training tasks can be, for example, text named entity recognition, text classification and other tasks.

[0079] Among them, the pre-trained model obtained after fine-tuning based on each text block contained in the long text to be processed by adopting the Transformer structure can be, after setting the training task, using the parameters of the pre-trained model with the Transformer structure as the initialization parameters of the model, and further training the model of the set training task, and continuously adjusting the parameters of the model during the training process to obtain the pre-trained model after fine-tuning.

[0080] Exemplarily, the training task is a text classification task, and the pre-trained model of the Transformer structure is a BERT pre-trained model. The process of obtaining the pre-trained model after fine-tuning can be: setting a label for the long text to be processed, and setting the same label for multiple text blocks obtained by segmenting the long text to be processed, where the label can represent the category of the text, etc. Then, using the parameters of the BERT pre-trained model as the initialization parameters, the text blocks contained in the long text to be processed and their corresponding labels are input into the BERT pre-trained model for training, and the parameters of the BERT pre-trained model are fine-tuned according to the loss of the model. The training ends when the loss of the model reaches a set threshold, and the fine-tuned BERT pre-trained model is obtained, that is, a pre-trained model obtained by adopting a Transformer structure and fine-tuning the text blocks contained in the long text to be processed. Then, the BERT pre-trained model obtained after fine-tuning can be used to encode the text blocks to obtain the feature vectors corresponding to each text block.

[0081] In an embodiment of the present invention, a Transformer structure is adopted, and a pre-trained model is obtained after fine-tuning each text block contained in a long text to encode the text block, so that the encoding of each text block is more accurate, and the feature vector corresponding to each text block obtained can more accurately represent the semantic feature information corresponding to each text block.

[0082] S103, obtaining candidate feature vectors.

[0083] In an embodiment of the present invention, obtaining candidate feature vectors from the electronic device can be done before segmenting the long text to be processed, or can be done simultaneously with processing the long text to be processed to obtain feature vectors corresponding to each text block, that is, the process of obtaining candidate feature vectors can be obtained by offline calculation or online calculation, and the embodiment of the present invention is not limited to this.

[0084] The candidate feature vector is obtained by encoding the target text. Optionally, the target text may be a text in a target application scenario, or a long text to be processed in an embodiment of the present invention.

[0085] In one implementation, the candidate feature vector may be obtained by encoding the text in the target application scenario. It is understandable that the target application scenario may be determined based on the actual task processing requirements for the long text. For example, the target application scenario may be a classification processing scenario for the long text, or an information extraction processing scenario for the long text, etc. The text in the target application scenario may be the text in the target application scenario pre-set by a person skilled in the art, and is used to perform attention interaction with the feature vector corresponding to the text block in the embodiment of the present invention to realize the processing of the long text.

[0086] In one implementation, the candidate feature vectors may also be obtained by encoding the long text to be processed, and used for attention interaction with the feature vectors corresponding to the text blocks in the embodiments of the present invention to implement processing of the long text.

[0087] S104, calculating a weight value corresponding to each text block based on the feature vector corresponding to each text block and the candidate feature vectors.

[0088] After obtaining the feature vectors and candidate feature vectors corresponding to the text blocks, attention interaction can be performed on the feature vectors and candidate feature vectors corresponding to each text block to calculate the weight value corresponding to each text block. The weight value is used to represent the semantic contribution of the text block to the long text.

[0089] As an optional implementation of the embodiment of the present invention, the implementation process of calculating the weight value corresponding to each text block based on the feature vector corresponding to each text block and the candidate feature vector may include:

[0090] For the feature vector corresponding to each text block, the feature vector corresponding to the text block is interactively calculated with the candidate feature vector, and the weight value corresponding to the text block is obtained based on the interactive result.

[0091] For the feature vector corresponding to each text block, the feature vector corresponding to the text block is interactively calculated with the candidate feature vector to realize the interaction between the text block and the target text, and then the semantic contribution of each text block to the long text is obtained based on the interactive result. Specifically, the interactive calculation method can be determined according to the specific task requirements for long text processing. For example, if the task requirement for long text processing is text classification or information extraction, the corresponding interactive calculation method can be: dot product calculation of the feature vector corresponding to the text block and the candidate feature vector, etc.

[0092] As an optional implementation of the embodiment of the present invention, the implementation process of obtaining the weight value corresponding to the text block based on the interaction result may include:

[0093] The interaction results are normalized, and the normalized results are determined as the weight values ​​corresponding to the text blocks.

[0094] After interactively calculating the feature vector corresponding to the text block and the candidate feature vector to obtain the interactive calculation result, the interactive result can be further normalized, and the normalized result is determined as the weight value corresponding to the text block.

[0095] Exemplarily, a softmax function can be used to normalize the interaction results between the feature vector corresponding to each text block and the candidate feature vector. The weight value corresponding to each text block obtained after normalization is a value between 0 and 1, and the sum of the weight values ​​corresponding to each text block is 1.

[0096] In the embodiment of the present invention, the interaction results are normalized and the weight value corresponding to each text block is mapped to between 0 and 1, thereby avoiding the calculated weight value being too large and increasing the calculation burden during the long text processing.

[0097] S105, based on the feature vectors corresponding to each text block and the weight values ​​corresponding to each text block, a feature vector of the long text is obtained, and the feature vector of the long text is used as a processing result of the long text.

[0098] After obtaining the feature vectors corresponding to each text block and the weight values ​​corresponding to each text block, as an optional implementation of the embodiment of the present invention, Figure 2 As shown, the feature vector of the long text can be obtained by the following steps:

[0099] S201, respectively calculating the product of the feature vector corresponding to each text block and the weight value corresponding to the text block, to obtain the target feature vector corresponding to the text block.

[0100] After obtaining the feature vector corresponding to each text block and the weight value corresponding to each text block, the product of the feature vector corresponding to the text block and the weight value corresponding to the text block can be calculated for each text block to obtain the target feature vector corresponding to the text block, and then the target feature vector corresponding to each text block can be obtained.

[0101] S202, summing up the target feature vectors corresponding to each text block, and determining the summation result as the feature vector of the long text.

[0102] The target feature vectors corresponding to each text block are summed up to retain the semantics contained in each text block, and the summation result is determined as the feature vector of the long text, and the feature vector of the long text is used as the processing result of the long text, so that the processing result of the long text contains the semantics contained in each text block and can be better applied to downstream tasks.

[0103] The embodiment of the present invention provides a method for processing a long text. Based on a preset text processing model, the long text to be processed is divided into a plurality of text blocks of a specific length, that is, the long text to be processed is converted into a short text that can be processed by the preset text processing model for processing. The method can adapt to the processing of the long text by the text processing model with a limited input length. Based on the feature vector corresponding to each text block and the candidate feature vector, the semantic contribution of each text block to the long text is calculated, and the semantic interaction between each text block and the target text is realized, and the needs of downstream tasks are taken into consideration. Furthermore, based on the feature vector corresponding to each text block and the weight value corresponding to each text block, the feature vector of the long text is obtained, and the feature vector of the long text is used as the processing result of the long text, thereby realizing the processing of the long text.

[0104] In the embodiment of the present invention, the process of obtaining the candidate feature vector can be completed offline or online. As an optional implementation of the embodiment of the present invention, the candidate feature vector can be obtained in the following manner:

[0105] When the text in the target application scenario is a short text, the text in the target application scenario is encoded to obtain a candidate feature vector.

[0106] In the case where the text in the target application scenario is a short text, the pre-trained model can be directly used to encode the text in the target application scenario to obtain a candidate feature vector. In an embodiment of the present invention, a short text is a text whose length is not greater than a preset length, and the preset length can be determined based on the maximum text length that can be processed by a preset text processing model. Among them, the preset text processing model can be determined according to actual needs, and can realize a text processing model for text processing tasks in different scenarios. The text processing model has certain restrictions on the length of the input text. Exemplarily, a text processing model based on a Transformer structure can process a text with a maximum length of 512 words, so a text with a length of no more than 512 words can be determined as a short text.

[0107] For an implementation method of using a pre-trained model to encode text in a target application scenario to obtain a candidate feature vector, refer to the implementation method of using a pre-trained model to encode a text block to obtain a feature vector corresponding to the text block, which will not be described in detail in the embodiments of the present invention.

[0108] As an optional implementation of the embodiment of the present invention, Figure 3 As shown, the candidate feature vectors can also be obtained in the following way:

[0109] S301, when the text in the target application scenario is a long text, the text in the target application scenario is divided into blocks to obtain a plurality of target text blocks.

[0110] When the text in the target application scenario is a long text, the text in the target application scenario can be divided into blocks. The implementation method of dividing the text in the target application scenario into blocks can refer to the implementation method of dividing the long text into multiple text blocks above, and the embodiment of the present invention will not be repeated here.

[0111] S302, encoding each target text block to obtain a feature vector corresponding to the target text block.

[0112] For each target text block, the target text block can be encoded using a pre-trained model to obtain a feature vector corresponding to the target text block. For the implementation method of encoding the target text block using a pre-trained model to obtain a feature vector corresponding to the target text block, please refer to the implementation method of encoding the text block using a pre-trained model to obtain a feature vector corresponding to the text block above, and the embodiments of the present invention will not be repeated here.

[0113] S303, performing aggregation processing on the feature vectors corresponding to each target text block to obtain a candidate feature vector.

[0114] After the feature vectors corresponding to the target text blocks are obtained, the feature vectors corresponding to the target text blocks may be aggregated to obtain candidate feature vectors that can represent the semantic feature information of the text in the target application scenario.

[0115] As an optional implementation of the embodiment of the present invention, the feature vectors corresponding to each target text block are aggregated to obtain candidate feature vectors, including:

[0116] The feature vectors corresponding to each target text block are concatenated to obtain a candidate feature vector;

[0117] Alternatively, the candidate feature vectors are determined by accumulating and summing the feature vectors corresponding to each target text block;

[0118] Alternatively, the weighted sum vector of the feature vectors corresponding to each target text block is used to determine the candidate feature vector.

[0119] The concatenation process of the feature vectors corresponding to the target text blocks may be to concatenate the feature vectors corresponding to the target text blocks head to tail according to the order of the target text blocks in the text in the target application scenario. For example, the dimension of the feature vector corresponding to a target text block is 1×768, and the dimension of the candidate feature vector obtained by concatenating the feature vectors corresponding to N target text blocks is N×768.

[0120] The feature vectors corresponding to each target text block are aggregated to avoid using the feature information of a single text block to represent the feature information of the text in the target application scenario, so that the obtained candidate feature vector information is more complete.

[0121] As an optional implementation of the embodiment of the present invention, the candidate feature vector may also be obtained in the following manner:

[0122] The long text is divided into blocks to obtain multiple text blocks; for each text block, the text block is encoded to obtain the feature vector corresponding to the text block; the feature vectors corresponding to each text block are aggregated to obtain a candidate feature vector.

[0123] The implementation method of obtaining the candidate feature vector can be found in the above Figure 3 The implementation method of obtaining the candidate feature vector is shown, and the embodiment of the present invention will not be repeated here.

[0124] The embodiment of the present invention provides a method for processing a long text. Based on a preset text processing model, the long text to be processed is divided into a plurality of text blocks of a specific length, that is, the long text to be processed is converted into a short text that can be processed by the preset text processing model for processing. The method can adapt to the processing of the long text by the text processing model with a limited input length. Based on the feature vector corresponding to each text block and the candidate feature vector, the semantic contribution of each text block to the long text is calculated, and the semantic interaction between each text block and the target text is realized, and the needs of downstream tasks are taken into consideration. Furthermore, based on the feature vector corresponding to each text block and the weight value corresponding to each text block, the feature vector of the long text is obtained, and the feature vector of the long text is used as the processing result of the long text, thereby realizing the processing of the long text.

[0125] For example, Figure 4 As shown, Figure 4 A schematic diagram of an architecture for an implementation method of determining a long text feature vector provided by an embodiment of the present invention.

[0126] For the long text to be processed, the long text can be divided into multiple non-intersecting text blocks to obtain multiple text blocks, such as Figure 4 In the text block 1, text block 2, ..., text block N. Then, for each text block, the pre-trained model is used to encode the text block to obtain the feature vector corresponding to each text block, such as Figure 4 The feature vector corresponding to text block 1, the feature vector corresponding to text block 2, ..., the feature vector corresponding to text block N.

[0127] Furthermore, when the target text, that is, the text in the target application scenario, is a long text, the target text is divided into blocks, and the target text is divided into multiple non-intersecting target text blocks to obtain multiple target text blocks, such as Figure 4 In the target text block 1, target text block 2, ..., target text block M. Then, for each target text block, the pre-trained model is used to encode the target text block to obtain the feature vector corresponding to each target text block, such as Figure 4 The feature vector corresponding to the target text block 1, the feature vector corresponding to the target text block 2, ..., the feature vector corresponding to the target text block M. Then, the feature vectors corresponding to each target text block are aggregated to obtain a candidate feature vector.

[0128] Furthermore, for the feature vector corresponding to each text block, the feature vector corresponding to the text block is interactively calculated with the candidate feature vector, and the interaction result is normalized, and the normalized result is determined as the weight value corresponding to the text block, that is, the attention interaction between the feature vector corresponding to each text block and the candidate feature vector.

[0129] Then, the feature vector corresponding to each text block is calculated, and the product of the feature vector and the weight value corresponding to the text block is obtained to obtain the target feature vector corresponding to the text block. The target feature vectors corresponding to each text block are summed up, and the summation result is determined as the feature vector of the long text.

[0130] With respect to the above-mentioned embodiments of the present invention, on the one hand, compared with directly truncating the long text, that is, retaining only the first preset length of words of the long text, it is easy to cause the loss of text or the loss of part of the semantics in the text. The long text processing method adopted in the embodiment of the present invention, based on a preset text processing model, divides the long text to be processed into multiple text blocks of specific lengths, and then calculates the semantic contribution of each text block to the long text based on the feature vector corresponding to each text block and the candidate feature vector, thereby realizing the semantic interaction between the target texts of each text block, and further, based on the feature vector corresponding to each text block and the weight value corresponding to each text block, obtains the feature vector of the long text, and uses the feature vector of the long text as the processing result of the long text, thereby retaining the semantics in each text block in the long text. On the basis of realizing the processing of the long text, the processing result of the long text includes the semantics contained in each text block, which can be better applied to downstream tasks.

[0131] On the other hand, based on the Transformer structure, Longformer, as a super-transformation form of Transformer, can solve the problems of memory explosion and low efficiency in document-level tasks.

[0132] Traditional Tranformer-based models have a natural disadvantage when processing long texts. This is because traditional models use a "fully connected" attention mechanism, that is, every word, term, or token must interact with all other tokens, making the attention complexity as high as o(n 2 ).

[0133] Longformer improves the traditional attention mechanism of Transformer: for each token, only local attention is calculated for nearby tokens of a fixed window size, and a small amount of global attention is calculated in combination with specific tasks.

[0134] Longformer provides three attention modes, namely the sliding window mechanism, the expansion sliding window mechanism, and the sliding window mechanism that integrates global information. In the sliding window mechanism, for each token, attention is only calculated for the w tokens near it, and the computational complexity is linearly related to the length of the text sequence. In the expansion sliding window mechanism, there will be a gap of size d between two adjacent tokens that are attended to in the sliding window. In the sliding window mechanism that integrates global information, a special token is added before the text sequence, and a small amount of global attention is added on the basis of local attention. For the token with global attention added, attention is paid to the entire sequence when encoding it, and it is also attended to when encoding all other tokens.

[0135] However, no matter it is the sliding window mechanism, the expanding sliding window mechanism, or the sliding window mechanism integrating global information, they all concentrate the attention calculation of long texts within a certain window range. Then, correspondingly, during each sliding process, there will inevitably be text overlap in the calculation of attention, which will cause a certain overlap in the semantics of texts in different sliding windows, resulting in a large amount of text processing calculations and the acquired semantics may be inaccurate.

[0136] The long text processing method adopted in the embodiment of the present invention, based on a preset text processing model, divides the long text to be processed into multiple non-intersecting text blocks of specific lengths, thereby avoiding the cross-overlap of text between the text blocks. Then, based on the feature vector corresponding to each text block and the candidate feature vector, the semantic contribution of each text block to the long text is calculated, thereby achieving semantic interaction between each text block and the target text, and improving the accuracy of long text semantic extraction. Furthermore, based on the feature vector corresponding to each text block and the weight value corresponding to each text block, the feature vector of the long text is obtained, and the feature vector of the long text is used as the processing result of the long text, thereby retaining the semantics of each text block in the long text. On the basis of realizing the processing of the long text, the processing result of the long text includes the semantics contained in each text block, which can be better applied to downstream tasks.

[0137] Corresponding to the above method embodiments, the embodiments of the present invention also provide corresponding device embodiments.

[0138] like Figure 5 As shown, an embodiment of the present invention provides a long text processing device, which may include:

[0139] The segmentation module 501 is used to obtain a long text to be processed and segment the long text into multiple text blocks, and the length of each text block is determined according to a preset text processing model;

[0140] The encoding module 502 is used to encode each text block to obtain a feature vector corresponding to the text block;

[0141] An acquisition module 503 is used to acquire a candidate feature vector, where the candidate feature vector is obtained by encoding the target text;

[0142] A calculation module 504, used to calculate a weight value corresponding to each text block based on the feature vector corresponding to each text block and the candidate feature vector;

[0143] The determination module 505 is used to obtain the feature vector of the long text based on the feature vector corresponding to each text block and the weight value corresponding to each text block, and use the feature vector of the long text as the processing result of the long text.

[0144] The embodiment of the present invention provides a long text processing device, which divides the long text to be processed into multiple text blocks of specific lengths based on a preset text processing model, that is, converts the long text to be processed into short texts that can be processed by the preset text processing model for processing, and can adapt to the processing of long texts by text processing models with limited input length. Based on the feature vector corresponding to each text block and the candidate feature vector, the semantic contribution of each text block to the long text is calculated, and the semantic interaction between each text block and the target text is realized, and the needs of downstream tasks are taken into consideration. Further, based on the feature vector corresponding to each text block and the weight value corresponding to each text block, the feature vector of the long text is obtained, and the feature vector of the long text is used as the processing result of the long text, thereby realizing the processing of the long text.

[0145] Optionally, the calculation module 504 includes:

[0146] The first calculation submodule is used to interactively calculate the feature vector corresponding to each text block with the candidate feature vectors, and obtain the weight value corresponding to the text block based on the interactive result.

[0147] Optionally, the first calculation submodule is specifically used to:

[0148] The interaction results are normalized, and the normalized results are determined as the weight values ​​corresponding to the text blocks.

[0149] Optionally, the determination module 505 includes:

[0150] The second calculation submodule is used to calculate the product of the feature vector corresponding to each text block and the weight value corresponding to the text block to obtain the target feature vector corresponding to the text block;

[0151] The determination submodule is used to sum the target feature vectors corresponding to each text block and determine the sum result as the feature vector of the long text.

[0152] Optionally, the encoding module 502 is specifically used to:

[0153] For each text block, the pre-trained model is used to encode the text block to obtain the feature vector corresponding to the text block.

[0154] Optionally, the target text is: a text in a target application scenario, or a long text; the acquisition module 503 includes:

[0155] A first acquisition submodule is used to encode the text in the target application scenario to obtain a candidate feature vector when the text in the target application scenario is a short text;

[0156] The second acquisition submodule is used for, when the text in the target application scenario is a long text, dividing the text in the target application scenario into blocks to obtain multiple target text blocks; for each target text block, encoding the target text block to obtain a feature vector corresponding to the target text block; performing aggregation processing on the feature vectors corresponding to each target text block to obtain a candidate feature vector;

[0157] The third acquisition submodule is used to divide the long text into blocks to obtain multiple text blocks; for each text block, encode the text block to obtain the feature vector corresponding to the text block; aggregate the feature vectors corresponding to each text block to obtain a candidate feature vector.

[0158] Optionally, the feature vectors corresponding to the target text blocks are aggregated to obtain candidate feature vectors, including:

[0159] The feature vectors corresponding to each target text block are concatenated to obtain a candidate feature vector;

[0160] Alternatively, the candidate feature vectors are determined by accumulating and summing the feature vectors corresponding to each target text block;

[0161] Alternatively, the weighted sum vector of the feature vectors corresponding to each target text block is used to determine the candidate feature vector.

[0162] The embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604.

[0163] Memory 603, used for storing computer programs;

[0164] The processor 601 is used to execute the program stored in the memory 603 to implement the following steps:

[0165] Obtain a long text to be processed, and divide the long text into multiple text blocks, where the length of each text block is determined according to a preset text processing model;

[0166] For each text block, the text block is encoded to obtain a feature vector corresponding to the text block to obtain a candidate feature vector, where the candidate feature vector is obtained by encoding the target text;

[0167] Based on the feature vector corresponding to each text block and the candidate feature vectors, a weight value corresponding to each text block is calculated;

[0168] Based on the feature vectors corresponding to each text block and the weight values ​​corresponding to each text block, a feature vector of the long text is obtained, and the feature vector of the long text is used as a processing result of the long text.

[0169] An electronic device provided by an embodiment of the present invention divides a long text to be processed into multiple text blocks of specific lengths based on a preset text processing model, that is, converts the long text to be processed into short text that can be processed by the preset text processing model for processing, and can adapt to the processing of long text by a text processing model with a limited input length. Based on the feature vector corresponding to each text block and the candidate feature vector, the semantic contribution of each text block to the long text is calculated, and the semantic interaction between each text block and the target text is realized, and the needs of downstream tasks are taken into consideration. Further, based on the feature vector corresponding to each text block and the weight value corresponding to each text block, the feature vector of the long text is obtained, and the feature vector of the long text is used as the processing result of the long text, thereby realizing the processing of the long text.

[0170] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0171] The communication interface is used for communication between the above electronic device and other devices.

[0172] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0173] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0174] In another embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the following steps are implemented:

[0175] Obtain a long text to be processed, and divide the long text into multiple text blocks, where the length of each text block is determined according to a preset text processing model;

[0176] For each text block, the text block is encoded to obtain a feature vector corresponding to the text block to obtain a candidate feature vector, where the candidate feature vector is obtained by encoding the target text;

[0177] Based on the feature vector corresponding to each text block and the candidate feature vectors, a weight value corresponding to each text block is calculated;

[0178] Based on the feature vectors corresponding to each text block and the weight values ​​corresponding to each text block, a feature vector of the long text is obtained, and the feature vector of the long text is used as a processing result of the long text.

[0179] A computer-readable storage medium provided by an embodiment of the present invention divides a long text to be processed into multiple text blocks of specific lengths based on a preset text processing model, that is, converts the long text to be processed into short text that can be processed by the preset text processing model for processing, and can adapt to the processing of long text by a text processing model with a limited input length. Based on the feature vector corresponding to each text block and the candidate feature vector, the semantic contribution of each text block to the long text is calculated, and the semantic interaction between each text block and the target text is realized, and the needs of downstream tasks are taken into consideration. Further, based on the feature vector corresponding to each text block and the weight value corresponding to each text block, the feature vector of the long text is obtained, and the feature vector of the long text is used as the processing result of the long text, thereby realizing the processing of the long text.

[0180] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, causes the computer to execute the following steps:

[0181] Obtain a long text to be processed, and divide the long text into multiple text blocks, where the length of each text block is determined according to a preset text processing model;

[0182] For each text block, the text block is encoded to obtain a feature vector corresponding to the text block to obtain a candidate feature vector, where the candidate feature vector is obtained by encoding the target text;

[0183] Based on the feature vector corresponding to each text block and the candidate feature vectors, a weight value corresponding to each text block is calculated;

[0184] Based on the feature vectors corresponding to each text block and the weight values ​​corresponding to each text block, a feature vector of the long text is obtained, and the feature vector of the long text is used as a processing result of the long text.

[0185] The computer program product including instructions provided by the embodiment of the present invention divides a long text to be processed into a plurality of text blocks of specific lengths based on a preset text processing model, that is, converts the long text to be processed into short texts that can be processed by the preset text processing model for processing, and can adapt to the processing of long texts by text processing models with limited input length. Based on the feature vector corresponding to each text block and the candidate feature vector, the semantic contribution of each text block to the long text is calculated, and the semantic interaction between each text block and the target text is realized, and the needs of downstream tasks are taken into consideration. Furthermore, based on the feature vector corresponding to each text block and the weight value corresponding to each text block, the feature vector of the long text is obtained, and the feature vector of the long text is used as the processing result of the long text, thereby realizing the processing of the long text.

[0186] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0187] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0188] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device / electronic device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0189] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for processing long text, characterized in that: The method comprises: Acquire a long text to be processed, and divide the long text into a plurality of text blocks, wherein the length of each text block is determined according to a preset text processing model; said dividing the long text into a plurality of text blocks includes: dividing the long text into a plurality of mutually non-intersecting text blocks, or dividing the long text into a plurality of text blocks in a sliding window manner; For each text block, encode the text block to obtain a feature vector corresponding to the text block; Obtaining a candidate feature vector, where the candidate feature vector is obtained by encoding the target text; Based on the feature vector corresponding to each of the text blocks and the candidate feature vectors, a weight value corresponding to each of the text blocks is calculated; the weight value is used to represent the semantic contribution of the text block to the long text; Based on the feature vectors corresponding to each text block and the weight values ​​corresponding to each text block, the feature vector of the long text is obtained, and the feature vector of the long text is used as the processing result of the long text.

2. The method according to claim 1, characterized in that The calculating the weight value corresponding to each text block based on the feature vector corresponding to each text block and the candidate feature vector includes: For each feature vector corresponding to the text block, the feature vector corresponding to the text block is interactively calculated with the candidate feature vector, and a weight value corresponding to the text block is obtained based on the interactive result.

3. The method according to claim 2, characterized in that The obtaining of the weight value corresponding to the text block based on the interaction result includes: The interaction result is normalized, and the normalized result is determined as the weight value corresponding to the text block.

4. The method according to claim 1, characterized in that: The step of obtaining the feature vector of the long text based on the feature vector corresponding to each text block and the weight value corresponding to each text block includes: Calculate the product of the feature vector corresponding to each text block and the weight value corresponding to the text block to obtain the target feature vector corresponding to the text block; The target feature vectors corresponding to the text blocks are summed, and the sum result is determined as the feature vector of the long text.

5. The method according to claim 1, characterized in that: For each text block, encoding the text block to obtain a feature vector corresponding to the text block includes: For each text block, the pre-trained model is used to encode the text block to obtain a feature vector corresponding to the text block.

6. The method according to claim 1, characterized in that The target text is: the text in the target application scenario, or the long text; the obtaining of the candidate feature vector includes: In the case where the text in the target application scenario is a short text, encoding the text in the target application scenario to obtain the candidate feature vector; Alternatively, when the text in the target application scenario is a long text, the text in the target application scenario is divided into blocks to obtain a plurality of target text blocks; for each target text block, the target text block is encoded to obtain a feature vector corresponding to the target text block; the feature vectors corresponding to each of the target text blocks are aggregated to obtain the candidate feature vector; Alternatively, the long text is divided into blocks to obtain a plurality of text blocks; for each text block, the text block is encoded to obtain a feature vector corresponding to the text block; the feature vectors corresponding to each of the text blocks are aggregated to obtain the candidate feature vector.

7. The method according to claim 6, characterized in that The step of aggregating the feature vectors corresponding to the target text blocks to obtain the candidate feature vectors includes: Concatenating the feature vectors corresponding to the target text blocks to obtain the candidate feature vectors; Alternatively, the candidate feature vector is determined by accumulating and summing the feature vectors corresponding to each target text block; Alternatively, the candidate feature vector is determined by taking the weighted sum vector of the feature vectors corresponding to each target text block.

8. A long text processing device, characterized in that: The device comprises: A segmentation module, used for obtaining a long text to be processed and segmenting the long text into a plurality of text blocks, wherein the length of each text block is determined according to a preset text processing model; The encoding module is used to encode each text block to obtain a feature vector corresponding to the text block; An acquisition module, used for acquiring a candidate feature vector, wherein the candidate feature vector is obtained by encoding the target text; A calculation module, used for calculating a weight value corresponding to each text block based on the feature vector corresponding to each text block and the candidate feature vector; the weight value is used to represent the semantic contribution of the text block to the long text; A determination module, used for obtaining a feature vector of the long text based on the feature vector corresponding to each text block and the weight value corresponding to each text block, and taking the feature vector of the long text as a processing result of the long text; The segmentation module is specifically used to segment the long text into a plurality of mutually non-intersecting text blocks, or to segment the long text into a plurality of text blocks in a sliding window manner.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

Images