Long text processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a long text processing method and device, equipment and a storage medium, and the method comprises the steps: obtaining a long text to be processed, segmenting the long text into a plurality of text blocks, and determining the length of each text block according to a preset text processing model; for each text block, encoding the text block to obtain a feature vector corresponding to the text block; obtaining a candidate feature vector, wherein the candidate feature vector is obtained by encoding the target text; calculating a weight value corresponding to each text block based on the feature vector corresponding to each text block and the candidate feature vector; and obtaining a feature vector of the long text based on the feature vector corresponding to each text block and the weight value corresponding to each text block, and taking the feature vector of the long text as a processing result of the long text. According to the embodiment of the invention, the long text can be processed.

Description

technical field [0001] The present invention relates to the technical field of natural language processing, in particular to a long text processing method, device, equipment and storage medium. Background technique [0002] With the development of the field of artificial intelligence, Natural Language Processing technology (Natural Language Processing, NLP) has been widely used in many scenarios, such as sentiment analysis, text similarity calculation, comment opinion extraction, text classification, lexical analysis and other scenarios. In these natural language processing scenarios, text processing is required. At present, most text processing models are trained or adjusted based on the trained pre-training model. Specifically, based on the pre-training model, fine-tune the parameters of the pre-training model to train downstream tasks to achieve text in different scenarios. The processing model is used to process the text, and the downstream tasks can be, for example, te...

AI Technical Summary

Problems solved by technology

However, the current Transformer-based pre-training model has limitations on the length of text input into the model at a single time. For example, for a 12-layer or 24-layer Transformer structure, the length of text processed at a time does not exceed 512 words. Therefore, for some needs For NLP tasks dealing with long text, the length of 512 cannot meet the demand

Images