Text processing method, device, computer equipment and computer readable storage medium

Abstract

The embodiment of the present application provides a text processing method, device, computer equipment and computer-readable storage medium. The text processing method is based on artificial intelligence technology and includes: obtaining a text to be processed, the text to be processed includes a text title, text keywords and text body; inputting the text to be processed into a long text recognition model for processing to obtain a target result, the target result is used to indicate the practicality category of the text to be processed; wherein the long text recognition model is obtained by pre-training an initial text recognition model using first text data and fine-tuning the pre-trained text recognition model using second text data; the first text data includes an incomplete text body, and the second text data includes a sample text title, sample text keywords, sample text body and corresponding reference practicality category labels. The embodiment of the present application can effectively improve the accuracy and robustness of the practicality recognition of long texts at the chapter level.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text processing method, apparatus, computer equipment, and computer-readable storage medium. Background Art

[0002] At present, the recognition of whether a picture and text are practical can be divided into two categories. One is the content quality judgment based on the text perspective, which belongs to supervised learning, and the other is the construction of atomic capabilities (i.e. fine-grained features) based on comments and keywords, which belongs to unsupervised learning. However, the above methods all have certain defects. The former has restrictions on the number of words included in the picture and text content. In the long text practicality scenario, the processing efficiency of articles with long text length (such as more than 512 characters) is not high, and the recognition accuracy is not ideal. The latter does not introduce the semantic information of the article, so it is easy to make wrong judgments in some special language scenarios and has low robustness. Summary of the invention

[0003] The embodiments of the present application provide a text processing method, apparatus, computer device and computer-readable storage medium, which can effectively improve the accuracy and robustness of practicality recognition of long text at the paragraph level.

[0004] An embodiment of the present application provides a text processing method, including:

[0005] Acquire a text to be processed, wherein the text to be processed includes a text title, text keywords and text body;

[0006] Inputting the text to be processed into a long text recognition model for processing to obtain a target result, wherein the target result is used to indicate the practicality category of the text to be processed;

[0007] Among them, the long text recognition model is obtained by pre-training the initial text recognition model with the first text data and then fine-tuning the pre-trained text recognition model with the second text data; the first text data includes an incomplete text body, and the second text data includes a sample text title, sample text keywords, sample text body and corresponding reference practicality category labels.

[0008] An embodiment of the present application provides a text processing device, including:

[0009] An acquisition module is used to acquire the text to be processed, wherein the text to be processed includes a text title, text keywords and text body;

[0010] A processing module, used for inputting the text to be processed into a long text recognition model for processing to obtain a target result, wherein the target result is used to indicate the practicality category of the text to be processed;

[0011] Among them, the long text recognition model is obtained by pre-training the initial text recognition model with the first text data and then fine-tuning the pre-trained text recognition model with the second text data; the first text data includes an incomplete text body, and the second text data includes a sample text title, sample text keywords, sample text body and corresponding reference practicality category labels.

[0012] On the one hand, an embodiment of the present application provides a computer device, including: a processor and a memory; the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the text processing method provided by the embodiment of the present application.

[0013] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the text processing method provided by the embodiment of the present application is executed.

[0014] Accordingly, the embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the text processing method provided in the embodiment of the present application.

[0015] The embodiment of the present application utilizes the long text recognition model obtained after pre-training and fine-tuning to process the three features of the title, keywords and text of the text to be processed, which can improve the accuracy of the target result indicating the practicality category. Specifically, through the pre-training of the incomplete text body included in the sample corpus data, the pre-trained text recognition model has a stronger semantic understanding ability for the long text at the chapter level, so as to achieve a more accurate feature expression of the long text. After fine-tuning the sample data of the title, keywords, text and practicality labels of the text, the long text recognition model is obtained to process the input text to be processed, so that the predicted text practicality recognition result is more reliable, thereby effectively improving the robustness and accuracy of the practicality recognition of the long text at the chapter level. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 is an architecture diagram of a text processing system provided in an embodiment of the present application;

[0018] Figure 2 It is a flowchart of a text processing method provided in an embodiment of the present application;

[0019] Figure 3 It is a schematic diagram of a representation vector corresponding to a text sequence for data selection sliding window processing provided in an embodiment of the present application;

[0020] Figure 4 This is a schematic diagram of the effect of a sparse self-attention mechanism provided in an embodiment of the present application;

[0021] Figure 5 It is a flowchart of another text processing method provided in an embodiment of the present application;

[0022] Figure 6 It is a flowchart of another text processing method provided in an embodiment of the present application;

[0023] Figure 7 It is a structural diagram of a self-attention mechanism provided in an embodiment of the present application;

[0024] Figure 8 is a schematic diagram of the structure of an encoder provided in an embodiment of the present application;

[0025] Fig. 9 It is a structural schematic diagram of a graphic and text practicality recognition model provided in an embodiment of the present application;

[0026] Fig.10 is a structural schematic diagram of a text processing device provided in an embodiment of the present application;

[0027] Fig.11 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] The following is an introduction to some key terms involved in the embodiments of the present application:

[0030] Atomic capabilities of image and text practicality: The construction of image and text atomic capabilities is mainly to explore some fine-grained high-quality atomic features, such as building atomic capabilities such as positive energy and practicality on the image and text side, and building atomic capabilities such as fun on the video side. The atomic capabilities of image and text practicality are a fine-grained feature description of image and text attributes. The construction of atomic capabilities is mainly to refine and deeply understand high-quality content, and at the same time split fine-grained high-quality atomic feature capabilities to facilitate the recommendation of better use of fine-grained atomic capability features on the content understanding side.

[0031] HAN model: The Hierarchical Attention Network (HAN) model has good classification accuracy in long text classification tasks. The overall structure of the model is as follows: Input word vector sequence w 2x After passing through the word-level bidirectional gated recurrent unit (Bi-GRU), each word will have a corresponding hidden vector output by the Bi-GRU, and then through u w The dot product of the vector and each vector is used to obtain the attention weight, and then the h sequence is weighted and summed according to the attention weight to obtain the sentence summary vector s2. Each sentence is passed through the same Bi-GRU structure and attention is added to obtain the final output document feature vector v vector, and then the v vector is passed through the back-end fully connected dense layer and the classifier to obtain the final text classification result. In summary, the HAN model structure is very consistent with the human understanding process from word -> sentence -> then to paragraph. It not only solves the problem of TextCNN, a text classification model based on convolutional neural network, losing text structure information, but also has strong interpretability.

[0032] Transformer model: Transformer is an encoder-decoder neural network model composed of multiple self-attention neural network layers. The model is divided into two parts: encoder and decoder. The encoder part consists of 6 identical encoders stacked together, and the decoder part consists of 6 identical decoders stacked together. The encoders do not share parameters. At the same time, positional encoding is performed before the word vector representation is sent to the encoder and decoder. The encoder consists of N = 6 identical layers, each layer consists of two sub-layers, namely the multi-head self-attention mechanism and the fully connected feed-forward network. Each sub-layer has a residual connection and layer normalization. The two sub-layers are Multi-head self-attention and Position-wise feed-forward networks in order: Multi-head self-attention projects Q, K, and V through h different linear transformations, and finally concatenates the different attention results. Position-wise feed-forward networks mainly provide nonlinear transformations. The dimension of the attention output is [bsz*seq_len, num_heads*head_size]. The second sub-layer is a fully connected layer. The reason why it is position-wise is that the transformation parameters of each position i are the same when passing through the linear layer. The structure of the Decoder is similar to that of the Encoder, but there is an additional attention sub-layer. The main difference here is that the attention in the Encoder has an additional mask, because the output during training is the real label ground truth, which ensures that future information will not be exposed when predicting the i-th position.

[0033] BERT model: BERT is a new language representation model, which stands for Bidirectional Encoder Representations from Transformers (BERT). When multiple Transformer Encoders are stacked layer by layer, BERT is assembled. BERT aims to pre-train deep bidirectional representations based on the left and right context of all layers. Therefore, the pre-trained BERT representation can be fine-tuned with only an additional output layer to create the current optimal model for many tasks (such as question answering and language reasoning) without making a lot of modifications to the task-specific architecture. BERT is simple in concept but powerful in experimental results, breaking the current state-of-the-art results on 11 natural language processing (NLP) tasks, including raising the GLUE benchmark for general language understanding evaluation to 80.4% (7.6% absolute improvement), raising the accuracy of MultiNLI (a multilingual natural language inference corpus) to 86.7% (5.6% absolute improvement), and raising the SQuADv1.1 (a dataset with a very wide audience for machine reading comprehension) question-answering test F1 score to 93.2 points (1.5 points absolute improvement) - 2.0 points higher than human performance.

[0034] LongFormer model: LongFormer is the full name of The Long-Document Transformer, which is the Transformer used in long text scenarios. In this solution, attention includes windowed local context selfattention and global attention activated by the terminal task, where local attention is used to establish local context representation, and global attention is used to establish a complete sequence representation for prediction. LongFormer adopts a combination of local self-attention and global self-attention (or sparse attention for short), that is, the attention mechanism "attention pattern" is used to sparse the complete self-attention matrix, and CUDA (a general parallel computing platform) is optimized for it, so that the model can accommodate texts of up to tens of thousands in length, while achieving better results, that is, using sparse self-attention to expand the model's text capacity.

[0035] In the existing text classification models, the HAN model uses a bidirectional RNN to aggregate information from two directions of words to obtain word annotations and the context information of sentences, and cannot train the model quickly and in parallel. The Transformer model improves the most criticized shortcoming of RNN, which is slow training. It uses the self-attention mechanism to achieve fast parallelism, and the Transformer can be increased to a very deep depth, fully exploring the characteristics of the DNN model and improving the model accuracy. The BERT model designed based on the Transformer model is also more effective in natural language processing tasks. It is worth mentioning that the Transformer architecture is very strong, and part of its success is that the self-attention mechanism enables the network to capture contextual information from the entire sequence. However, although the self-attention mechanism is very effective, the memory and computing power it requires will increase quadratically with the length of the sequence, which makes current hardware infeasible or very expensive to process long sequences. The existing method is to shorten or divide the context into smaller sequences to limit the length of these sequences to 512. Although the model can be processed in segments, the preprocessing of the data is also quite troublesome, and this division may cause important information to be lost.

[0036] Based on this, the embodiment of the present application provides a new text practicality identification scheme (ie, the text processing method provided in the embodiment of the present application) to improve the accuracy of text judgment on practicality in text classification tasks.

[0037] The text practicality recognition solution (text processing method) provided in this application is based on artificial intelligence technology. Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The text processing method provided in this application specifically relates to natural language processing technology and machine learning technology in artificial intelligence technology.

[0038] Among them, natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0039] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0040] The text processing method provided in the present application uses a long text recognition model to process the text to be processed, and establishes a representation vector belonging to the text to be processed from the three feature dimensions of the text title, keywords and text and processes it, so as to obtain a target result for indicating the text practicality category. Specifically, the long text recognition model is obtained after pre-training and fine-tuning using different sample text data. The long text recognition model is applied to downstream tasks, that is, the determination of text practicality categories, which can effectively improve the accuracy of practicality recognition in long text scenarios. In one embodiment, the long text recognition model can be a LongFormer model, the structure of which is based on the BERT model, keeping the original structure and parameters unchanged, and after pre-training, it is fine-tuned using the title, keywords, text of the sample text and the corresponding reference practicality labels of the sample text, and the LongFormer model uses sparse self-attention (i.e., a combination of global self-attention and local self-attention) to replace the traditional self-attention mechanism, and selectively associates all or part of the words in the text for the target task, so that the text capacity is well broadened, and a better recognition effect is achieved in the recognition scenario of long text practicality.

[0041] The text processing method provided in this application is applied to a text processing system. The architecture of the text processing system can be as follows: Figure 1 As shown in FIG. 1 , it is an architecture diagram of a text processing system provided by an embodiment of the present application. Figure 1As shown, the architecture may include: a processing end 100 , a user terminal 101 and a database 102 .

[0042] The processing end 100 can be a server or other forms of computer equipment, which is mainly used to obtain the title, keywords and text of the text to be processed from the database 102, and obtain the corresponding title vector, keyword vector and text vector. In one embodiment, the above three vectors can be fused to obtain the representation vector corresponding to the text to be processed, and then the representation vector can be processed. Specifically, it can be processed according to the processing logic in the LongFomer model to obtain the result of text practicality recognition, and then the text with practicality is sent to the user terminal 101 through recommendation processing. Optionally, the training process of the LongFomer model can also be executed at the processing end. Correspondingly, the database 102 can also store sample text data, including the text data required in the pre-training stage and the text data required in the fine-tuning stage, so as to achieve the optimization adjustment of the LongFomer model. The processing end 100 communicates with the user terminal 101 and the database 102 respectively through wired or wireless means to interact with data.

[0043] The user terminal 101 can be a desktop computer, a laptop computer or other intelligent terminal device, for example, it can refer to a device that can upload or browse graphic content, such as a smart phone, a tablet computer, or an intelligent wearable device. The user terminal 101 can be installed with a target application (for example, a community platform covering topics such as entertainment, emotions, and fitness), wherein the target application has the function of displaying data information such as text and pictures, and the user can upload the edited graphic content through the target application, or browse the graphic content posted by other users in the corresponding partitions of the platform (for example, daily life records, fitness, news, etc.). The graphic content uploaded by the user can be stored in the database 102 as the text to be processed, so that the processing end 100 can obtain it from it to perform practical recognition processing on the text to be processed.

[0044] The database 102 can be used to store the graphic and text content uploaded by the user, and provide the processing end 100 with the corresponding text to be processed, so that the processing end 100 processes the text to be processed and sends the practical text to the user terminal 101 according to the practicality category result after processing, so that the graphic and text content with practicality can be displayed to the user terminal 101. As an optional example, the practical text can also be stored in the database 102 and then recommended weighted and sent to the user terminal 101. In one embodiment, the database 102 can also use the text content uploaded by the user as a sample data set. When training the model, the sample data stored in the database 102 can be obtained to adjust the model parameters or structure.

[0045] It is understandable that the method provided in the embodiments of the present application can be executed by a computer device, including but not limited to a user terminal or a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0046] For further understanding, please see Figure 2 , Figure 2 This embodiment of the application is based on Figure 1 The network architecture diagram of the present invention provides a flowchart of a text processing method. The method can be performed by a user terminal (such as the above Figure 1 The user terminal 101 shown in the figure can also be executed by the processing end and the user terminal (as shown in the figure above). Figure 1 The text processing method may be executed by the processing end 100 in the corresponding embodiment, and is not limited here. For ease of understanding, this embodiment takes the method executed by the processing end 100 as an example for explanation. The text processing method may at least include the following steps S201-S202:

[0047] Step S201, obtaining the text to be processed, wherein the text to be processed includes a text title, text keywords and text body.

[0048] In a possible embodiment, the text to be processed includes multiple types of articles, and the types are not limited. The types here mean that the text to be processed can be graphic content in entertainment, society, music, humanities, science, history, sports, technology, education, etc. These graphic contents are uploaded by users and stored in the corresponding database. Through the background processing of the graphic content, they can be displayed to users in a personalized or unified manner, allowing different users to browse and view the target content according to their own interests. In these types of articles, the text length is usually higher than 512 characters. Such articles can be regarded as long texts and chapter-level articles. Correspondingly, the text to be processed is generally the graphic content corresponding to the long text. Each article has a title and a body. Therefore, the text to be processed will include a text title and a text body. For text keywords, it is the content extracted from the title or body, generally in the form of one or more words indicating the core content of the article, that is, the keyword can be a set, including multiple keywords. For example, an article about cooking tutorials is titled "Home Cooking Methods for Fish". The text details two methods, namely braised fish with sauce and steamed fish, with pictures. Therefore, the keywords of this article can include "home cooking methods", "braised fish with sauce", "steamed fish" and other words that indicate the key information of the article. Using the features of the three dimensions of text title, text keywords and text body to describe the information of the article more comprehensively can ensure the accuracy of the final recognition results.

[0049] Step S202: input the text to be processed into a long text recognition model for processing to obtain a target result, where the target result is used to indicate the practicality category of the text to be processed.

[0050] In a possible embodiment, the long text recognition model is obtained by pre-training the initial text recognition model using the first text data, and then fine-tuning the pre-trained text recognition model using the second text data. The initial text recognition model can be a language representation model using a sparse self-attention mechanism (combining a global self-attention mechanism and a local self-attention mechanism), such as a LongFomer model. The first text data includes an incomplete text body, and the second text data includes a sample text title, sample text keywords, sample text body, and corresponding reference practicality category labels.

[0051] In a possible embodiment, the target result indicating the practicality category of the text to be processed may be the prediction value of each category. That is, the prediction value of whether the text to be processed is practical or not practical. Since the practicality includes two categories, namely, knowledge practicality or practical practicality that helps the user's daily life, the target result may be subdivided into the prediction values ​​corresponding to the three categories of no practicality, knowledge practicality, and practical practicality. The specific form of this prediction value may be a probability of 0-1, which is not limited here. The understanding of knowledge-based practicality and practical-based practicality in the practicality category can be explained by the following examples: For example, an article about cars introduces the advantages and disadvantages of various types of cars and car selection skills. Such content can be used as a popular science knowledge for reference and understanding by users, and it is not necessary to obtain a unique and definite result based on these contents. In contrast, if it is an article about how to repair a car when it breaks down, the content involves how to troubleshoot the fault and the steps to solve the problem. According to the above content, a definite result can be obtained on whether the actual problem can be solved. It is knowledge that is biased towards practical operation and belongs to practical content that is helpful to users' daily lives. It has certain practical significance. Both of the above two types are practical articles. If any of the predicted values ​​in the target results is the largest, it can be regarded as the text to be processed is practical. On the contrary, if the probability of the above two practical categories being identified is very small relative to the probability of no practicality, it can be regarded as the text to be processed is not practical.

[0052] In a possible embodiment, the text to be processed is input into the long text recognition model for processing, and the specific steps of obtaining the target result may also include: first determining the title vector, keyword vector and text vector corresponding to the title, text keywords and text body respectively. Specifically, it is necessary to determine the title vector corresponding to the text title, the keyword vector corresponding to the text keyword and the text vector corresponding to the text body, so as to perform corresponding processing on these vectors and realize the required functions. Since the long text recognition model adopts the attention mechanism, the order of words or characters in the text sequence is not considered, which will cause the different sentences composed of the same words to express the same. In this case, the specific method of determining the title vector corresponding to the text title may include: after determining the word vector and position vector corresponding to the text title, the word vector and the position vector are fused to obtain the title vector corresponding to the text title. Specifically, the word vector and position vector corresponding to the text title can be obtained through training and learning. For example, the word vector is pre-trained by algorithms such as Word2Vec and Glove, or it can be obtained by other methods, for example, the position vector uses a fixed calculation formula. Correspondingly, the word vector indicates the semantic information of the title, which can be a vector representation containing context information, and the position vector indicates the positional relationship between the various words included in the title. The relative or absolute position of the word in the text sequence can be saved by using the position vector. Since the dimensions of the word vector and the position vector corresponding to the title are the same, the word vector and the position vector of the title can be added to obtain the corresponding title vector. The specific method for determining the keyword vector corresponding to the text keyword and the text vector corresponding to the text body is the same as the method for determining the title vector corresponding to the text title mentioned above. The corresponding word vector and the position vector are first obtained, and then the two are merged to obtain the corresponding vector, which will not be repeated here. It should be noted that the title vector, the keyword vector and the text vector represent the underlying features of the text to be processed, and these vectors need to be further processed to achieve the high-level expression of the text to be processed. In addition, the text body part can also be extracted by segmented processing to obtain the representation vector, but each representation vector needs to include the vectors corresponding to the title and the keyword respectively, or the vector of the keyword corresponding to the content of the part of the text, in addition to the text vector.

[0053] Then, the representation vector of the text to be processed is determined according to the title vector, keyword vector and text vector. Specifically, the dimensions of the corresponding title vector, keyword vector and text vector obtained by processing the text title, text keyword and text body are consistent, so these vectors can be fused into a matrix, so that for multiple articles included in a batch, the title vector, keyword vector and text vector can be processed in parallel to improve the centralized processing efficiency. For example, if there are 3 words in the title, 2 words in the keyword, and 512 words in the text, and each has a corresponding position vector, if the extracted vector dimensions are all 512, the corresponding title vector 3×512, keyword vector 2×512, and text vector 512×512 are obtained. The above-mentioned title vector, keyword vector and text vector are spliced ​​together to form a matrix of size 517×512. Since the practicality of the text to be processed needs to be identified, the token [CLS] can be used to identify the article. Specifically, the vector corresponding to [CLS] is fused with the title vector, keyword vector and text vector to form the representation vector of the text to be processed. It should be noted that, from the perspective of word granularity, each word in the title, keyword or text can be regarded as a token. Therefore, [CLS] can also be regarded as a word in the text to be processed and added to the front of the first word in the article. That is, the vectors corresponding to [CLS], title, keyword, and text are concatenated in sequence.

[0054] For some language scenarios, the representation vector of the above text to be processed also has good robustness in expressing the semantic information of the text content. This is because the same words may express very different semantics in different articles. For example, the word "threshold" may represent the structure of a house in some articles, or it may represent the difficulty of entering a certain field. If you simply extract features from keywords or comments, you cannot accurately express the content of the article, which will result in low accuracy in the final classification result. However, in addition to keywords, the representation vector also extracts representation vectors from multiple aspects for the article title and text, which can ensure that even in different contexts, the text semantics can be accurately expressed according to the current context, thereby achieving the accuracy of practical text recognition.

[0055] Finally, the representation vector is processed to obtain the target result. In one embodiment, the text to be processed is generally a text including graphic content. The construction of graphic atomic capabilities is mainly to explore some fine-grained high-quality atomic features, so as to refine and deeply understand the high-quality content, so as to recommend better use of the fine-grained atomic capability features on the content understanding side. In simple terms, atomic capabilities are some tiny capabilities, which are fine-grained descriptions of the attributes or characteristics of graphic content. In the aforementioned embodiment, the language representation model using sparse attention, that is, the long text recognition model, is used to process the text to be processed, so as to realize the construction of practical atomic capability features. Of course, other text recognition models can also construct practical atomic capabilities by processing representation vectors. Correspondingly, the processing of the representation vector in this step to obtain the target result can also be understood as the process of constructing practical atomic capability features.

[0056] In one embodiment, the specific steps of processing the characterization vector to obtain the target result may include: first selecting the current data to be processed from the characterization vector according to the data selection sliding window, and processing the current data to be processed to obtain the first intermediate data, and then selecting new data to be processed from the characterization vector according to the data selection sliding window and the processing step size corresponding to the data selection sliding window, and processing the new data to be processed to obtain the second intermediate data, and finally determining the target result based on the first intermediate data and the second intermediate data.

[0057] Specifically, the processing mechanism of the data selection sliding window is similar to that of the CNN sliding window, which can realize dimensionality reduction processing on the corresponding data, thereby reducing the corresponding amount of calculation. The amount of data corresponding to the size of the data selection sliding window can be M times the amount of data corresponding to the processing step size, where M is a positive integer, that is, the data sliding window size w = M × step, where step is the processing step size. For details on the data selection sliding window processing, please refer to Figure 3 . According to the data selection sliding window, a part of the vector is determined from the representation vector as the current data to be processed, and attention calculation is performed on the current data to be processed, so that attention indication information can be obtained as the first intermediate data, and then the data selection sliding window is moved according to the length of the processing step, and the next group of data is selected as the new data to be processed, and the second intermediate data is obtained by the same processing method as the aforementioned current data to be processed. Similarly, multiple groups of data to be processed can be obtained through the data selection sliding window, and they are all processed in the same way to obtain corresponding multiple groups of intermediate data. Finally, according to these intermediate data, the practicality category of the text to be processed corresponding to the representation vector can be determined.

[0058] Furthermore, in the process of processing the data to be processed and obtaining the intermediate data, the first mapping matrix can be used to process the data to be processed to obtain the global self-attention indication information corresponding to the data to be processed, and the second mapping matrix can be used to process the data to be processed to obtain the local self-attention indication information corresponding to the data to be processed, and then the global self-attention indication information and the local self-attention indication information are used as the intermediate data. Among them, for the target data object, the self-attention indication information between the target data object and each data object other than the target data object in the data to be processed can be determined by the first mapping matrix, and the self-attention indication information between the target data object and its adjacent data objects can be determined by the second mapping matrix, where the target data object is any data object in the data to be processed.

[0059] Specifically, the target data object is the vector data corresponding to a certain character or some characters, that is, in a text sequence, the representation vector is the vector representation of all the characters or words in the text. The data to be processed uses the data selection sliding window to select part of the vector from the representation vector, and then the target data object can be determined from this part of the vector. For example, the representation vector is [x1,x2,…,x 1024 ]Use the data selection sliding window to select the data to be processed as [x1,x2,…,x 512 ], the target data object determined by the data to be processed can be [x1,x2,…,x 512 ], any one or more of them, such as x1 or x2, x3, x4, x5, for the target data object, the self-attention indication information determined by the first mapping matrix is ​​global, for example, x1 and x2, ..., x5 except the target data object x1 512 The attention score between the two data objects, the self-attention indication information determined by the second mapping matrix is ​​local, such as the attention score between x1 and the adjacent data objects x2, x3, x4. By selecting the sliding window, the amount of attention calculation can be effectively reduced. For example, when the text sequence N = 2048, the data selection sliding window size w = 512, the speedup is about 3.25. In the LongFomer model, the amount of attention calculation is w 2 (N / w+1), while the attention calculation of the BERT model is N 2 . From this, we can also see the advantage of sliding window in reducing the amount of calculation. Optionally, the data to be processed selected by the data selection sliding window is all text sequences corresponding to the representation vector, such as [x1, x2, …, x 1024 ], for the target data object x1 and the remaining data objects x2,…,x 1024 The global self-attention indication information of mutual attention between them.

[0060] It should also be noted that the first mapping matrix is ​​Q for local self-attentions ,K s ,V s , the second mapping matrix is ​​Q for global self-attention g ,K g ,V g . In the bottom layer of the long text recognition model (for example, the encoder near the input end), a smaller data selection sliding window size is used to model local information. In the high layer (for example, the encoder near the output end), a larger data selection sliding window size is used to expand the receptive field. Here, the local self-attention based on CNN moves in the form of a window. The characters in each window can pay attention to each other. "Global" means that a small part of the characters can pay attention to all other characters, and all other characters can pay attention to it. If the number of these characters is small, then the whole can also be regarded as O(n) complexity. If the characters are replaced with words, the principle is similar. In this embodiment, this small part of the characters can be the [CLS] mark added to the front of each article input. It can be understood that one article corresponds to one [CLS] mark, which and the title, keywords and text constitute the feature vector corresponding to the text to be processed, such as [CLS, x1, x2, x3..., x n ], where CLS is the vector corresponding to the [CLS] flag, x1,...,x n Including title vector, keyword vector, text vector, and the second mapping matrix mainly acts on the [CLS] flag, and the first mapping matrix acts on all vectors included in the representation vector, such as [CLS,x1,x2,x3...,x n ], for a schematic diagram of the effect of sparse self-attention, see Figure 4 , we can see that there are a large number of "white squares", indicating that they do not need to be paid attention to. As the length of the text increases, the number of such white squares will increase quadratically, but in fact, the number of "grey squares" required is very small. The self-attention indication information can be a correlation coefficient matrix composed of the correlation between any two vectors, that is, the self-attention matrix. The global self-attention indication information is the correlation coefficient between the vector corresponding to the [CLS] flag and other vectors of the text to be processed. The local self-attention indication information is the correlation coefficient between the vector in the representation vector of the text to be processed and the vector at a fixed position. The sparse self-attention indication information can be formed by the global self-attention indication information and the local self-attention indication information. Among them, only the [CLS] flag element is related to all elements in the sequence (that is, all words in the text), and each of the remaining elements is only related to a part of the elements in the sequence. In this way, for long texts, there is no need to pay attention to the correlation between two elements, but to selectively add global self-attention and pay more attention to truly meaningful relationships. In this way, the computational cost and computing resources required for task processing can be reduced to a certain extent, and the processing efficiency can be greatly improved.

[0061] The above self-attention indication information is further processed, such as normalization and residual connection, and then the sparse self-attention indication information after residual connection and normalization is input into the feedforward neural network for processing to obtain the output result, and then the output result is passed through the classifier to obtain the target result. The output result here is a matrix, which has the same dimension as the representation vector of the text to be processed, but a higher-level feature expression of the text to be processed. The output result is input into the classifier for classification, and a numerical value indicating the practicality of the text to be processed can be obtained, that is, the target result mentioned above. For example, after the softmax classifier, a 3-dimensional vector [y1, y2, y3] is output, in which the numerical values ​​included respectively represent the probability that the text to be processed belongs to the knowledge type of practicality, the practical type of practicality, and the probabilities of not having practicality. According to the probability, it can be determined whether the text to be processed is practical and which category of practicality it belongs to.

[0062] In summary, the embodiments of the present application include at least the following advantages:

[0063] The information of the text to be processed is expressed more comprehensively through the three dimensions of the title, keywords and text of the text to be processed. The long text recognition model obtained after pre-training and fine-tuning is used to process the above text to be processed, which can improve the accurate expression of the semantic information of the article, reduce the misjudgment rate in some special language scenarios, and thus improve the accuracy of practical recognition of the text to be processed. At the same time, the data selection sliding window is used to process the representation vector corresponding to the text to reduce the dimension of vector processing, thereby effectively reducing the amount of calculation and improving the processing efficiency. For long texts at the chapter level, the correlation coefficient between some words (or characters) and all other words (or characters) is obtained through global self-attention, and the correlation coefficient between each word (or character) and some words (or characters) is obtained through local self-attention, which also effectively saves the resources of attention calculation, reduces the corresponding calculation cost, and improves the accuracy of practical category recognition of long texts within limited resources.

[0064] See also Figure 5 , Figure 5 This embodiment of the application is based on Figure 1 The network architecture diagram of FIG. 1 provides a flowchart of another text processing method, the method at least comprising steps S501-502:

[0065] S501, pre-training an initial text recognition model using first text data to obtain a pre-trained text recognition model, wherein the pre-trained text recognition model has semantic and grammatical recognition capabilities.

[0066] In a possible embodiment, for the understanding of long text at the chapter level, its quality characteristics tend to be semantic understanding, which is difficult to learn. Pre-training the model through an open source corpus data set can reduce the learning difficulty of semantic understanding of long text. The first text data is used as sample data in the pre-training stage, including a large amount of text data, specifically, a large number of articles such as news and other unlimited long texts, the number of which can be about 1.5 million to 1 billion, which is not limited here. Some words in the article can be removed to form incomplete text body data to feed the initial text recognition model. The pre-trained model can predict which words are missing in the article, predict contextual relationships, or be used for other prediction tasks. By pre-training the initial text recognition model with a large amount of corpus data, the text recognition model has stronger semantic grammar recognition or understanding capabilities, can better understand the chapter, and make the feature vector have better semantic expression capabilities.

[0067] For example, the pre-trained text recognition model can quickly extract the semantic representation of words in the context of the input text. This is because pre-training uses self-supervised learning to obtain a model that is unrelated to a specific task from large-scale data. It is an application of transfer learning. After learning human language knowledge, the model can be substituted into specific tasks, that is, the knowledge learned in the open field can be transferred to downstream tasks, such as text classification, to improve low-resource tasks. It is a very friendly method for low-resource language processing tasks.

[0068] In one embodiment, the initial text recognition model can be an improvement based on the BERT model. The Transformer Encoder in the BERT model is improved to obtain a new model, namely the LongFomer model. The specific improvement is to replace the global attention full attention structure in the Transformer Encoder with the sparse attention LongFomerattention. During pre-training, the existing weights of the BERT model are reused, and its 512 position embedding vectors are copied to initialize the position embedding vector of the LongFomer model. Optionally, the data set is divided into a training set and a validation set. The LongFomer model can be trained using a training set with a data size of 230,000 and a validation set with a data size of 10,000 to obtain a pre-trained LongFomer model. The pre-trained LongFomer model can be used to learn a more accurate chapter-level document representation to improve the quality recognition effect of the article text.

[0069] S502, fine-tuning the pre-trained text recognition model using the second text data, fine-tuning the model parameters of the pre-trained text recognition model to obtain a trained long text recognition model, wherein the trained long text recognition model is used to identify the practicality category of the long text.

[0070] In a possible embodiment, unlike the incomplete text body included in the first text data, the second text data includes sample text titles, sample text keywords, sample text bodies, and corresponding reference practicality category labels, and is a labeled sample data set for specific downstream tasks. In order to apply the pre-trained text recognition model to specific downstream tasks to obtain better task results, it is necessary to re-train it using the labeled sample data included in the second text data to slightly adjust the model parameters of the pre-trained text recognition model to obtain a trained text recognition model. This process is also called fine-tuning, which can make the fine-tuned text recognition model suitable for the downstream tasks to be performed, that is, the recognition of text practicality categories. For example, the pre-trained text recognition model can also be a LongFormer model. By fine-tuning the model parameters of the LongFormer model, the LongFormer model can accurately identify the practicality category of the text.

[0071] In summary, the embodiments of the present application include at least the following advantages:

[0072] The initial text recognition model is pre-trained using the incomplete text body included in the first text data, which enhances the text recognition model's ability to understand the semantics of paragraph-level text. The pre-trained text recognition model is fine-tuned using the text, title, keywords and practicality category labels included in the second text data to obtain a model suitable for specific tasks. The combination of these two training stages can effectively save the training time of the model and obtain a long text recognition model with excellent performance under limited data resources for downstream tasks. During training, the pre-trained prior knowledge is fully utilized to adjust the model, saving the cost and resources of model training and calculation. The long text recognition model obtained through pre-training and fine-tuning can ensure the effectiveness of long text practicality recognition, improve the performance of long text practicality processing, and improve the effect of quality judgment.

[0073] See also Figure 6 , is based on the embodiments of this application Figure 1 The network architecture diagram of text processing shown in FIG. 1 provides a flowchart of another text processing method. The method at least includes steps S601-S605. It should be understood that although Figure 6The steps in the flowchart are shown in sequence according to the arrows, but these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be performed in other orders.

[0074] S601, obtaining a text to be processed, wherein the text to be processed includes a text title, text keywords and text body.

[0075] S602: Input the text to be processed into a long text recognition model for processing to obtain a target result, where the target result is used to indicate the practicality category of the text to be processed.

[0076] As an optional example, the self-attention mechanism in the deep self-attention network Transformer enables the network to capture contextual information from the entire sequence, which is widely used in text classification. However, although the self-attention mechanism is very effective, in practice, too long text will cause memory explosion. The reason is that the time and space complexity of self-attention is quadratic. See Figure 7 , is the self-attention mechanism in Transformer. Each character can pay attention to all other characters. Therefore, when the text is longer, the space consumption increases significantly. Therefore, in actual use, the sentence length is usually truncated to 512 and then sent to the model in segments. Although this method can solve the problem of insufficient memory to a certain extent, it avoids the inherent defects of self-attention and is only a temporary solution. At the same time, this method is very troublesome and requires preprocessing of the data set. At the same time, this division may cause important information to be lost. Many tasks contain a large amount of long text. In practice, it is required to be able to process the long text data included in these tasks at one time. The Transformer applied in the long text scenario, referred to as LongFormer, adopts a combination of local self-attention and global self-attention to enable the model to accommodate texts with a maximum length of tens of thousands. It also makes full use of the prior knowledge obtained by pre-training, and efficiently adjusts the optimal model during fine-tuning, making the model more robust and achieving better recognition results. , In the process of using the LongFormer model to process the text to be processed, the main thing is the effective calculation of the attention score by the sparse self-attention it includes. Optionally, the long text recognition model may also be other language representation models to which sparse self-attention is applied.

[0077] In a possible embodiment, similar to the Transformer model, LongFormer is composed of multiple identical encoders, which can be 3 encoders or 6 encoders, to process articles of different length levels to achieve the best processing effect. The internal structure of the encoder can be found in Figure 8 , the multiple encoders in the LongFormer model do not share parameters, and position encoding is performed before the vector representation is sent to the encoder, that is, the vector sent to the encoder is the representation vector of the aforementioned text to be processed (corresponding to Figure 8 =The input embedding vector + the position vector obtained by position encoding), each encoder includes a self-attention neural network layer and a fully connected layer, and between these two layers there are also residual connections and layer normalization. Residual connections can solve the problem of multi-layer network training, allowing the network to only focus on the current difference, prevent network degradation, and improve the training effect. Layer normalization can speed up the convergence speed, and the self-attention neural network layer is a multi-head attention mechanism. The query vector Q, key value vector K, and value vector V are projected through multiple different linear transformations, and then the results are concatenated to obtain an output matrix, which can capture the correlation coefficients between words in multiple dimensions. The fully connected layer is also a feedforward neural network, including two layers. The activation function of the first layer is ReLU, and the second layer is a linear activation function. After the representation vector is processed by multiple encoders, the encoded representation vector can be obtained, which can be regarded as the feature vector of the text to be processed. For the self-attention neural network layer, the difference between the attention mechanism used by LongFormer and Transformer is that LongFormer uses a sparse self-attention mechanism, including local attention used to establish local context representation and global attention used to establish a complete sequence representation for prediction. The output calculation of the self-attention mechanism uniformly follows the following formula (1):

[0078]

[0079] Among them, d k is the vector dimension, QK T Represents the attention strength (or attention score) between words, divided by d k The square root of is to prevent the inner product of each row vector obtained by calculating the matrices Q and K from being too large.

[0080] Compared with many previous works, Longformer is the first to apply this sparse self-attention to three tasks involving long texts: question answering, coreference resolution, and document classification, and proves its effectiveness. Based on the above research results, the Longformer model has also achieved good results in text practicality category determination. For details, see Table 1 below. Therefore, the long text recognition model in the embodiment of this application can be the LongFormer model, and the structure diagram of the picture and text practicality recognition model can be seen in Fig. 9 , the embedding vectors corresponding to the article title, article keywords and article body are processed and input into the long text recognition model (LongFormer model). The embedding vector here corresponds to the word vector mentioned in the above embodiment. Each word vector must be fused with the position vector. This is because LongFormer uses the global information of the article and does not use the order information of the words. However, this part of information is very important for natural language processing. The position vector can be used to save the relative or absolute position of the word in the sequence. In addition, the title embedding vector, keyword embedding vector and text embedding vector can be spliced ​​into a large matrix by row, and then input into the LongFormer model. After being processed by the network included in multiple encoders, an output matrix with the same dimension as the input matrix is ​​obtained, and then it is input into the classifier to obtain the classification result, that is, the above-mentioned target result. Optionally, the long text recognition model can also be other models that can process long texts. The experimental results obtained by using different models to identify the practicality of text are shown in Table 1 below:

[0081] Table 1 Experimental results of text practicality recognition under different models

[0082] Model Overall classification accuracy Recall Accuracy Classification ability TextCNN 0.8906 0.8706 0.8886 0.8912 LSTM Long Short-Term Memory Network 0.8924 0.8641 0.9034 0.8856 HAN 0.9217 0.9328 0.9372 0.9256 BERT 0.9300 0.9412 0.9443 0.9475 Hierarchical Transformer Encoding Network 0.9302 0.9434 0.9456 0.9482 LongFormer long text classification model 0.9402 0.9465 0.9507 0.9515

[0083] It can be found that the results obtained by using the attention mechanism network are superior to those obtained by the conventional RNN model in terms of various evaluation indicators. In a longitudinal comparison of the same evaluation indicators of different models, the corresponding values ​​of the LongFormer model's evaluation indicators are at least one percentage point higher than any other model. Therefore, it can be seen that the LongFormer model has excellent ability in processing long texts at the chapter level. By using this model, more accurate chapter-level document representations can be learned, and the effect of identifying the quality of article texts can be effectively improved.

[0084] The model for identifying the atomic capabilities of image and text practicality (i.e., the long text recognition model) was used in the task of judging the quality of the image and text content in the content center, and it performed well. The specific data showed that the accuracy rate of the first phase of the model evaluation (food, health, life, automobiles, technology, emotions, etc.) reached 90%, and the recall rate reached 95%. The accuracy of identifying practical images and texts was improved by 1.5% compared with the BERT model.

[0085] For other details of the above steps S601-S602, please refer to Figure 2 The corresponding steps S201-S202 are not repeated here.

[0086] S603: Determine a recommendation weight of the text to be processed according to the practicality category of the text to be processed indicated by the target result.

[0087] In a possible embodiment, the target result corresponds to the practicality category of the text to be processed, such as knowledge-based practical articles or practical operation-based practical articles. For articles with this type of practicality, a high or low recommendation weight can be obtained. Specifically, there is a large difference in recommendation weights for practical articles and non-practical articles. According to different recommendation weights, the possibility of the corresponding article content being recommended and the frequency or position of the recommendation will be different. In this way, practical graphic content and other graphic content can be better distinguished, and the recommendation of practical graphic content can be achieved in a targeted manner.

[0088] S604: Associating the practicality category and the recommendation weight with the text to be processed.

[0089] In a possible embodiment, this step can be performed in a server that generates the target result, or the target result can be sent to other servers to perform this step. The target result can be a set of data such as [0.6, 0.3, 0.1], which represents the probability of the text to be processed belonging to knowledge-based practicality, practical-based practicality, and non-practicality from left to right. Therefore, the practicality category of the text to be processed can be determined by the corresponding numerical value: among the three predicted values, 0.6 is the largest value, indicating that the text to be processed is very likely to belong to the article of knowledge-based practicality, so it is determined as a text of knowledge-based practicality. Among them, the recommendation weight corresponding to the knowledge-based practicality will be associated with the text to be processed, indicating the possibility of recommendation of the text to be processed. As an optional example, when the text to be processed has both knowledge content about popular science and content about practical operation, it may include both knowledge-based and practical-based practicality at the time of final identification. In addition, for the text to be processed that is not practical, there will also be a corresponding practicality category and recommendation weight associated with it, so as to obtain the atomic ability characteristics of the text and recommend the practical text.

[0090] S605: Recommend processing of the text to be processed according to the recommendation weight associated with the text to be processed.

[0091] In a possible embodiment, the practicality category includes three types, namely, knowledge practicality, practical operation practicality, and non-practicality. The recommendation process may be to perform a recommendation weighting operation on all categories of texts. For example, a higher recommendation weight is configured for practical texts, while the recommendation weight of non-practical texts is not changed. Of course, the recommendation weight of non-practical texts may also be reduced. The specific method is not limited here. Specifically, the server corresponding to the browser side (e.g. Figure 1 The processing end 100) performs a recommendation weighting operation on the identified graphic and text practical content, and then sends it to the user terminal, so as to prioritize the recommendation of knowledge-based practical and practical content that helps the user's daily life to the user. Optionally, the specific application scenario of the above content can be: in the content processing link of the content center, all graphic and text content is judged for graphic and text practicality, and then it is released from the library and distributed to the terminal side, such as recommending weighting for the identified practical content.

[0092] In one embodiment, the main processing method of recommendation weighting can be to indicate whether the content is practical based on the practicality indicator, assign a higher recommendation weight to practical content, and reduce the recommendation weight of impractical content, such as raising the recommendation priority of practical articles to a certain level, and reducing the recommendation priority of impractical articles, thereby achieving priority recommendation of practical content. When recommending to the user terminal device, the frequency of recommendation to the user can be set according to the recommendation weight, or the recommended display position of the practical content can be set preferentially, so that the recommended practical content is displayed in an area that the user can easily pay attention to. As an extensible example, the recommendation weight can also be configured in combination with user-related data.

[0093] In an optional embodiment, if the target result indicates that the practicality category of the text to be processed is practical, the text to be processed is recommended based on the first recommendation weight; if the target result indicates that the practicality category of the text to be processed is knowledge practical, the text to be processed is recommended based on the second recommendation weight, wherein the first recommendation weight is greater than the second recommendation weight. Specifically, practical content includes knowledge practical articles and practical practical articles. Different recommendation weights can be adopted for these two types of practical articles. For most users, practical practical articles can bring users practical operation experience and more direct sense of practicality due to their teaching-style tutorials, and may be more popular than knowledge practical articles. Therefore, the recommendation weight of such practical articles can be configured to the highest level, and practical articles with a wider user audience may be recommended to relevant pages, while the recommendation weight of knowledge practical articles is configured to a slightly lower level, thereby giving priority to displaying practical practical articles to users. However, for some application scenarios or web pages with specific functions, practical articles and knowledge-based practical articles can be distinguished. For example, in cooking teaching applications, there will be more practical content, so its recommendation weight will be relatively larger. However, in some technical forums or popular science applications or web pages, knowledge-based practical articles will be more popular than practical articles. Therefore, the recommendation weight of knowledge-based practical content can be configured higher.

[0094] Experiments have proved that pushing practical content to users through weighted recommendations can bring users a good reading experience and achieve good business results on the business side. The specific data are: on the browser side, the overall page views (Page View, PV) increased by 0.45%, the overall clicks increased by 0.17%, and the picture and text clicks increased by 0.5%; at the same time, the number of daily active users (Daily Active User, DAU) retained on the next day increased by 0.064%, the unique visitor (UV) in the interactive indicator data increased by 0.337%, the like PV increased by 3.090%, and the comment UV increased by 0.325%.

[0095] As an optional example, the recommendation process may also be performed on other servers (eg Figure 1 After the processing end 100 in the processing is completed, the result is sent to the corresponding server on the browser side, so that the server on the browser side sends it to the browser page, or directly sends it to the user terminal to display the recommended practical graphic content to the user.

[0096] The above-mentioned weighted recommendation experiment on the recognized graphic and text practical content on the browser side has achieved the priority recommendation of knowledge-based practical and practical content that helps users in their daily lives to users, bringing users a good reading experience while also achieving good business results on the business side.

[0097] In summary, the embodiments of the present application include at least the following advantages:

[0098] Through the long text recognition model LongFormer, sparse attention combining global self-attention and local self-attention is used to expand the capacity of text length to the level of 10,000, and to process the text to be processed that exceeds the text length threshold, which greatly reduces the processing difficulty of long text recognition and improves the efficiency and accuracy of long text practical recognition. The LongFormer model introduces the position information between words, and on the basis of maintaining the basic information of the text sequence, the feature expression information of the processed text is more accurate. In addition, based on the recommendation method on the graphic content side, different recommendation weights are configured for different practical categories. In specific application scenarios, the importance of the recommendation of practical category articles is divided into levels according to the recommendation weight, and practical content is pushed to users in a targeted manner, which can effectively improve the accuracy and business effect of the recommendation, give full play to the function of the long text recognition model in text practical recognition, and improve user satisfaction and user stickiness in the corresponding products.

[0099] See also Fig.10 , is a structural diagram of a text processing device provided in an embodiment of the present application. The text processing device may be a computer program (including program code) running in a computer device, for example, the text processing device is an application software; the device may be used to execute the corresponding steps in the method provided in an embodiment of the present application. The text processing device 1000 includes: an acquisition module 1001, a processing module 1002, wherein:

[0100] The acquisition module 1001 is used to acquire the text to be processed, wherein the text to be processed includes a text title, text keywords and text body;

[0101] Processing module 1002 is used to input the text to be processed into a long text recognition model for processing to obtain a target result, and the target result is used to indicate the practicality category of the text to be processed; wherein the long text recognition model is obtained by pre-training an initial text recognition model using first text data and then fine-tuning the pre-trained text recognition model using second text data; the first text data includes an incomplete text body, and the second text data includes a sample text title, sample text keywords, sample text body and corresponding reference practicality category labels.

[0102] In one embodiment, the apparatus 1000 further includes a training module 1003, wherein:

[0103] A training module 1003 is used to pre-train the initial text recognition model using the first text data to obtain a pre-trained text recognition model, wherein the pre-trained text recognition model has semantic and grammatical recognition capabilities;

[0104] The training module 1003 is also used to use the second text data to fine-tune the pre-trained text recognition model, fine-tune the model parameters of the pre-trained text recognition model, and obtain a trained long text recognition model, and the trained long text recognition model is used to identify the practicality category of the long text.

[0105] In one embodiment, the processing module 1002 is specifically used to: determine the title vector, keyword vector and text vector corresponding to the text title, the text keyword and the text body respectively; determine the representation vector of the text to be processed based on the title vector, the keyword vector and the text vector; process the representation vector to obtain the target result.

[0106] In one embodiment, the processing module 1002 is specifically used to: select current data to be processed from the characterization vector according to the data selection sliding window, and process the current data to be processed to obtain first intermediate data; select new data to be processed from the characterization vector according to the data selection sliding window and the processing step corresponding to the data selection sliding window, and process the new data to be processed to obtain second intermediate data; determine the target result according to the first intermediate data and the second intermediate data.

[0107] In one embodiment, the processing module 1002 is further specifically used to: use a first mapping matrix to process the data to be processed to obtain global self-attention indication information corresponding to the data to be processed; use a second mapping matrix to process the data to be processed to obtain local self-attention indication information corresponding to the data to be processed; use the global self-attention indication information and the local self-attention indication information as intermediate data; wherein, for a target data object, the self-attention indication information between the target object and each data object in the data to be processed except the target object is determined by the first mapping matrix; the self-attention indication information between the target object and its adjacent objects is determined by the second mapping matrix; the target data object is any data object in the data to be processed.

[0108] In one embodiment, the apparatus 1000 further includes a determination module 1004 and an association module 1005, wherein:

[0109] A determination module 1004, configured to determine a recommendation weight of the text to be processed according to the practicality category of the text to be processed indicated by the target result;

[0110] An associating module 1005 is used to associate the practicality category and the recommendation weight with the text to be processed;

[0111] The recommendation module 1006 is used to recommend the text to be processed according to the recommendation weight associated with the text to be processed.

[0112] In one embodiment, the determination module 1004 is specifically used to: if the practicality category of the text to be processed is practical practicality, recommend the text to be processed based on a first recommendation weight; if the practicality category of the text to be processed is knowledge practicality, recommend the recommendation weight of the text to be processed based on a second recommendation weight, and the first recommendation weight is greater than the second recommendation weight.

[0113] It can be understood that the functions of each functional module of the text processing device described in the embodiment of the present application can be specifically implemented according to the method in the above-mentioned method embodiment. The specific implementation process can refer to the relevant description and beneficial effects of the above-mentioned method embodiment, which will not be repeated here.

[0114] See also Fig.11 , is a schematic diagram of the structure of a computer device provided in an embodiment of the present application, the computer device 110 may include a processor 1101, a memory 1102, a network interface 1103 and at least one communication bus 1104. Among them, the processor 1101 is used to schedule computer programs, and may include a central processing unit, a controller, and a microprocessor; the memory 1102 is used to store computer programs, and may include a high-speed random access memory, a non-volatile memory, such as a disk storage device, a flash memory device; the network interface 1103 provides a data communication function, and the communication bus 1104 is responsible for connecting various communication elements.

[0115] The processor 1101 may be used to call a computer program in the memory to perform the following operations:

[0116] Acquire a text to be processed, wherein the text to be processed includes a text title, text keywords and text body;

[0117] Inputting the text to be processed into a long text recognition model for processing to obtain a target result, wherein the target result is used to indicate the practicality category of the text to be processed;

[0118] Among them, the long text recognition model is obtained by pre-training the initial text recognition model with the first text data and then fine-tuning the pre-trained text recognition model with the second text data; the first text data includes an incomplete text body, and the second text data includes a sample text title, sample text keywords, sample text body and corresponding reference practicality category labels.

[0119] In one embodiment, the processor 1101 is also used to: use the first text data to pre-train the initial text recognition model to obtain a pre-trained text recognition model, and the pre-trained text recognition model has semantic and grammatical recognition capabilities; use the second text data to fine-tune the pre-trained text recognition model, fine-tune the model parameters of the pre-trained text recognition model, and obtain a trained long text recognition model, and the trained long text recognition model is used to identify the practicality category of the long text.

[0120] In one embodiment, the processor 1101 is specifically used to: determine the title vector, keyword vector and text vector corresponding to the text title, the text keyword and the text body respectively; determine the representation vector of the text to be processed based on the title vector, the keyword vector and the text vector; process the representation vector to obtain the target result.

[0121] In one embodiment, the processor 1101 is specifically used to: select current data to be processed from the characterization vector according to the data selection sliding window, and process the current data to be processed to obtain first intermediate data; select new data to be processed from the characterization vector according to the data selection sliding window and the processing step corresponding to the data selection sliding window, and process the new data to be processed to obtain second intermediate data; determine the target result according to the first intermediate data and the second intermediate data.

[0122] In one embodiment, the processor 1101 is specifically used to: use a first mapping matrix to process the data to be processed to obtain global self-attention indication information corresponding to the data to be processed; use a second mapping matrix to process the data to be processed to obtain local self-attention indication information corresponding to the data to be processed; use the global self-attention indication information and the local self-attention indication information as intermediate data; wherein, for a target data object, the self-attention indication information between the target object and each data object in the data to be processed except the target object is determined by the first mapping matrix; the self-attention indication information between the target object and its adjacent objects is determined by the second mapping matrix; the target data object is any data object in the data to be processed.

[0123] In one embodiment, the processor 1101 is specifically used to: determine the recommendation weight of the text to be processed according to the practicality category of the text to be processed indicated by the target result; associate the practicality category and the recommendation weight with the text to be processed; and recommend the text to be processed according to the recommendation weight associated with the text to be processed.

[0124] In one embodiment, the processor 1101 is specifically used to: if the practicality category of the text to be processed is practical practicality, recommend the text to be processed based on a first recommendation weight; if the practicality category of the text to be processed is knowledge practicality, recommend the recommendation weight of the text to be processed based on a second recommendation weight, and the first recommendation weight is greater than the second recommendation weight.

[0125] In a specific implementation, the processor 1101, memory 1102 and network interface 1103 described in the embodiment of the present application can execute the implementation method of the computer device described in a text processing method provided in an embodiment of the present application, and can also execute the implementation method and beneficial effects described in a text processing device provided in an embodiment of the present application, which will not be repeated here.

[0126] The embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored, and when the instructions are executed on a computer, the computer executes the text processing method as described in the embodiment of the present application. The specific implementation method can be referred to the above description, and will not be repeated here.

[0127] The embodiment of the present application also provides a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the text processing method as described in the embodiment of the present application. The specific implementation method can be referred to the above description, which will not be repeated here.

[0128] It should be noted that, for the above-mentioned various method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0129] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0130] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A text processing method, characterized in that: include: Acquire a text to be processed, wherein the text to be processed includes a text title, text keywords and text body; The text to be processed is input into a long text recognition model for processing to obtain a target result, and the target result is used to indicate the practicality category of the text to be processed; wherein the long text recognition model is obtained by pre-training an initial text recognition model using first text data and then fine-tuning the pre-trained text recognition model using second text data; the first text data includes an incomplete text body, and the second text data includes a sample text title, sample text keywords, sample text body, and corresponding reference practicality category labels; Determining a recommendation weight of the text to be processed according to the practicality category of the text to be processed indicated by the target result; Associating the practicality category and the recommendation weight with the text to be processed; The text to be processed is recommended according to the recommendation weight associated with the text to be processed.

2. The method according to claim 1, characterized in that The method further comprises: Pre-training the initial text recognition model using the first text data to obtain a pre-trained text recognition model, wherein the pre-trained text recognition model has semantic and grammatical recognition capabilities; The pre-trained text recognition model is fine-tuned using the second text data, and the model parameters of the pre-trained text recognition model are fine-tuned to obtain a trained long text recognition model, wherein the trained long text recognition model is used to identify the practicality category of the long text.

3. The method according to claim 1, characterized in that The inputting the to-be-processed text into the long text recognition model for processing to obtain the target result comprises: Determine a title vector, a keyword vector, and a text vector corresponding to the text title, the text keyword, and the text body, respectively; Determine a representation vector of the text to be processed according to the title vector, the keyword vector and the text vector; The characterization vector is processed to obtain a target result.

4. The method according to claim 3, characterized in that The processing of the characterization vector to obtain a target result includes: Selecting current data to be processed from the characterization vector according to the data selection sliding window, and processing the current data to be processed to obtain first intermediate data; Selecting new data to be processed from the characterization vector according to the data selection sliding window and the processing step size corresponding to the data selection sliding window, and processing the new data to be processed to obtain second intermediate data; A target result is determined according to the first intermediate data and the second intermediate data.

5. The method according to claim 4, characterized in that The amount of data corresponding to the data selection sliding window is M times the amount of data corresponding to the processing step, where M is a positive integer.

6. The method according to claim 4, characterized in that The initial text recognition model is a language representation model that uses a global self-attention mechanism and a local self-attention mechanism.

7. The method according to claim 6, characterized in that Process the data to be processed to obtain intermediate data, including: The first mapping matrix is ​​used to process the data to be processed to obtain global self-attention indication information corresponding to the data to be processed; The data to be processed is processed using the second mapping matrix to obtain local self-attention indication information corresponding to the data to be processed; Using the global self-attention indication information and the local self-attention indication information as intermediate data; Among them, for the target data object, the self-attention indication information between the target data object and each data object in the data to be processed except the target data object is determined by the first mapping matrix; the self-attention indication information between the target data object and its adjacent data objects is determined by the second mapping matrix; the target data object is any data object in the data to be processed.

8. The method according to claim 1, characterized in that The practicality category includes knowledge practicality or practical operation practicality, and the recommending processing of the text to be processed according to the recommendation weight associated with the text to be processed includes: If the practicability category of the text to be processed is practical practicability, recommending the text to be processed based on a first recommendation weight; If the practicality category of the text to be processed is knowledge-based practicality, a recommendation weight of the text to be processed is recommended based on a second recommendation weight, and the first recommendation weight is greater than the second recommendation weight.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the text processing method according to any one of claims 1 to 8 is executed.

10. A computer program product, characterized in that The computer program product includes program instructions, which are stored in a computer-readable storage medium. A processor reads the program instructions from the computer-readable storage medium, and the processor executes the program instructions to execute the text processing method according to any one of claims 1 to 8.

Images