Text processing method, electronic device, computer storage medium and program product

By replacing the Attention mechanism in the Transformer model with local, global, and segmented pooling processing, the problem of excessive computational complexity in long text processing is solved, and efficient text feature extraction and contextual relationship modeling is achieved.

CN113887169BActive Publication Date: 2024-12-03ALIBABA DAMO (HANGZHOU) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111146756.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-28
Publication Date
2024-12-03
Estimated Expiration
2041-09-28

AI Technical Summary

Technical Problem

When the Transformer model modeling text context relationships, the computational complexity of the Multi-Head Attention mechanism is squared with the text length, resulting in excessive computational and memory consumption when processing long text.

Method used

Local pooling, global pooling and segmented pooling are adopted to replace the traditional Attention mechanism, and text features are extracted through these pooling layers, thereby reducing the computational complexity to the linear level.

Benefits of technology

It effectively reduces the computational complexity and memory consumption, realizes contextual relationship modeling without relying on the Attention mechanism, and improves the efficiency of long text processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113887169B_ABST
    Figure CN113887169B_ABST
Patent Text Reader

Abstract

The embodiment of the present application provides a text processing method, an electronic device, a computer storage medium and a computer program product, wherein the text processing method includes: performing vectorization processing on the text to be processed to obtain the character vector and position vector corresponding to the text to be processed; performing local pooling processing based on adjacent characters, global pooling processing based on the entire text to be processed and segmented pooling processing based on segments based on the character vector and the position vector, respectively, to obtain the text features corresponding to the text to be processed; obtaining the corresponding encoding vector based on the text features; performing target output processing on the encoding vector to obtain the text processing result of the text to be processed. Through the embodiment of the present application, the computational complexity of text modeling contextual relationships is greatly reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a text processing method, an electronic device, a computer storage medium, and a computer program product. Background Art

[0002] Text is a type of sequence data of characters and symbols. In many work and life scenarios, it is necessary to perform text modeling operations such as semantic analysis, keyword analysis and extraction to provide basic analysis support for the further application of the text.

[0003] With the development of deep learning technology, more and more neural network models are applied to text modeling to improve the efficiency and accuracy of text modeling. Transformer model is one of the most widely used and outstanding models. For example, the structure of a Transformer model is as follows: Figure 1 As shown, it is divided into an encoder part and a decoder part, corresponding to Figure 1 The left and right halves of .

[0004] Generally, the encoder uses Nx (for example, 6) identical blocks, in which the PositionEmbedding is used to represent the position information of words or characters, thereby describing the contextual relationship of words or characters. The Attention mechanism uses the attention function to extract text features based on the input text vectors Input Embedding and Position Embedding. The Attention function can be regarded as a process of mapping a query and a series of key-value pairs into an output output. It obtains the three vectors K, Q, and V by performing three different linear transformations on the input, and then obtains the attention-weighted features through scaled dot-production processing, and then filters useless information through mask. The Multi-Head Attention mechanism executes the above process H times, then splices the outputs together, and outputs a text vector carrying contextual information. For the Add&Norm part, Add represents the residual connection to prevent the gradient from disappearing or exploding due to the excessive depth of the model; Norm represents normalization. The Feedforward part undergoes two linear transformations and then passes through the relu activation function, after which the Add&Norm part is performed again to output the final encoding vector.

[0005] Decoder is similar to Embedding and also has Nx layers, but it has an additional Masked Multi-Head Attention at the beginning to ensure that the prediction result of the pos position can only depend on the prediction result before pos. In the model training phase, Encoder and Decoder complete the model training together; in the model inference phase, Encoder is used to encode the data.

[0006] The Transformer model can effectively extract text features. However, the computational complexity of the Multi-Head Attention mechanism used by the Transformer model to model contextual relationships is proportional to the square of the text length (O(n 2 )), which results in excessive computation and memory consumption when modeling text, especially long texts of more than a thousand words. Summary of the invention

[0007] In view of this, an embodiment of the present application provides a text processing solution to at least partially solve the above-mentioned problem.

[0008] According to the first aspect of the embodiment of the present application, a text processing method is provided, including: performing vectorization processing on the text to be processed to obtain a character vector and a position vector corresponding to the text to be processed; performing local pooling processing based on adjacent characters, global pooling processing based on the entire text to be processed, and segment pooling processing based on segments based on the character vector and the position vector to obtain text features corresponding to the text to be processed; obtaining a corresponding encoding vector based on the text features; performing target output processing on the encoding vector to obtain a text processing result of the text to be processed.

[0009] According to the second aspect of an embodiment of the present application, another text processing method is provided, including: receiving a voice question input through a smart device, and converting the voice question into a corresponding question text; performing vectorization processing on the question text to obtain a character vector and a position vector corresponding to the question text; performing local pooling processing based on adjacent characters, global pooling processing based on the entire question text, and segmented pooling processing based on segments in the question text based on the character vector and the position vector, respectively, to obtain text features corresponding to the question text; obtaining a corresponding encoding vector based on the text features; performing question classification and output processing on the encoding vector to obtain a question classification corresponding to the question text; obtaining an answer text corresponding to the question text according to the question classification, converting the answer text into a voice answer and outputting it through the smart device.

[0010] According to a third aspect of an embodiment of the present application, a text processing device is provided, comprising: a first acquisition module, used to perform vectorization processing on a text to be processed, and obtain a character vector and a position vector corresponding to the text to be processed; a second acquisition module, used to perform local pooling processing in units of adjacent characters, global pooling processing in units of the entire text to be processed, and segmented pooling processing in units of segments based on the character vector and the position vector, and obtain text features corresponding to the text to be processed; a third acquisition module, used to obtain a corresponding encoding vector based on the text features; and a fourth acquisition module, used to perform target output processing on the encoding vector, and obtain a text processing result of the text to be processed.

[0011] According to the fourth aspect of the embodiments of the present application, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the text processing method described in the first aspect or the second aspect.

[0012] According to a fifth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the text processing method as described in the first aspect or the second aspect is implemented.

[0013] According to a sixth aspect of an embodiment of the present application, a computer program product is provided, comprising computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the text processing method described in the first aspect or the second aspect.

[0014] According to the text processing scheme provided by the embodiment of the present application, unlike the traditional Multi-Head Attention mechanism that requires the use of the Transformer model to model contextual relationships, in the scheme of the embodiment of the present application, a variety of pooling processing methods are used to achieve the same purpose. Among them, local pooling processing is used to perform pooling processing based on each character vector and the character vector of its adjacent characters, and a feature vector that integrates the semantics between each character and its adjacent characters can be obtained; global pooling processing is used to perform pooling processing based on the character vector of each character, and the character vectors of all other characters corresponding to each character and the entire text to be processed, and a feature vector that integrates the semantics of each character and the entire text to be processed can be obtained; segmented pooling processing is used to perform pooling processing based on the paragraphs where each character is located, and a feature vector that integrates the semantics of the entire paragraph can be obtained. It can be seen that the text features corresponding to the text to be processed obtained based on these feature vectors reflect the relationship between a certain character and other characters from multiple dimensions, thereby realizing contextual relationship modeling that does not rely on the Attention mechanism. Because the computational complexity of the pooling process is linearly related to the text length of the text to be processed, through the pooling process, on the basis of effectively obtaining text features, the computational complexity can also reach linear complexity. Compared with the computational complexity of the traditional Transformer model modeling contextual relationships O(n 2 ), which greatly reduces the amount of calculation and saves memory consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0016] Figure 1 This is a schematic diagram of the structure of a traditional Transformer model;

[0017] Figure 2A is a schematic diagram of a model structure including an encoder according to an embodiment of the present application;

[0018] Figure 2B for Figure 2A Schematic diagram of the pooling process of the local pooling layer in the encoder shown;

[0019] Figure 2C for Figure 2A Schematic diagram of the pooling process of the global pooling layer in the encoder shown;

[0020] Figure 2D for Figure 2ASchematic diagram of the pooling process of the segmented pooling layer in the encoder shown;

[0021] Figure 3A A flowchart of a text processing method according to Embodiment 1 of the present application;

[0022] Figure 3B for Figure 3A A schematic diagram of an example scenario of the illustrated embodiment;

[0023] Figure 4 A flowchart of a text processing method according to Embodiment 2 of the present application;

[0024] Figure 5 is a structural block diagram of a text processing device according to Embodiment 3 of the present application;

[0025] Figure 6 This is a schematic diagram of the structure of an electronic device according to the fourth embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0027] To facilitate understanding of the solution of the embodiment of the present application, the following Figures 2A-2D First, a model structure including an encoder provided in an embodiment of the present application is described.

[0028] Similar to the Transformer model, the encoder of the model in the embodiment of the present application also includes Nx identical encoding blocks Block, but the structure of each Block is different from the structure of the Block in the Encoder part of the Transformer model, and three different POOLING layers are used instead of the traditional Multi-Head Attention.

[0029] Among them, each Block is a functional unit that can convert the input vector from one data form to another data form. Specifically, in the embodiment of the present application, it can encode the input character vector and position vector into an encoding vector with semantic features. Pooling POOLING is a process of reducing the dimension and abstracting information. The information before entering the pooling layer is usually large in data volume, which contains a large amount of redundant information. Pooling reduces the dimension of information and reduces the data volume on the one hand; on the other hand, text features can be effectively extracted. In the embodiment of the present application, multiple pooling layers are used to replace the Attention mechanism of the Transformer model. Attention is similar to the attention mechanism of people as its translation "attention". The Attention mechanism in deep learning is similar to the selective visual attention mechanism of people. For example, in machine translation, if the attention mechanism is not used, when the sentence "It's a table" is translated into Chinese, the influence factor of each English word on the translated word "table" can also be simply understood as the same weight for it. However, in fact, it should be the word "table" that has the greatest influence on "table". When the Attention mechanism is introduced, a weight mechanism is added to each word to reflect the different influences of English words on the translation of the current Chinese word, thereby obtaining more effective semantic features. However, the computational complexity of Attention is quadratic with the length of the text (O(n 2 )) The computational complexity and memory consumption are high. However, through multiple pooling layers that pool information from different dimensions (local, global, segmented), combined with subsequent residual, feedforward and other processing, the needs of effective feature extraction for text can be fully met.

[0030] Based on this, Figure 2A As shown in , each Block includes: three parallel POOLING layers, namely local pooling, global pooling and segment pooling; after these three parallel POOLING layers, there is a first residual and normalization layer (Add&Norm); after the first residual and normalization layer is a feedforward network layer (Feed Forward); and, after the feedforward network layer, there is another residual and normalization layer, namely the second residual and normalization layer (Add&Norm). Among them, the residual is the difference between the actual observation value and the estimated value, which can be obtained through the residual function, which can prevent the gradient from disappearing or exploding due to the excessive depth of the model.

[0031] Among them, the local pooling layer is used to perform pooling processing based on each character vector and the character vectors of its adjacent characters, and a feature vector that integrates the semantics between each character and its adjacent characters can be obtained. An illustration of a local pooling layer performing pooling processing on character vectors is as shown in Figure 2B shown. As can be seen from Figure 2B , for the current character vector, such as the character vector shown in the middle of the three gray color blocks in the upper row of the figure, pooling processing is performed in combination with the character vector corresponding to the gray color block on its left and the character vector corresponding to the gray color block on its right, and the obtained result is as shown in Figure 2B the gray color block in the lower row of the figure. For example, for the text "Thank you, everyone", when performing local pooling processing on the character vector corresponding to "苦 (kǔ)", not only the character "苦 (kǔ)" itself is considered, but also the characters "辛 (xīn)" and "了 (le)" before and after it are considered, so that the feature vector of "苦 (kǔ)" obtained by pooling can integrate the information of the adjacent characters "辛 (xīn)" and "了 (le)".

[0032] Global pooling processing is used to perform pooling processing based on the character vectors of each character and the character vectors of all other characters corresponding to the entire text to be processed, and a feature vector that integrates the semantics of each character and the entire text to be processed can be obtained. An illustration of a global pooling layer performing pooling processing on character vectors is as shown in Figure 2C shown. As can be seen from Figure 2C , for the current character vector, such as the character vector of the leftmost gray color block in the upper row of the figure, pooling processing is performed in combination with the character vectors of all other characters in the entire text, such as the character vectors corresponding to the 2nd - 5th gray color blocks in the upper row on one side of the figure, and the obtained result is as shown in Figure 2C the gray color block in the lower row of the figure. For example, for the text "Thank you, everyone", when performing local pooling processing on the character vector corresponding to "辛 (xīn)", not only the character "辛 (xīn)" itself is considered, but also the other characters "苦了大家 (kǔ le dà jiā)" are considered, so that the feature vector of "辛 (xīn)" obtained by pooling can integrate the information of "苦了大家 (kǔ le dà jiā)".

[0033] Segment pooling processing is used to perform pooling processing based on the paragraphs where each character is located, taking paragraphs as units, and a feature vector that integrates the semantics of the entire paragraph can be obtained. An illustration of a segment pooling layer performing pooling processing on character vectors is as shown in Figure 2D shown. As can be seen from Figure 2D , the text is divided into two paragraphs, with the left three columns corresponding to one paragraph and the right two columns corresponding to another paragraph. Then, for each paragraph, the pooling processing of the current character vector will consider the character vectors of other characters in the paragraph to which the character corresponding to the current character vector belongs. For example, for the current character vectors of the first paragraph (the character vector corresponding to the leftmost gray block) and the second paragraph (the character vector corresponding to the second gray block from the right), performing paragraph pooling processing on them will obtain Figure 2D The feature vectors shown by the two gray color blocks in the bottom row in the middle. For example, for the text "You've had a hard time, everyone", it is divided into two paragraphs, namely the first paragraph "You've had a hard time" and the second paragraph "everyone". When performing segment pooling on the character vector corresponding to "Xin", not only the character "Xin" itself is considered, but also other characters "Ku le" in the first paragraph where it is located, so that the feature vector of "Xin" obtained by pooling can integrate the information of "Ku le". At the same time, when performing segment pooling on the character vector corresponding to "Da", not only the character "Da" itself is considered, but also other characters "Jia" in the second paragraph where it is located, so that the feature vector of "Da" obtained by pooling can integrate the information of "Jia".

[0034] After three pooling layers, there is an Add&Norm layer, whose function is similar to that of the Add&Norm in the encoder of the traditional Transformer model, where Add is used for residual connection and Norm is used for layer normalization.

[0035] The feed-forward network layer consists of a non-linear layer and a linear layer to further process the vector output by the Add&Norm layer, and perform feature extraction again through function approximation to obtain the required feature vector.

[0036] Another Add&Norm after the feed-forward network layer performs residual connection and normalization on the feature vector output by the feed-forward network.

[0037] After each Block completes the above process, an encoded vector will be output.

[0038] As Figure 2A shown, when the input information, including the character vector Input Embedding corresponding to the text, the position vector Position Embedding, and the paragraph marker information (used in the segment pooling layer), is sequentially processed by Nx Blocks of the encoder and then input to other parts of the model connected to the encoder, specifically the fully connected layer of the model, and then input to the output projection layer (Output Projection) after being processed by the fully connected layer, so as to perform the output of the entire model, such as the text classification result, or the text sequence annotation result, etc.

[0039] Hereinafter, in combination with the above encoder and the model where the encoder is located, the text processing solution of the present application will be described through multiple embodiments.

[0040] Embodiment 1

[0041] Refer to Figure 3A, shows a step flow chart of a text processing method according to embodiment 1 of the present application.

[0042] The text processing method of this embodiment includes the following steps:

[0043] Step S102: performing vectorization processing on the text to be processed to obtain a character vector and a position vector corresponding to the text to be processed.

[0044] In the embodiment of the present application, the text to be processed may be a text containing at least one of characters (letters, words, numbers, etc.), symbols, etc.

[0045] The vectorization of the text to be processed includes vectorization of each character in the text to generate a corresponding vector, i.e., a character vector. In addition, since the encoder of the embodiment of the present application does not have a sequential structure, it is necessary to add the position information of the character to explicitly indicate the character context relationship, and therefore, it is also necessary to generate a corresponding position vector.

[0046] In one possible approach, the position vector can be generated as follows:

[0047]

[0048]

[0049] Among them, PE is a two-dimensional matrix, pos represents the position of the character in the text, d model represents the dimension of the character vector, and i represents the position of the character vector in the text. Odd positions use cosine, and even positions use sin, so that vector information of different positions can be obtained according to different pos and i.

[0050] The vectorization processing of characters can be implemented by those skilled in the art using any appropriate method or algorithm, such as word2vec, etc., and the embodiments of the present application are not limited to this.

[0051] Step S104: Based on the character vector and the position vector, local pooling processing is performed based on adjacent characters, global pooling processing is performed based on the entire text to be processed, and segment pooling processing is performed based on segments to obtain text features corresponding to the text to be processed.

[0052] As mentioned above, local pooling is used to perform pooling based on each character vector and the character vectors of its adjacent characters, so as to obtain a feature vector that integrates the semantics of each character and its adjacent characters; global pooling is used to perform pooling based on the character vector of each character and the character vectors of all other characters corresponding to each character and the entire text to be processed, so as to obtain a feature vector that integrates the semantics of each character and the entire text to be processed; segmented pooling is used to perform pooling based on the paragraphs in which each character is located, so as to obtain a feature vector that integrates the semantics of the entire paragraph.

[0053] When the above encoder is used for processing, the character vector and the position vector can be input into the local pooling layer, the global pooling layer and the segmented pooling layer respectively to perform the corresponding local pooling processing, the global pooling processing and the segmented processing, and obtain the corresponding local pooling processing results, the global pooling processing results and the segmented pooling processing results. Then, the three parts of the pooling processing results can be merged to obtain the text features corresponding to the text to be processed, which not only can characterize the text features of the text to be processed, but also contain the contextual relationship between characters obtained from multiple dimensions.

[0054] Step S106: Obtain a corresponding encoding vector based on the text feature.

[0055] Based on the above-mentioned text features carrying contextual relationships, further operations such as residual connection, normalization, and feedforward network processing can be performed to obtain the encoding vector input by the encoder. For example, after the pooling results of the local pooling layer, the global pooling layer, and the segmented pooling layer in the above-mentioned encoder are merged, they can be sent to the Add&Norm layer, and then sent to the Feed Forward layer after processing by the Feed Forward layer, and then processed by the Add&Norm layer again, and finally the encoding vector is output.

[0056] Step S108: Perform target output processing on the encoding vector to obtain a text processing result of the text to be processed.

[0057] The target output processing is implemented according to the actual output requirements. For example, for text classification, text sequence labeling, etc., different output targets are set according to different output requirements to perform target output processing on the encoding vector for the output target, and finally obtain the text processing result of the text to be processed. Figure 2A As shown, the encoding vector output by the encoder is processed by the fully connected layer and the output mapping layer (different target output processing settings are performed according to different output targets), and then the text processing result can be output.

[0058] The following is an illustrative example of the above process using a specific scenario. Figure 3B shown.

[0059] This scenario example is set to automatically generate dialogue replies in a human-computer dialogue scenario. Suppose user A sends a voice to the smart speaker, "What's the weather like today? Is it suitable for traveling?" After the smart speaker receives the voice, it converts it into text. Then, the character vector, position vector and paragraph marking information corresponding to the text are generated. For example, each character in "What's the weather like today?" is marked as 0, indicating that it belongs to the first paragraph, and each character in "Is it suitable for traveling?" is marked as 1, indicating that it belongs to the second paragraph. However, it is not limited to this. In practical applications, other paragraph marking methods are also applicable. In addition, paragraph recognition and segmentation can also be implemented by those skilled in the art in accordance with actual needs in an appropriate manner, such as neural network model recognition and segmentation, finding symbols in the text through algorithms, and determining paragraphs after matching and discrimination. The embodiments of the present application do not limit the specific methods of paragraph recognition and marking.

[0060] After obtaining the character vector, position vector and paragraph mark information corresponding to the text "What is the weather like today? Is it suitable for traveling?", it is input into the encoder to perform local pooling processing on the character vector through the local pooling layer, global pooling processing through the global pooling layer, and segmented pooling processing through the segmented pooling layer; obtain the corresponding feature vector after local pooling processing, feature vector after global pooling processing and feature vector after segmented pooling processing; these three feature vectors are merged to form text features. The text features are then sequentially input into the Add&Norm layer of the encoder, and then sent to the Feed Forward layer after processing in the Add&Norm layer. After processing in the Feed Forward layer, it is processed again by the Add&Norm layer, and finally the encoding vector is output. The encoding vector carries rich semantic information and context information. Then, the encoding vector is input into the fully connected layer after the encoder for further feature extraction and classification, and then output through the output mapping layer. For example, in this example, the output is the text classification "weather". Then, based on the "weather" classification and the attribute information obtained from the text, such as "today" and "travel", the weather information of "today" can be obtained from the server storing the weather information and the travel strategy corresponding to the weather information, and the corresponding travel strategy can be found based on the weather information. Then, the obtained weather information and travel strategy are returned to the smart speaker, such as the data receiving unit in the smart speaker, and converted into voice playback by the smart speaker to answer the question of user A.

[0061] It should be noted that the above example takes a smart speaker as an example, but in actual applications, other smart devices that can receive voice input can also refer to the above example to implement the text processing method of the embodiment of the present application. When the smart device also has a voice output device such as a speaker, it can also be output after converting the above-obtained text into voice as described above. For example, the smart device receives an input voice question and converts the voice question into a corresponding question text; vectorizes the question text to obtain a character vector and a position vector corresponding to the question text; based on the character vector and the position vector, local pooling processing is performed in units of adjacent characters, global pooling processing is performed in units of the entire question text, and segmented pooling processing is performed in units of segments in the question text to obtain text features corresponding to the question text; based on the text features, a corresponding encoding vector is obtained; the encoding vector is subjected to question classification output processing to obtain a question classification corresponding to the question text; the answer text corresponding to the question text is obtained according to the question classification, and the answer text is converted into a voice answer and output through the smart device.

[0062] It should be noted that in this example, only the human-computer dialogue scenario is used as an example, but technical personnel in this field should understand that all applications and scenarios that require encoding text to obtain corresponding information can be applicable to the solutions of the embodiments of the present application, including but not limited to: for example, meeting minutes generation scenarios: including intelligent segmentation of meeting records, title generation, text summarization, reading comprehension, etc.; dialogue scenarios: including multi-round dialogue understanding, dialogue management, dialogue generation, dialogue response selection, etc.

[0063] Through this embodiment, unlike the traditional Multi-Head Attention mechanism that requires the use of the Transformer model to model contextual relationships, in the solution of this embodiment, multiple pooling processing methods are used to achieve the same purpose. Among them, local pooling processing is used to perform pooling processing based on each character vector and the character vectors of its adjacent characters, and a feature vector that integrates the semantics between each character and its adjacent characters can be obtained; global pooling processing is used to perform pooling processing based on the character vector of each character and the character vectors of all other characters corresponding to each character and the entire text to be processed, and a feature vector that integrates the semantics of each character and the entire text to be processed can be obtained; segmented pooling processing is used to perform pooling processing based on the paragraphs where each character is located, and a feature vector that integrates the semantics of the entire paragraph can be obtained. It can be seen that the text features corresponding to the text to be processed obtained based on these feature vectors reflect the relationship between a certain character and other characters from multiple dimensions, thereby realizing contextual relationship modeling that does not rely on the Attention mechanism. Because the computational complexity of the pooling processing is linearly related to the text length of the text to be processed, through the pooling processing, on the basis of effectively obtaining text features, the computational complexity can also reach linear complexity. Compared with the computational complexity of the traditional Transformer model modeling contextual relationships O(n 2 ), which greatly reduces the amount of calculation and saves memory consumption.

[0064] The text processing method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PCs, etc.

[0065] Embodiment 2

[0066] Reference Figure 4 , shows a step flow chart of a text processing method according to embodiment 2 of the present application.

[0067] The text processing method of this embodiment includes the following steps:

[0068] Step S202: Obtain the text to be processed.

[0069] In the embodiments of the present application, there is no limitation on the source of the text to be processed. It may be text originally, or text obtained by recognizing speech, or text obtained by recognizing a text image, or text obtained by performing text recognition on a text image area in a non-text image, etc., all of which are applicable to the text processing scheme of the embodiments of the present application.

[0070] Step S204: vectorize the text to be processed to obtain a character vector and a position vector corresponding to the text to be processed.

[0071] The vectorization processing of the processed text includes converting the processed text into a character vector after word segmentation. In the embodiment of the present application, the word segmentation of the processed text means dividing the processed text into individual characters such as individual words, and the converted character vector is word embedding.

[0072] The position vector (position embedding) of the characters in the text to be processed can be obtained by referring to the method described in step S102 of the first embodiment, which will not be described in detail here.

[0073] In addition, in the embodiment of the present application, optionally, before the processed text is vectorized, the processed text can also be segmented to obtain at least one text segment, and paragraph marking information is generated for the characters in each text segment. For example, the text of "I love you, China", its corresponding paragraph marking information can be

[00011] , wherein, "000" corresponds to "I love you" to indicate that these three characters are in the same paragraph, and "11" corresponds to "China" to indicate that these two characters are in the same paragraph. But not limited to this, other paragraph marking forms are also applicable. For example, only the characters at the end of the beginning of a paragraph are marked, such as marking [ST0] on "I", marking [SE0] on "you", marking [ST1] on "China", marking [SE1] on "country", etc., are all applicable to the scheme of the embodiment of the present application.

[0074] The character vector and position vector are added and then input into the encoder, while the paragraph tag information is separately input into the segment pooling layer in the encoder.

[0075] Step S206: Based on the character vector and the position vector, local pooling processing is performed in units of adjacent characters, global pooling processing is performed in units of the entire text to be processed, and segment pooling processing is performed in units of segments to obtain text features corresponding to the text to be processed.

[0076] in:

[0077] The local pooling process based on the character vector and the position vector and taking adjacent characters as units includes: for each character vector, determining the adjacent character vector that is closest to the character vector according to the position vector, and performing pooling process based on the character vector and the adjacent character vector. Figure 2B As shown, the current character and characters adjacent to the current character are determined, and the character vectors corresponding to these characters are pooled together to obtain the feature vector corresponding to the current character, that is, the result of local pooling processing.

[0078] The global pooling process based on the character vector and the position vector with the entire text to be processed as a unit includes: based on the position vector, all the character vectors corresponding to the text to be processed are pooled. An example is Figure 2C As shown, the current character and other characters in the text where the current character is located are determined, and the character vectors corresponding to these characters are pooled together to obtain the feature vector corresponding to the current character, that is, the result of the global pooling process.

[0079] The segmented pooling process based on the character vector and the position vector includes: obtaining the paragraph mark information corresponding to each character; and performing the pooling process based on the text segment marked by the paragraph mark information according to the character vector, the paragraph mark information and the position vector. Figure 2D As shown, the current character and other characters in the paragraph where the current character is located are determined, and the character vectors corresponding to these characters are pooled together to obtain the feature vector corresponding to the current character, that is, the result of the segmented pooling process.

[0080] If this step uses the above-mentioned encoder to perform vector pooling, this step can be implemented as follows: input the character vector and the position vector into the encoder, and perform corresponding local pooling, global pooling and segmented pooling respectively through the local pooling layer, global pooling layer and segmented pooling layer in the encoder. In the case where the paragraph mark information of the character is obtained in advance, the paragraph mark information will also be input into the encoder together to provide the segmentation information of the character for the segmented pooling layer to perform segmented pooling on the character. However, it is not limited to this. In practical applications, the generation part of the text segmentation and paragraph mark information can also be set inside the encoder, and the functional module of the corresponding paragraph recognition, segmentation and marking functions is set inside the encoder. In this case, there is no need to pre-generate the paragraph mark information and input it into the encoder. And by the way of pre-generation and input, on the one hand, the implementation complexity of the encoder is greatly simplified; on the other hand, the processing efficiency of the encoder and even the entire model is also improved.

[0081] After obtaining the results of the three pooling processes, they can be merged to obtain text features corresponding to the text to be processed. That is, the features obtained after the local pooling process, the features obtained after the global pooling process, and the features obtained after the segmented pooling process are merged, and the text features corresponding to the text to be processed are obtained according to the merged results. Among them, the specific feature merging method can be selected by those skilled in the art according to actual needs, including but not limited to the combine method, the concat method, etc. By merging the feature vectors obtained after multiple pooling processes to obtain text features, the text characteristics of the text to be processed can be effectively characterized, and rich contextual information can be carried.

[0082] In addition, it should be noted that the local pooling, global pooling and segmented pooling in the embodiment of the present application can all be maximum pooling. Alternatively, the local pooling, global pooling and segmented pooling in the embodiment of the present application can all be average pooling. When the maximum pooling method is used, the maximum value of the feature points in the field is taken, which can highlight useful or representative features. When the average pooling method is used, the average value of the feature points in the field is taken, which can retain more feature point information.

[0083] Step S208: Obtain a corresponding encoding vector based on the text feature.

[0084] For example, residual processing, normalization, feedforward processing and other operations can be performed on the text features to obtain the final encoding vector. When the above encoder is used to implement this step, the text features can be sequentially input into the first residual and normalization layer, the feedforward network layer and the second residual and normalization layer of the encoder for processing to obtain the encoding vector corresponding to the text features. The encoding vector thus obtained not only carries rich context information, but also can more effectively represent the text features.

[0085] Step S210: Perform target output processing on the encoding vector to obtain a text processing result of the text to be processed.

[0086] In this step, different target output processing can be performed on the encoded vector according to different output requirements to obtain text processing results that meet the requirements, such as text classification results, text sequence labeling results, etc. For example, the encoded vector is fully connected, and the vector after the fully connected processing is output mapped to obtain the text processing result of the text to be processed.

[0087] In adopting Figure 2A When using the model shown in the figure, this step can be implemented as follows: input the encoding vector into the fully connected layer (FC) for full connection processing; then input the output result of the fully connected layer into the output mapping layer (Output Projection) to achieve the final output that meets the requirements.

[0088] It can be seen that, through this embodiment, unlike the traditional Multi-Head Attention mechanism that requires the use of the Transformer model to model contextual relationships, in the solution of this embodiment, multiple pooling processing methods are used to achieve the same purpose. Among them, local pooling processing is used to perform pooling processing based on each character vector and the character vectors of its adjacent characters, and a feature vector that integrates the semantics between each character and its adjacent characters can be obtained; global pooling processing is used to perform pooling processing based on the character vector of each character, and the character vectors of all other characters corresponding to each character and the entire text to be processed, and a feature vector that integrates the semantics of each character and the entire text to be processed can be obtained; segmented pooling processing is used to perform pooling processing based on the paragraphs where each character is located, and a feature vector that integrates the semantics of the entire paragraph can be obtained. It can be seen that the text features corresponding to the text to be processed obtained based on these feature vectors reflect the relationship between a certain character and other characters from multiple dimensions, thereby realizing contextual relationship modeling that does not rely on the Attention mechanism. Because the computational complexity of the pooling processing is linearly related to the text length of the text to be processed, through the pooling processing, on the basis of effectively obtaining text features, the computational complexity can also reach linear complexity. Compared with the computational complexity of the traditional Transformer model modeling contextual relationships O(n 2 ), which greatly reduces the amount of calculation and saves memory consumption.

[0089] The text processing method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PCs, etc.

[0090] Embodiment 3

[0091] Reference Figure 5 , shows a structural block diagram of a text processing device according to embodiment three of the present application.

[0092] The text processing device of this embodiment includes: a first acquisition module 302, which is used to perform vectorization processing on the text to be processed to obtain the character vector and position vector corresponding to the text to be processed; a second acquisition module 304, which is used to perform local pooling processing in units of adjacent characters, global pooling processing in units of the entire text to be processed, and segment pooling processing in units of segments based on the character vector and the position vector, to obtain text features corresponding to the text to be processed; a third acquisition module 306, which is used to obtain the corresponding encoding vector based on the text features; and a fourth acquisition module 308, which is used to perform target output processing on the encoding vector to obtain the text processing result of the text to be processed.

[0093] Optionally, the second acquisition module 304 performs local pooling processing based on the character vector and the position vector in units of adjacent characters, including: for each character vector, determining the adjacent character vector closest to the character vector according to the position vector, and performing pooling processing based on the character vector and the adjacent character vector.

[0094] Optionally, the second acquisition module 304 performs global pooling processing based on the character vector and the position vector with the entire text to be processed as a unit, including: based on the position vector, performing pooling processing on all character vectors corresponding to the text to be processed.

[0095] Optionally, the text processing device of this embodiment further includes a generating module 310, which is used to segment the text to be processed, obtain at least one text segment, and generate paragraph mark information for the characters in each text segment before the first acquiring module 302 performs vectorization processing on the text to be processed. Based on this, the second acquiring module 304 performs segment pooling processing based on the character vector and the position vector in units of segments, including: obtaining the paragraph mark information corresponding to each character; and performing pooling processing based on the text segment marked by the paragraph mark information according to the character vector, the paragraph mark information and the position vector.

[0096] Optionally, the second acquisition module 304 obtains the text features corresponding to the text to be processed, including: merging the features obtained after local pooling processing, the features obtained after global pooling processing, and the features obtained after segmented pooling processing, and obtaining the text features corresponding to the text to be processed according to the merging result.

[0097] Optionally, the second acquisition module 304 performs local pooling processing, global pooling processing and segmented pooling processing based on the character vector and the position vector, respectively, including: inputting the character vector and the position vector into the encoder, and performing corresponding local pooling processing, global pooling processing and segmented pooling processing through the local pooling layer, global pooling layer and segmented pooling layer in the encoder, respectively.

[0098] Optionally, the encoder includes multiple encoding blocks connected in sequence, each encoding block includes a local pooling layer, a global pooling layer, and a segmented pooling layer: a first residual and normalization layer connected after the local pooling layer, the global pooling layer, and the segmented pooling layer; a feedforward network layer connected after the residual and normalization layer; and a second residual and normalization layer connected after the feedforward network layer.

[0099] Optionally, the third acquisition module 306 is used to input the text features into the first residual and normalization layer, the feedforward network layer and the second residual and normalization layer in sequence for processing to obtain the encoding vector corresponding to the text features.

[0100] Optionally, local pooling processing, global pooling processing and segmented pooling processing are all maximum pooling processing; or, local pooling processing, global pooling processing and segmented pooling processing are all average pooling processing.

[0101] Optionally, the fourth acquisition module 308 is used to perform full connection processing on the encoded vector, output mapping on the vector after the full connection processing, and obtain a text processing result of the text to be processed.

[0102] The text processing device of this embodiment is used to implement the corresponding text processing methods in the aforementioned multiple method embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. In addition, the functional implementation of each module in the text processing device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be repeated here.

[0103] Embodiment 4

[0104] Reference Figure 6 , shows a schematic diagram of the structure of an electronic device according to the fourth embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device.

[0105] like Figure 6 As shown, the electronic device may include: a processor (processor) 402 , a communication interface (Communications Interface) 404 , a memory (memory) 406 , and a communication bus 408 .

[0106] in:

[0107] The processor 402 , the communication interface 404 , and the memory 406 communicate with each other via a communication bus 408 .

[0108] The communication interface 404 is used to communicate with other electronic devices or servers.

[0109] The processor 402 is used to execute the program 410, and specifically can execute the relevant steps in the above text processing method embodiment.

[0110] Specifically, the program 410 may include program codes, which include computer operation instructions.

[0111] The processor 402 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0112] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0113] The program 410 can be specifically used to enable the processor 402 to execute any text processing method described in any of the aforementioned method embodiments.

[0114] The specific implementation of each step in program 410 can refer to the corresponding description of the corresponding steps and units in the above text processing method embodiment, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and achievable technical effects of the above-described devices and modules can refer to the corresponding process description in the above method embodiment, which will not be repeated here.

[0115] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to any text processing method in the above-mentioned multiple method embodiments.

[0116] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0117] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and to be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as an ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the text processing method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the text processing method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the text processing method shown here.

[0118] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0119] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A text processing method, include: Perform vectorization processing on the text to be processed to obtain a character vector and a position vector corresponding to the text to be processed; Based on the character vector and the position vector, local pooling processing is performed in units of adjacent characters to obtain a local pooling processing result; The global pooling process is performed on the entire text to be processed, and the global pooling process result is obtained; the segmented pooling process is performed on the segmented text, and the segmented pooling process result is obtained; Merging the local pooling processing result, the global pooling processing result and the segmented pooling processing result to obtain text features corresponding to the text to be processed; Obtaining a corresponding encoding vector based on the text feature; The encoding vector is subjected to target output processing to obtain a text processing result of the text to be processed.

2. The method according to claim 1, in, Based on the character vector and the position vector, local pooling processing is performed in units of adjacent characters, including: For each character vector, an adjacent character vector that is most adjacent to the character vector is determined according to the position vector, and pooling processing is performed based on the character vector and the adjacent character vectors.

3. The method according to claim 1, in, Based on the character vector and the position vector, a global pooling process is performed with the entire text to be processed as a unit, including: Based on the position vector, all character vectors corresponding to the text to be processed are pooled.

4. The method according to claim 1, in, Before vectorizing the text to be processed, the method further includes: segmenting the text to be processed to obtain at least one text segment, and generating paragraph mark information for characters in each text segment; Based on the character vector and the position vector, segment pooling processing is performed in units of segments, including: obtaining paragraph mark information corresponding to each character; and according to the character vector, the paragraph mark information and the position vector, pooling processing is performed in units of text segments marked by the paragraph mark information.

5. The method according to any one of claims 1 to 4, in, The obtaining of text features corresponding to the text to be processed includes: The features obtained after the local pooling process, the features obtained after the global pooling process, and the features obtained after the segmented pooling process are merged, and the text features corresponding to the text to be processed are obtained according to the merging result.

6. The method according to any one of claims 1 to 4, in, The method of performing local pooling processing based on adjacent characters as units, global pooling processing based on the entire text to be processed as units, and segment pooling processing based on segments as units based on the character vector and the position vector respectively includes: The character vector and the position vector are input into an encoder, and local pooling layer, global pooling layer and segment pooling layer in the encoder respectively perform corresponding local pooling processing, global pooling processing and segment pooling processing.

7. The method according to claim 6, in, The encoder includes a plurality of sequentially connected encoding blocks, each encoding block including the local pooling layer, the global pooling layer, and the segmented pooling layer; a first residual and normalization layer connected after the local pooling layer, the global pooling layer, and the segmented pooling layer; a feedforward network layer connected after the residual and normalization layer; and a second residual and normalization layer connected after the feedforward network layer.

8. The method according to claim 7, in, The obtaining a corresponding encoding vector based on the text feature includes: The text features are sequentially input into the first residual and normalization layer, the feedforward network layer and the second residual and normalization layer for processing to obtain the encoding vector corresponding to the text features.

9. The method according to any one of claims 1 to 4, in, The local pooling process, the global pooling process and the segmented pooling process are all maximum pooling processes; or, The local pooling process, the global pooling process and the segmented pooling process are all average pooling processes.

10. The method according to any one of claims 1 to 4, in, Performing target output processing on the encoding vector to obtain a text processing result of the text to be processed includes: Fully connect the encoded vector, perform output mapping on the fully connected vector, and obtain a text processing result of the text to be processed.

11. A text processing method, include: Receive a voice question input through a smart device, and convert the voice question into a corresponding question text; Performing vectorization processing on the question text to obtain a character vector and a position vector corresponding to the question text; Based on the character vector and the position vector, local pooling processing is performed in units of adjacent characters to obtain a local pooling processing result; Perform global pooling processing on the entire question text to obtain the global pooling processing result; Performing segment pooling processing based on the segments in the question text to obtain a segment pooling processing result; The local pooling processing result, the global pooling processing result and the segmented pooling processing result are combined to obtain text features corresponding to the question text; Obtaining a corresponding encoding vector based on the text feature; Performing question classification output processing on the encoding vector to obtain the question classification corresponding to the question text; According to the question classification, an answer text corresponding to the question text is obtained, and the answer text is converted into a voice answer and outputted through the smart device.

12. An electronic device, include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the text processing method according to any one of claims 1-11.

13. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the text processing method according to any one of claims 1 to 11 is implemented.

14. A computer program product, comprising computer instructions, wherein the computer instructions instruct a computing device to execute operations corresponding to the text processing method according to any one of claims 1 to 11.