Text processing method, text feature extraction method, device, equipment and medium

Abstract

A text processing method is provided, which relates to the field of artificial intelligence. The method comprises: inputting a first text into a pre-trained feature extractor to obtain a first text feature; inputting a second text into the pre-trained feature extractor to obtain a second text feature; obtaining a similarity prediction result between the first text feature and the second text feature; wherein the feature extractor comprises N cyclic encoders, each of which is constructed according to a cyclic memory mechanism and an encoder in a Transformer model, and the cyclic memory mechanism is used to process data according to state information at the previous moment and input information at this moment. A text feature extraction method is also provided, which comprises: inputting a third text into the pre-trained feature extractor to obtain a third text feature. A text processing device, a text feature extraction device, a device, a storage medium and a program product are also provided.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and more specifically to a text processing method, a text feature extraction method, an apparatus, a device, a medium, and a program product. Background Art

[0002] Using deep learning methods to process text can be done based on a recurrent neural network model or a convolutional neural network model. However, due to the problem of gradient diffusion, the structure of the recurrent neural network cannot be too deep in the vertical stacking, resulting in the loss of regularization and strong encoding capabilities brought by the deep network. The generalization performance is limited and cannot be improved with the help of large data sets. Convolutional neural networks are often used for images and are not suitable for processing the serialized data structure of text. Summary of the invention

[0003] In view of the above problems, the present disclosure provides a text processing method, a text feature extraction method, an apparatus, a device, a medium and a program product.

[0004] One aspect of an embodiment of the present disclosure provides a text processing method, including: inputting a first text into a pre-trained feature extractor to obtain a first text feature, wherein the first text includes text input by a user; inputting a second text into the pre-trained feature extractor to obtain a second text feature, wherein the second text includes a candidate text to be matched with the first text; obtaining a similarity prediction result between the first text feature and the second text feature; wherein the feature extractor includes N cyclic encoders, N is greater than or equal to 2, each of the cyclic encoders is constructed according to a cyclic memory mechanism and an encoder in a Transformer model, and the cyclic memory mechanism is used to process data according to state information at a previous moment and input information at this moment.

[0005] According to an embodiment of the present disclosure, the step of inputting the first text into a pre-trained feature extractor to obtain a first text feature includes: inputting the first text into N circular encoders to obtain the first text feature; wherein, for any two adjacent circular encoders, the step includes: using the output obtained by processing the first text by the previous circular encoder as the input of the next circular encoder.

[0006] According to an embodiment of the present disclosure, input data is obtained through a first text, the current moment includes the tth moment, the previous moment includes the t-1th moment, and the first text is input into N of the circular encoders to obtain the first text feature, including using each of the circular encoders to process the input data, specifically including: using the encoder to process the input data to obtain current input information at the tth moment, t being greater than or equal to 1; using the circular memory mechanism to process the current input information and the previous state information of the input data to obtain the current output of each of the circular encoders at the tth moment, wherein the previous state information is obtained at the t-1th moment.

[0007] According to an embodiment of the present disclosure, obtaining the current output of each of the cyclic encoders at the tth moment includes: obtaining the current historical information at the tth moment based on the current input information and the previous state information; obtaining the current state information at the tth moment based on the current input information and the current historical information; and obtaining the current output based on the current state information.

[0008] According to an embodiment of the present disclosure, obtaining the similarity prediction result between the first text feature and the second text feature includes: inputting the first text feature and the second text feature into a pre-trained matching interaction layer to obtain a matching confidence matrix; and calculating the similarity prediction result based on the matching confidence matrix.

[0009] According to an embodiment of the present disclosure, the pre-trained matching interaction layer includes a matching weight matrix and a mapping weight matrix, and the first text feature and the second text feature are input into the pre-trained matching interaction layer to obtain a matching confidence matrix, including: calculating a first matching matrix based on the matching weight matrix, the first text feature and the second text feature; calculating a second matching matrix based on the mapping weight matrix, the first text feature and the second text feature; and obtaining the matching confidence matrix based on the first matching matrix and the second matching matrix.

[0010] According to an embodiment of the present disclosure, calculating the similarity prediction result based on the matching confidence matrix includes: inputting the matching confidence matrix into a pre-trained bidirectional long short-term memory network layer to obtain a first intermediate feature; inputting the first intermediate feature into a pre-trained multi-layer perceptron layer to obtain a second intermediate feature; and obtaining the similarity prediction result based on the second intermediate feature.

[0011] According to an embodiment of the present disclosure, the text processing model includes the feature extractor, the matching interaction layer, the bidirectional long short-term memory network layer and the multi-layer perceptron layer. Before the second text is input into the pre-trained feature extractor, it also includes a training text processing model, which specifically includes: inputting the first training text and the second training text into the text processing model; using the feature extractor, the matching interaction layer, the bidirectional long short-term memory network layer and the multi-layer perceptron layer in the text processing model to process the first training text and the second training text to obtain training prediction results; training the text processing model according to the contrast loss between the training prediction results and the similarity labels.

[0012] According to an embodiment of the present disclosure, before inputting the first text into a pre-trained feature extractor, it also includes training the feature extractor, specifically including: pre-training the feature extractor using a first training set, wherein the first training set includes at least two training texts; fine-tuning the feature extractor obtained after pre-training using a second training set, wherein the second training set includes at least two training texts, and the number of training texts in the second training set is less than the number of training texts in the first training set.

[0013] According to an embodiment of the present disclosure, the second training set includes a third training text, and the fine-tuning of the feature extractor obtained after pre-training using the second training set includes: masking M words in the third training text, wherein the M words are used to characterize semantic information of the third training text, and M is greater than or equal to 1; and fine-tuning the feature extractor obtained after pre-training according to the third training text after masking.

[0014] According to an embodiment of the present disclosure, before performing masking processing on the M words in the third training text, it also includes determining the M words, specifically including: extracting the minimum number of words from the third training text as the M words when the semantic information does not change.

[0015] Another aspect of an embodiment of the present disclosure provides a text feature extraction method, comprising: inputting a third text into the pre-trained feature extractor as described above to obtain a third text feature, wherein the third text includes search content input by a user or content to be classified.

[0016] Another aspect of an embodiment of the present disclosure provides a text processing device, including: a first feature module, used to input a first text into a pre-trained feature extractor to obtain a first text feature, wherein the first text includes text input by a user; a second feature module, used to input a second text into the pre-trained feature extractor to obtain a second text feature, wherein the second text includes a candidate text to be matched with the first text; a similarity module, used to obtain a similarity prediction result between the first text feature and the second text feature; wherein the feature extractor includes N cyclic encoders, N is greater than or equal to 2, each of the cyclic encoders is constructed according to a cyclic memory mechanism and an encoder in a Transformer model, and the cyclic memory mechanism is used to process data according to state information at a previous moment and input information at this moment.

[0017] Another aspect of an embodiment of the present disclosure provides a text feature extraction device, including: a third feature module, used to input a third text into the pre-trained feature extractor as described above to obtain a third text feature, wherein the third text includes search content input by a user or content to be classified.

[0018] Another aspect of an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the method as described above.

[0019] Another aspect of the embodiments of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method as described above.

[0020] Another aspect of the embodiments of the present disclosure further provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0021] One or more of the above embodiments have the following beneficial effects: a cyclic memory mechanism is added to the encoder in the Transformer model to construct a cyclic encoder, which can process data according to the state information of the previous moment and the input information of this moment, thereby having the ability to cache historical information, and can achieve deep stacking in the vertical direction, which is suitable for processing the serialized data structure of text, and to a certain extent alleviates the problem of incomplete semantic extraction in the text processing process. Therefore, the feature extractor can obtain relatively complete semantics of the first text and the second text, and obtain more accurate similarity prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0023] Figure 1 An application scenario diagram suitable for implementing a text processing method or a text feature extraction method according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 2 The schematic diagram of the structure of the cyclic encoder according to the embodiment of the present disclosure is shown;

[0025] Figure 3 The structure diagram of the feature extractor according to the embodiment of the present disclosure is schematically shown;

[0026] Figure 4 A flowchart of a feature extractor training method according to an embodiment of the present disclosure is schematically shown;

[0027] Figure 5 A flowchart of a fine-tuning phase according to an embodiment of the present disclosure is schematically shown;

[0028] Figure 6 The structure diagram of the text processing model according to the embodiment of the present disclosure is schematically shown;

[0029] Figure 7 A flowchart schematically shows a method for training a text processing model according to an embodiment of the present disclosure;

[0030] Figure 8 A flowchart of a text processing method according to an embodiment of the present disclosure is schematically shown;

[0031] Fig. 9 A flowchart schematically shows data processing according to an embodiment of the present disclosure;

[0032] Fig.10 A flowchart for calculating the current output at time t according to an embodiment of the present disclosure is schematically shown;

[0033] Fig.11 A flowchart of calculating similarity prediction results according to an embodiment of the present disclosure is schematically shown;

[0034] Fig.12 The flowchart of obtaining the matching confidence matrix according to the embodiment of the present disclosure is schematically shown;

[0035] Fig.13 Schematically shows a flow chart of calculating similarity prediction results according to another embodiment of the present disclosure;

[0036] Fig.14 The structure block diagram of the text processing device according to the embodiment of the present disclosure is schematically shown;

[0037] Fig.15 A block diagram of an electronic device suitable for implementing a text processing method or a text feature extraction method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0038] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0039] In the related art, in addition to text processing based on recurrent neural networks or convolutional neural networks, text processing can be implemented based on pre-trained models, but there is a limit on the number of characters that can be input. For example, the BERT (Bidirectional Encoder Representations from Transformers) model supports a maximum input of 510 characters. Text processing of long documents, that is, long text processing, may far exceed 510 characters. In one processing method of the BERT model, a segmentation method (Pooling method) is used to convert long texts into a processable form. Specifically, the entire long document is divided into multiple fixed-length segments (segments), and the number of characters in each segment does not exceed 510. The vector representation of each segment is calculated by the BERT model, and finally the document vector is obtained by pooling. Since each segment is a fixed fragment, there is no correlation between each segment, which will cause the semantic incompleteness of the segment for long texts.

[0040] Embodiments of the present disclosure provide a text processing method, a text feature extraction method, an apparatus, a device, a medium, and a program product. The text processing method comprises: inputting a first text into a pre-trained feature extractor to obtain a first text feature, wherein the first text includes text input by a user. Inputting a second text into a pre-trained feature extractor to obtain a second text feature, wherein the second text includes a candidate text to be matched with the first text. Obtaining a similarity prediction result between the first text feature and the second text feature. The feature extractor comprises N cyclic encoders, N being greater than or equal to 2, each cyclic encoder being constructed based on a cyclic memory mechanism and an encoder in a Transformer model, and the cyclic memory mechanism being used to process data based on state information at a previous moment and input information at this moment.

[0041] According to the embodiments of the present disclosure, a cyclic memory mechanism is added to the encoder in the Transformer model to construct a cyclic encoder, which can process data according to the state information of the previous moment and the input information of this moment, thereby having the ability to cache historical information and being suitable for processing the serialized data structure of text, and to a certain extent alleviating the problem of incomplete semantic extraction during text processing. Therefore, the feature extractor can obtain relatively complete semantics of the first text and the second text, and obtain more accurate similarity prediction results.

[0042] The text processing method, text feature extraction method, device, equipment, medium and program product of the embodiments of the present disclosure can be applied to text retrieval, news recommendation, text clustering, text deduplication or machine translation in the financial field. It can also be used in any field other than the financial field, and the application field of the text processing method, text feature extraction method, device, equipment, medium and program product of the embodiments of the present disclosure is not limited.

[0043] Figure 1 An application scenario diagram suitable for implementing a text processing method or a text feature extraction method according to an embodiment of the present disclosure is schematically shown.

[0044] like Figure 1 As shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0045] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0046] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0047] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0048] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0049] The following will be based on Figure 1 The scene described by Figure 2~Figure 13 The text feature extraction method and text processing method of the embodiments of the present disclosure are described in detail.

[0050] Figure 2 The structural diagram of the cyclic encoder according to the embodiment of the present disclosure is schematically shown. Figure 3 The structural diagram of the feature extractor according to the embodiment of the present disclosure is schematically shown.

[0051] like Figure 2 As shown, the cyclic encoder 200 includes a Transformer Encoder (ie, encoder) module 210 and a cyclic memory module 220 .

[0052] The Encoder module 210 may include at least a multi-head self-attention layer, two add & norm layers, and a feed forward network feed forward layer.

[0053] For example, given a sentence or a paragraph as input data (Inputs), first perform an input embedding operation, that is, convert the text in the Inputs into corresponding word vectors. Then, perform a Positional encoding operation to perform position encoding. Finally, input it into the Encoder structure 210 for processing.

[0054] The loop memory module 220 is constructed according to the loop memory mechanism and can cache segment history information. By caching segment history information and controlling the gate mechanism, the loop encoder 200 can alleviate the problem of incomplete segment semantics to a certain extent. The loop encoder 200 can be called a Recurrent-Encoder, or rEncoder for short.

[0055] Exemplarily, the cyclic encoder 200 is constructed based on the Encoder of the natural language processing model Transformer. On the one hand, it can achieve deep stacking in the vertical direction to obtain a large model structure, and on the other hand, it can obtain the output as a fixed-length vector, which can be convenient for subsequent processing.

[0056] like Figure 3 As shown, the feature extractor 300 can be considered as a deep learning model, which includes N stacked recurrent encoders (rEncoder) 200, where N is an integer greater than or equal to 1. Similar to the long short-term memory neural network, the rEncoder in the feature extractor 300 can use the information of the previous moment and the input information of this moment to jointly process data.

[0057] Figure 4 The flowchart of the feature extractor training method according to the embodiment of the present disclosure is schematically shown.

[0058] like Figure 4 As shown, the feature extractor training method of this embodiment includes operations S410 to S420. Pre-training refers to the process of pre-training a model. Fine-tuning is the process of applying the pre-trained model to a data set of a specific scene and adapting the parameters to the data set.

[0059] In operation S410, a feature extractor is pre-trained using a first training set, wherein the first training set includes at least two training texts.

[0060] Exemplarily, the training of the feature extractor adopts the training mode of the large model, which is divided into two stages: pre-training and fine-tuning. The goal of the feature extractor in the pre-training stage is to train a feature extractor with sufficiently strong feature representation capabilities, and a large amount of data in the Internet public database trained with the large model can be used as the first training set.

[0061] In operation S420, the feature extractor obtained after pre-training is fine-tuned using a second training set, wherein the second training set includes at least two training texts, and the number of training texts in the second training set is less than the number of training texts in the first training set.

[0062] For example, during the pre-training or fine-tuning stage, the cross entropy loss function can be used to calculate the loss between the training samples and the labels, and the training effect can be determined based on the degree of convergence of the loss.

[0063] Exemplarily, a data set sampled according to a specific scenario is used as the second training set for fine-tuning. The specific scenario includes news, literature, financial reports, or reviews.

[0064] According to the embodiments of the present disclosure, by training a feature extractor in two stages, namely, pre-training and fine-tuning, a feature extractor having a strong feature extraction capability for texts in specific scenarios can be obtained.

[0065] Figure 5 The flowchart of the fine-tuning stage according to an embodiment of the present disclosure is schematically shown.

[0066] like Figure 5 As shown, fine-tuning the pre-trained feature extractor in operation S420 includes operation S510 to operation S520. The second training set includes a third training text, and the third training text can be any one in the second training set.

[0067] In operation S510 , masking is performed on M words in the third training text, where the M words are used to represent semantic information of the third training text, and M is greater than or equal to 1.

[0068] According to an embodiment of the present disclosure, before performing masking processing on the M words in the third training text, the M words are also determined, specifically including: extracting the minimum number of words from the third training text as the M words without changing the semantic information.

[0069] The purpose of training the feature extractor is to improve the ability to extract the best semantic features of the document. In order to achieve the best semantic extraction training of the feature extractor, the dataset is labeled based on the best semantic principle after sampling. The best semantic principle means that the label information contains the shortest description of the complete semantics of the document (such as a sentence or a paragraph).

[0070] In some embodiments, some words are masked during the feature extractor training process to restore them through feature extraction, thereby allowing the feature extractor to learn the structural semantic information of the sentence.

[0071] Table 1

[0072]

[0073] Table 1 schematically shows the structure of the input data of the feature extractor. <s>is the start mark of input, ···[mask]··· indicates input, [mask] indicates mask, <e>Indicates the end mark. The difference between the pre-training stage and the fine-tuning stage is that the words that are masked in the fine-tuning stage can obtain the complete semantics of the sentence through the sequential combination in the original sentence, that is, the shortest meaning masking implemented according to the shortest meaning principle. The pre-training stage adopts a random masking strategy.

[0074] According to the embodiments of the present disclosure, the text in a segment includes multiple words, some of which have no effect on semantics, such as modal particles or function words, etc. The masking effect of these words is not ideal, and if these words are labeled, the workload or computing resources will increase. By obtaining M words through the shortest meaning principle and masking the M words according to the shortest meaning mask, the workload of labeling training samples can be reduced.

[0075] In operation S520, the feature extractor obtained after pre-training is fine-tuned according to the third training text after masking.

[0076] According to an embodiment of the present disclosure, fine-tuning can be accomplished by training with a small number of samples.

[0077] The text feature extraction method provided by the embodiment of the present disclosure includes: inputting a third text into a pre-trained feature extractor 300 to obtain a third text feature. The third text includes search content or content to be classified input by a user. Figure 3 As shown, each recurrent encoder is constructed according to the recurrent memory mechanism and the encoder in the Transformer model. The recurrent memory mechanism is used to process data according to the state information of the previous moment and the input information of this moment.

[0078] In some embodiments, after the user enters the search content, the semantic feature can be extracted as the third text feature. Then, the candidate content is matched according to the third text feature. Finally, the target content is displayed according to the matching result to realize the semantic search function. For example, if the user enters a description of a loan business, one or more loan business information that matches the description can be returned by extracting the semantic feature of the description.

[0079] In other embodiments, the comments, news or financial reports crawled by users on the web can be subjected to classification operations such as topic identification, extraction or clustering. Taking the comments on a certain financial business as an example, the semantic features of each comment can be extracted. Then, the comment opinions can be extracted and analyzed based on the semantic features, so as to analyze the advantages and disadvantages of the financial business.

[0080] According to an embodiment of the present disclosure, inputting the third text into a pre-trained feature extractor 300 to obtain the third text feature includes: inputting the third text into N cyclic encoders to obtain the third text feature. The N cyclic encoders process the input data in series, and for any two adjacent cyclic encoders, the steps include: using the output obtained by processing the first text by the previous cyclic encoder as the input of the next cyclic encoder.

[0081] Reference Figure 2 and Figure 3 , taking any cyclic encoder as an example, the process of obtaining the third text feature is further explained. Wherein, the current moment includes the tth moment, and the previous moment includes the t-1th moment.

[0082] Exemplarily, the third text is subjected to Input Embedding and Positional Encoding processing to obtain input data. In some embodiments, the third text may be a long text, and multiple segments may be obtained based on the third text. According to the order of words in each segment and the order of each segment in the long text, each moment corresponds to a certain word as the time axis. In some embodiments, the third text may be a segment, in which the order of words is used as the time axis. At the initial moment (t=0), the previous state information is obtained according to the previous segment of the third text in the long text, for example, the state information is obtained according to the last word of the previous segment. The previous state information includes the comprehensive semantic information of the words at the previous moment and all words before the previous moment.

[0083] Next, the encoder is used to process the input data to obtain the current input information x at time t t , t is greater than or equal to 1. Use the loop memory mechanism to process the current input information and the previous state information of the input data h t-1 , get the current output y of each cyclic encoder at time t t , the previous state information h t-1 Obtained at time t-1.

[0084] Specifically, according to the current input information x t and the previous status information h t-1 , get the current historical information at time t t . t It is obtained by the following formula 1):

[0085] Formula 1)

[0086] Among them, tanh represents the hyperbolic tangent function, W h is the parameter determined during the training process, and γ represents the previous state information h t-1 The control threshold of the current historical information. γ is obtained by the following formula 2):

[0087] Formula 2)

[0088] Among them, σ represents the sigmoid function, W γ are parameters determined during the training process.

[0089] According to the current input information x t and current historical information t , get the current state information h at time t t .h t The following formula 3) is used to obtain:

[0090] Formula 3)

[0091] Among them, α represents the current input information x t and current historical information t The control threshold of the encoder flowing into the next cycle. α is obtained by the following equation 4):

[0092] Formula 4)

[0093] Among them, σ represents the sigmoid function, W α are parameters determined during the training process.

[0094] According to the current status information h t , get the current output y t .y t The following formula 5) is used to obtain:

[0095] Formula 5)

[0096] Among them, σ represents the sigmoid function, W O are parameters determined during the training process.

[0097] According to an embodiment of the present disclosure, the previous state information h t-1 According to x at time t-1 t-1 and t-1 Get, and so on.

[0098] According to the embodiments of the present disclosure, the feature extractor has a strong feature extraction capability of a large model, which can better extract the features of the document and improve the accuracy of text retrieval. The Recurrent Encoder has the ability to cache historical information, which can alleviate the problem of incomplete segment semantics to a certain extent. Compared with the commonly used RNN-type models, the model using the Transformer Encoder can more effectively utilize ultra-large data sets for training, improve the upper limit of text retrieval capabilities, especially improve the retrieval capabilities for long documents.

[0099] Exemplarily, based on the feature extractor 300, a text processing model is constructed based on the idea of ​​a twin network. The twin network refers to a network comprising two sub-networks with the same structure and shared weights, each of which receives an input and outputs a corresponding representation, and then the semantic relationship between the two inputs is obtained by calculating the distance between the two representations.

[0100] Figure 6 The structural diagram of the text processing model according to the embodiment of the present disclosure is schematically shown.

[0101] like Figure 6 As shown in FIG. 1 , the text processing model includes two feature extractors (i.e., Extractors), a matching interaction layer (MIL), a bidirectional long short-term memory network (Bi-Direction LongShort Term Memory, BiLSTM), a multi-layer perceptron (MLP), and a SoftMax layer. In some embodiments, the SoftMax layer can be replaced by a sigmoid layer.

[0102] Figure 7 A flowchart of a method for training a text processing model according to an embodiment of the present disclosure is schematically shown.

[0103] like Figure 7 As shown, the training method of the text processing model of this embodiment includes operations S710 to S730.

[0104] In operation S710, a first training text and a second training text are input to a text processing model.

[0105] Table 2

[0106]

[0107] Table 2 schematically shows the input format of the text processing model, where P (first training text) and Q (second training text) are sentences whose similarity needs to be calculated. score is the manually labeled similarity score label of P and Q.

[0108] In operation S720, the first training text and the second training text are processed using the feature extractor, the matching interaction layer, the bidirectional long short-term memory network layer, and the multi-layer perceptron layer in the text processing model to obtain a training prediction result.

[0109] Reference Figure 6 ,First, sentences P and Q are feature extracted by the ,feature extractor.

[0110] Then, the text features obtained based on P and Q are respectively input to the MIL layer. The MIL layer is configured to receive the text features output by the two feature extractors and calculate the matching confidence matrix of the two.

[0111] Exemplarily, the matching confidence matrix F is calculated according to the following formula 5):

[0112] Formula 6)

[0113] in and Respectively express sentences , Features, and is a learnable multi-dimensional similarity matching weight matrix and a linear mapping weight matrix. is the bias term. Represents the matching confidence matrix. f() represents a preset function, which may not affect the calculation result in the brackets, or may further process the calculation result, which can be selected according to the actual situation.

[0114] Next, the matching confidence matrix is ​​input to the BiLSTM layer. The BiLSTM layer is configured to process the matching confidence matrix to obtain the first intermediate feature. The BiLSTM layer squeezes the matching confidence matrix to obtain a fixed-length feature vector, namely the first intermediate feature.

[0115] Next, the first intermediate feature is input to the MLP layer, which is configured to process the first intermediate feature to obtain a second intermediate feature.

[0116] Finally, the second intermediate feature is input to the SoftMax layer. The SoftMax layer is configured to process the second intermediate feature based on the SoftMax function to obtain a similarity prediction result.

[0117] In operation S730, a text processing model is trained according to the contrast loss between the training prediction results and the similarity labels.

[0118] For example, the contrast loss can be calculated using the SmoothL1 loss function, and the training effect can be judged according to the degree of convergence of the loss.

[0119] In some embodiments, the feature extractor can be trained first, and the trained feature extractor can converge on the training set and have sufficient feature extraction capabilities. Then it can participate in the training of the text processing model. In other embodiments, the untrained feature extractor can be involved in the training process of the text processing model. Since the text processing model adopts an end-to-end training method, after the text processing model training is completed, it is considered that the feature extractor is also trained.

[0120] After training the text processing model, you can use the model to process text, which is further described below.

[0121] Figure 8 The flowchart of the text processing method according to the embodiment of the present disclosure is schematically shown.

[0122] like Figure 8 As shown, the text processing method of this embodiment includes operations S810 to S830.

[0123] In operation S810, a first text is input into a pre-trained feature extractor to obtain a first text feature, wherein the first text includes text input by a user.

[0124] In operation S820, the second text is input into a pre-trained feature extractor to obtain second text features, wherein the second text includes candidate texts to be matched with the first text.

[0125] For example, refer to Figure 6 , the first text (P) and the second text (Q) can be input into two feature extractors respectively. In other embodiments, the first text can also be input into a feature extractor first, and then the second text can be input into the feature extractor after the first text feature is obtained.

[0126] Exemplarily, text retrieval is also called natural language retrieval, which means searching for target data directly by matching terms in natural language through computers without any indexing of documents. In the text retrieval scenario, the first text can be a search text input by a user, such as a keyword, sentence, paragraph, or a long text consisting of multiple paragraphs. The second text can be a candidate text within the search scope of the search engine, such as content stored in a local database or content obtained on the Internet.

[0127] In operation S830, a similarity prediction result between the first text feature and the second text feature is obtained.

[0128] Reference Figure 6 , the similarity prediction result can be output through the SoftMax layer, which can include the probability value calculated using the SoftMax function. In other embodiments, the similarity prediction result can be output through the sigmod layer, which can include the probability value calculated using the sigmod function.

[0129] According to the embodiments of the present disclosure, a cyclic memory mechanism is added to the encoder in the Transformer model to construct a cyclic encoder, which can process data according to the state information of the previous moment and the input information of this moment, thereby having the ability to cache historical information, and can achieve deep stacking in the vertical direction, which is suitable for processing the serialized data structure of text, and to a certain extent alleviates the problem of incomplete semantic extraction in the text processing process. Therefore, the feature extractor can obtain relatively complete semantics of the first text and the second text, and obtain more accurate similarity prediction results.

[0130] Referring to the above process of using the feature extractor to process the third text to obtain the third text features, the process of using the feature extractor to obtain the first text features is described again below by taking the processing of the first text as an example.

[0131] According to an embodiment of the present disclosure, the first text is input into a pre-trained feature extractor to obtain the first text feature, which includes: inputting the first text into N circular encoders to obtain the first text feature. Wherein, for any two adjacent circular encoders, it includes: using the output obtained by processing the first text with the previous circular encoder as the input of the next circular encoder. By processing the input data in series through N circular encoders, a deep stacking with a deeper vertical direction can be achieved, which has the regularization and strong encoding ability of a deep network and has good generalization performance.

[0132] Fig. 9 The flowchart of data processing according to an embodiment of the present disclosure is schematically shown.

[0133] like Fig. 9 As shown, inputting the first text into N circular encoders to obtain the first text feature includes using each circular encoder to process the input data, which may specifically include operations S910 to S920.

[0134] In operation S910 , input data is processed using an encoder to obtain current input information x(t) at time t, where t is greater than or equal to 1.

[0135] Exemplarily, for the first loop encoder, the input data processed by it is obtained by performing InputEmbedding and Positional Encoding on the first text. For non-first loop encoders, the input data processed by it is the output of the previous loop encoder.

[0136] Exemplarily, the order of words in the first text is used as the time axis.

[0137] In operation S920, the current input information x(t) and the previous state information h of the input data are processed using a loop memory mechanism. t-1 , get the current output y of each cyclic encoder at time t t , the previous state information h t-1 Obtained at time t-1.

[0138] Fig.10 The flowchart of calculating the current output at the tth moment according to an embodiment of the present disclosure is schematically shown.

[0139] like Fig.10 As shown, obtaining the current output of each cyclic encoder at the tth time in operation S920 includes operations S1010 to S1030.

[0140] In operation S1010, according to the current input information x t and the previous status information h t-1 , get the current historical information at time t t , please refer to formula 1).

[0141] In operation S1020, according to the current input information x t and current historical information t , get the current state information h at time t t , please refer to formula 3).

[0142] In operation S1030, according to the current state information h t , get the current output y t , please refer to formula 5).

[0143] Exemplarily, the process of obtaining the second text feature is the same as the process of obtaining the first text feature, and is not described in detail here.

[0144] Fig.11 The flowchart of calculating the similarity prediction result according to the embodiment of the present disclosure is schematically shown.

[0145] like Fig.11 As shown, obtaining the similarity prediction result between the first text feature and the second text feature in operation S230 includes operations S1110 to S1120.

[0146] In operation S1110, the first text feature and the second text feature are input into a pre-trained matching interaction layer to obtain a matching confidence matrix, which can be obtained by referring to Formula 6).

[0147] In operation S1120, a similarity prediction result is calculated according to the matching confidence matrix.

[0148] According to an embodiment of the present disclosure, the correlation between the first text feature and the second text feature is first calculated using the matching interaction layer, and represented by a matching confidence matrix to facilitate subsequent processing and improve the accuracy of the similarity calculation result.

[0149] Fig.12 The flowchart of obtaining the matching confidence matrix according to an embodiment of the present disclosure is schematically shown.

[0150] like Fig.12 As shown, the matching confidence matrix obtained in operation S1110 includes operations S1210 to S1230. The pre-trained matching interaction layer includes a matching weight matrix and a mapping weight matrix.

[0151] In operation S1210, a first matching matrix is ​​calculated based on a matching weight matrix, a first text feature, and a second text feature.

[0152] Referring to Equation 6), the first matching matrix can be obtained by "Part. Among them, the matching weight matrix is ​​M, and are the first text feature and the second text feature respectively.

[0153] In operation S1220, a second matching matrix is ​​calculated based on the mapping weight matrix, the first text feature, and the second text feature.

[0154] Referring to Equation 6), the second matching matrix can be obtained by "Part of it is obtained. Among them, the mapping weight matrix is .

[0155] In operation S1230, a matching confidence matrix is ​​obtained based on the first matching matrix and the second matching matrix.

[0156] Fig.13 The flowchart of calculating the similarity prediction result according to another embodiment of the present disclosure is schematically shown.

[0157] like Fig.13 As shown, calculating the similarity prediction result according to the matching confidence matrix in operation S1120 includes operations S1310 to S1330.

[0158] In operation S1310, the matching confidence matrix is ​​input into a pre-trained bidirectional long short-term memory network layer to obtain a first intermediate feature.

[0159] In operation S1320, the first intermediate feature is input into a pre-trained multi-layer perceptron layer to obtain a second intermediate feature.

[0160] In operation S1330, a similarity prediction result is obtained according to the second intermediate feature.

[0161] Reference Figure 6 , the second intermediate feature can be input to the SoftMax layer. The SoftMax layer is configured to process the second intermediate feature based on the SoftMax function to obtain a similarity prediction result.

[0162] Based on the above feature extractor training method, the present disclosure also provides a feature extractor training device.

[0163] According to an embodiment of the present disclosure, a feature extractor training device includes a pre-training module and a fine-tuning module.

[0164] The pre-training module may perform operation S410 for pre-training the feature extractor using a first training set, wherein the first training set includes at least two training texts.

[0165] The fine-tuning module may perform operation S420 to fine-tune the feature extractor obtained after pre-training using a second training set, wherein the second training set includes at least two training texts, and the number of training texts in the second training set is less than the number of training texts in the first training set.

[0166] The fine-tuning module can also perform operations S510 to S520, which are not described in detail herein. Before performing masking on the M words in the third training text, the M words are also determined, specifically including: extracting the minimum number of words from the third training text as the M words without changing the semantic information.

[0167] Based on the above text feature extraction method, the present disclosure also provides a text feature extraction device.

[0168] According to an embodiment of the present disclosure, the text feature extraction device includes a third feature module. The third feature module is used to input a third text into a pre-trained feature extractor to obtain a third text feature, wherein the third text includes search content or content to be classified input by a user. The feature extractor includes N cyclic encoders, N is greater than or equal to 2, and each cyclic encoder is constructed according to a cyclic memory mechanism and an encoder in a Transformer model, and the cyclic memory mechanism is used to process data according to state information at the previous moment and input information at this moment.

[0169] Based on the above text processing model training method, the present disclosure also provides a text processing model training device.

[0170] According to an embodiment of the present disclosure, a text processing model training device includes a text input module, a text processing module and a model training module.

[0171] The text input module may perform operation S710 for inputting the first training text and the second training text into the text processing model.

[0172] The text processing module can perform operation S720 to process the first training text and the second training text using the feature extractor, matching interaction layer, bidirectional long short-term memory network layer and multi-layer perceptron layer in the text processing model to obtain a training prediction result.

[0173] The model training module may perform operation S730 for training the text processing model according to the contrast loss between the training prediction results and the similarity labels.

[0174] Based on the above text processing method, the present disclosure also provides a text processing device. Fig.14 The device is described in detail.

[0175] Fig.14 The structural block diagram of the text processing device according to an embodiment of the present disclosure is schematically shown.

[0176] like Fig.14 As shown, the text processing device 1400 of this embodiment includes a first feature module 1410 , a second feature module 1420 and a similarity module 1430 .

[0177] The first feature module 1410 may perform operation S810 to input a first text into a pre-trained feature extractor to obtain a first text feature, wherein the first text includes text input by a user.

[0178] The first feature module 1410 is further used to input the first text into N circular encoders to obtain the first text feature. For any two adjacent circular encoders, the output obtained by processing the first text with the previous circular encoder is used as the input of the next circular encoder.

[0179] The first feature module 1410 may also perform operations S910 to S920 and operations S1010 to S1030, which are not described in detail herein.

[0180] The second feature module 1420 may perform operation S820 for inputting the second text into a pre-trained feature extractor to obtain second text features, wherein the second text includes candidate text to be matched with the first text.

[0181] The similarity module 1430 may perform operation S830 to obtain a similarity prediction result between the first text feature and the second text feature.

[0182] The similarity module 1430 may perform operations S1110 to S1120, operations S1210 to S1220, and operations S1310 to S1320, which will not be described in detail herein.

[0183] It should be noted that the implementation methods, technical problems solved, functions realized, and technical effects achieved of each module / unit / sub-unit in the device part embodiment are the same or similar to the implementation methods, technical problems solved, functions realized, and technical effects achieved of each corresponding step in the method part embodiment, and will not be repeated here.

[0184] According to an embodiment of the present disclosure, any multiple modules in the feature extractor training device, the text feature extraction device, the text processing model training device, or the text processing device can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module.

[0185] According to an embodiment of the present disclosure, at least one module in a feature extractor training device, a text feature extraction device, a text processing model training device, or a text processing device can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one module in a feature extractor training device, a text feature extraction device, a text processing model training device, or a text processing device can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0186] Fig.15 A block diagram of an electronic device suitable for implementing a text processing method or a text feature extraction method according to an embodiment of the present disclosure is schematically shown.

[0187] like Fig.15 As shown, the electronic device 1500 according to an embodiment of the present disclosure includes a processor 1501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1502 or a program loaded from a storage part 1508 to a random access memory (RAM) 1503. The processor 1501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 1501 may also include an onboard memory for caching purposes. The processor 1501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0188] In RAM 1503, various programs and data required for the operation of electronic device 1500 are stored. Processor 1501, ROM 1502 and RAM 1503 are connected to each other through bus 1504. Processor 1501 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1502 and / or RAM 1503. It should be noted that the program can also be stored in one or more memories other than ROM 1502 and RAM 1503. Processor 1501 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0189] According to an embodiment of the present disclosure, the electronic device 1500 may further include an input / output (I / O) interface 1505, which is also connected to the bus 1504. The electronic device 1500 may further include one or more of the following components connected to the I / O interface 1505: an input portion 1506 including a keyboard, a mouse, etc. An output portion 1507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc. A storage portion 1508 including a hard disk, etc. And a communication portion 1509 including a network interface card such as a LAN card, a modem, etc. The communication portion 1509 performs communication processing via a network such as the Internet. A drive 1510 is also connected to the I / O interface 1505 as needed. A removable medium 1511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1510 as needed, so that the computer program read therefrom is installed into the storage portion 1508 as needed.

[0190] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments. It may also exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0191] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1502 and / or RAM 1503 described above and / or one or more memories other than ROM 1502 and RAM 1503.

[0192] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.

[0193] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 1501. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0194] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1509, and / or installed from the removable medium 1511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0195] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1509, and / or installed from the removable medium 1511. When the computer program is executed by the processor 1501, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0196] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0197] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0198] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0199] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.< / e> < / s>

Claims

1. A text processing method, comprising: Inputting a first text into a pre-trained feature extractor to obtain a first text feature, wherein the first text includes text input by a user; Inputting a second text into the pre-trained feature extractor to obtain second text features, wherein the second text includes candidate text to be matched with the first text; Obtaining a similarity prediction result between the first text feature and the second text feature; The feature extractor includes N cyclic encoders, where N is greater than or equal to 2, and each of the cyclic encoders is constructed according to a cyclic memory mechanism and an encoder in a Transformer model, and the cyclic memory mechanism is used to process data according to state information at a previous moment and input information at this moment; Wherein, for any two adjacent cyclic encoders, including: Using the output obtained by processing the first text by the previous circular encoder as the input of the next circular encoder; The input data is obtained through the first text, the current moment includes the tth moment, the previous moment includes the t-1th moment, the first text is input into N of the cyclic encoders, and obtaining the first text feature includes using each of the cyclic encoders to process the input data, specifically including: Processing the input data using the encoder to obtain current input information at the t-th time, where t is greater than or equal to 1; The current input information and the previous state information of the input data are processed by the cyclic memory mechanism to obtain the current output of each cyclic encoder at the tth moment, wherein the previous state information is obtained at the t-1th moment, and the previous state information includes the comprehensive semantic information of the words at the previous moment and all words before the previous moment.

2. The method according to claim 1, wherein: Inputting the first text into a pre-trained feature extractor to obtain the first text feature comprises: The first text is input into the N cyclic encoders to obtain the first text feature.

3. The method according to claim 1, wherein: The obtaining of the current output of each of the cyclic encoders at the tth moment comprises: Obtaining the current historical information at the tth moment according to the current input information and the previous state information; Obtaining current state information at time t according to the current input information and the current historical information; The current output is obtained according to the current state information.

4. The method according to claim 1, wherein: The obtaining of the similarity prediction result between the first text feature and the second text feature comprises: Inputting the first text feature and the second text feature into a pre-trained matching interaction layer to obtain a matching confidence matrix; The similarity prediction result is calculated according to the matching confidence matrix.

5. The method according to claim 4, wherein: The pre-trained matching interaction layer includes a matching weight matrix and a mapping weight matrix, and the first text feature and the second text feature are input into the pre-trained matching interaction layer to obtain a matching confidence matrix, including: Calculate a first matching matrix based on the matching weight matrix, the first text feature and the second text feature; Calculate a second matching matrix based on the mapping weight matrix, the first text feature and the second text feature; The matching confidence matrix is ​​obtained based on the first matching matrix and the second matching matrix.

6. The method according to claim 4, wherein: The calculating the similarity prediction result according to the matching confidence matrix includes: Inputting the matching confidence matrix into a pre-trained bidirectional long short-term memory network layer to obtain a first intermediate feature; Inputting the first intermediate feature into a pre-trained multi-layer perceptron layer to obtain a second intermediate feature; The similarity prediction result is obtained according to the second intermediate feature.

7. The method according to claim 6, wherein: The text processing model includes the feature extractor, the matching interaction layer, the bidirectional long short-term memory network layer and the multi-layer perceptron layer. Before the second text is input into the pre-trained feature extractor, the text processing model is further trained, specifically including: Inputting a first training text and a second training text into the text processing model; Processing the first training text and the second training text by using the feature extractor, the matching interaction layer, the bidirectional long short-term memory network layer and the multi-layer perceptron layer in the text processing model to obtain a training prediction result; The text processing model is trained according to the contrast loss between the training prediction results and the similarity labels.

8. The method according to claim 1, wherein: Before inputting the first text into the pre-trained feature extractor, the method further includes training the feature extractor, specifically including: Pre-training the feature extractor using a first training set, wherein the first training set includes at least two training texts; The feature extractor obtained after pre-training is fine-tuned using a second training set, wherein the second training set includes at least two training texts, and the number of training texts in the second training set is less than the number of training texts in the first training set.

9. The method according to claim 8, wherein: The second training set includes a third training text, and fine-tuning the feature extractor obtained after pre-training by using the second training set includes: Performing masking processing on M words in the third training text, wherein the M words are used to represent semantic information of the third training text, and M is greater than or equal to 1; The feature extractor obtained after pre-training is fine-tuned according to the third training text after mask processing.

10. The method according to claim 9, wherein: Before performing masking processing on the M words in the third training text, the method further includes determining the M words, specifically including: When the semantic information does not change, a minimum number of words are extracted from the third training text as the M words.

11. A text feature extraction method, comprising: Input the third text into the pre-trained feature extractor according to claim 1 to obtain third text features, wherein the third text includes search content input by the user or content to be classified.

12. A text processing device, comprising: A first feature module, configured to input a first text into a pre-trained feature extractor to obtain a first text feature, wherein the first text includes text input by a user; A second feature module, configured to input a second text into the pre-trained feature extractor to obtain a second text feature, wherein the second text includes a candidate text to be matched with the first text; A similarity module, used to obtain a similarity prediction result between the first text feature and the second text feature; The feature extractor includes N cyclic encoders, where N is greater than or equal to 2, and each of the cyclic encoders is constructed according to a cyclic memory mechanism and an encoder in a Transformer model, and the cyclic memory mechanism is used to process data according to state information at a previous moment and input information at this moment; Wherein, for any two adjacent cyclic encoders, including: Using the output obtained by processing the first text by the previous circular encoder as the input of the next circular encoder; The input data is obtained through the first text, the current moment includes the tth moment, the previous moment includes the t-1th moment, the first text is input into N of the cyclic encoders, and obtaining the first text feature includes using each of the cyclic encoders to process the input data, specifically including: Processing the input data using the encoder to obtain current input information at the t-th time, where t is greater than or equal to 1; The current input information and the previous state information of the input data are processed by the cyclic memory mechanism to obtain the current output of each cyclic encoder at the tth moment, wherein the previous state information is obtained at the t-1th moment, and the previous state information includes the comprehensive semantic information of the words at the previous moment and all words before the previous moment.

13. A text feature extraction device, comprising: The third feature module is used to input the third text into the pre-trained feature extractor according to claim 12 to obtain the third text feature, wherein the third text includes the search content input by the user or the content to be classified.

14. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 11.

15. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 11.

16. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images