Text data processing method and device, electronic equipment and medium

Abstract

The invention relates to the technical field of text data processing, and provides a text data processing method and device, electronic equipment and a medium. The method comprises the following steps: acquiring training sample data, and inputting the training sample data into a recall model; respectively carrying out vector representation processing on the training query text and the training matching text to obtain a feature vector sequence of the training query text and a feature vector sequence of the training matching text; performing self-attention processing on each feature vector in the feature vector sequence of the training query text and each feature vector in the feature vector sequence of the training matching text to obtain an interaction matrix between the training query text and the training matching text; determining maximum edge loss according to an interaction matrix between the training query text and the training matching text; and updating parameters in the recall model based on the maximized edge loss. By introducing the interaction information, the problem caused by neglecting the interaction information between the query and the text in the training process is avoided.

Description

Technical Field

[0001] The present disclosure relates to the technical field of text data processing, and in particular to a text data processing method, device, electronic device and medium. Background Art

[0002] In the field of text retrieval, traditional methods are mainly based on the bag-of-words model or vector space model, using indicators such as word frequency, inverse document frequency, and cosine similarity to calculate the similarity between the query and the text. However, these methods have the disadvantage of being unable to process semantic and grammatical information, and cannot effectively deal with problems such as polysemy, multiple words with one meaning, and semantic matching.

[0003] In recent years, neural retrieval models based on pre-trained language models, such as the dual encoder model, have made significant progress. The dual encoder model uses two pre-trained language models as encoders to encode the query and text respectively, and then calculates the similarity between them. This model can make full use of the semantic and grammatical representation capabilities of the pre-trained language model to improve the retrieval effect. In addition, since the encoding can be completed before retrieval, the retrieval efficiency is maintained. However, the current dual encoder model ignores the interactive information between the query and the text during training, that is, the matching or alignment relationship between the query and each word or phrase in the text. This defect leads to low accuracy of the output text when using the model. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide a text data processing method, device, electronic device and computer-readable storage medium to solve the technical problem in the prior art that the current dual encoder model ignores the interaction information between the query and the text during the training process, resulting in low accuracy of the output text when using the model.

[0005] In a first aspect of an embodiment of the present disclosure, a text data processing method is provided, the method comprising: obtaining training sample data and inputting the training sample data into a recall model, the training sample data comprising training query text and training matching text; performing vector representation processing on the training query text and the training matching text respectively to obtain a feature vector sequence of the training query text and a feature vector sequence of the training matching text; performing self-attention processing on each feature vector in the feature vector sequence of the training query text and each feature vector in the feature vector sequence of the training matching text to obtain an interaction matrix between the training query text and the training matching text; determining a maximum edge loss based on the interaction matrix between the training query text and the training matching text; and updating parameters in the recall model based on the maximum edge loss.

[0006] According to a second aspect of an embodiment of the present disclosure, a text data processing device is provided, which includes: an acquisition module, which is used to acquire training sample data and input the training sample data into a recall model, wherein the training sample data includes training query text and training matching text; a vector representation module, which is used to perform vector representation processing on the training query text and the training matching text respectively, to obtain a feature vector sequence of the training query text and a feature vector sequence of the training matching text; a self-attention processing module, which is used to perform self-attention processing on each feature vector in the feature vector sequence of the training query text and each feature vector in the feature vector sequence of the training matching text, to obtain an interaction matrix between the training query text and the training matching text; a loss determination module, which is used to determine the maximum edge loss based on the interaction matrix between the training query text and the training matching text; and a parameter updating module, which is used to update the parameters in the recall model based on the maximum edge loss.

[0007] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0008] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0009] Compared with the prior art, the embodiments of the present disclosure have the following beneficial effects: the embodiments of the present disclosure can effectively capture the interactive information of each feature vector between the query and the text by performing vector representation processing on the training query text and the training matching text and applying the self-attention mechanism. This helps the model to more comprehensively understand the semantic and grammatical relationship between the query and the text, helps to improve the accuracy and efficiency of text retrieval, and thus improves the user experience. Updating the model parameters by maximizing the edge loss can effectively reduce the risk of overfitting of the model, and help to improve the stability and reliability of the model in real scenarios. Training the model in this way can learn richer semantic information, thereby improving the generalization ability of the model. This enables the model to better adapt to different contexts and semantic relationships when processing new query texts and matching texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied;

[0012] Figure 2 It is a flowchart of a text data processing method provided by an embodiment of the present disclosure;

[0013] Figure 3 is a flowchart of another text data processing method provided by an embodiment of the present disclosure;

[0014] Figure 4 is a flowchart of another text data processing method provided by an embodiment of the present disclosure;

[0015] Figure 5 is a block diagram of a text data processing device provided by an embodiment of the present disclosure;

[0016] Figure 6 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0018] It should be noted that the user information (including but not limited to terminal device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0019] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of an embodiment of the present invention can be applied.

[0020] like Figure 1 As shown, the system architecture 100 may include one or more of a first terminal device 101, a second terminal device 102, and a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0021] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0022] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send data, etc. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, desktop computers, and the like.

[0023] The server 105 may be a server that provides various services. For example, the server 105 may obtain training sample data from the first terminal device 103 (or the second terminal device 102 or the third terminal device 103), and input the training sample data into the recall model, wherein the training sample data includes a training query text and a training matching text; the training query text and the training matching text are respectively subjected to vector representation processing to obtain a feature vector sequence of the training query text and a feature vector sequence of the training matching text; self-attention processing is performed on each feature vector in the feature vector sequence of the training query text and each feature vector in the feature vector sequence of the training matching text to obtain an interaction matrix between the training query text and the training matching text; the maximum edge loss is determined according to the interaction matrix between the training query text and the training matching text; and the parameters in the recall model are updated based on the maximum edge loss.

[0024] In some embodiments, the text data processing method provided in the embodiments of the present invention is generally executed by the server 105, and accordingly, the text data processing device is generally disposed in the server 105. In other embodiments, some terminal devices may have functions similar to those of the server to execute the method. Therefore, the text data processing method provided in the embodiments of the present invention is not limited to being executed on the server side.

[0025] The text data processing method and device according to the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0026] Figure 2 is a flowchart of a text data processing method provided by an embodiment of the present disclosure. The method provided by an embodiment of the present disclosure can be executed by any electronic device with computer processing capabilities, for example, the electronic device can be Figure 1 The server is shown.

[0027] like Figure 2 As shown, the text data processing method includes steps S210 to S250.

[0028] In step S210, training sample data is obtained and input into a recall model, wherein the training sample data includes a training query text and a training matching text.

[0029] Step S220 , performing vector representation processing on the training query text and the training matching text respectively, to obtain a feature vector sequence of the training query text and a feature vector sequence of the training matching text.

[0030] In step S230, self-attention processing is performed on each feature vector in the feature vector sequence of the training query text and each feature vector in the feature vector sequence of the training matching text to obtain an interaction matrix between the training query text and the training matching text.

[0031] In step S240, the maximum edge loss is determined according to the interaction matrix between the training query text and the training matching text.

[0032] In step S250 , the parameters in the recall model are updated based on the maximized marginal loss.

[0033] This method can effectively capture the interactive information of each feature vector between the query and the text by performing vector representation processing on the training query text and the training matching text and applying the self-attention mechanism. This helps the model to more comprehensively understand the semantic and grammatical relationship between the query and the text, helps to improve the accuracy and efficiency of text retrieval, and thus improves the user experience. Updating the model parameters by maximizing the marginal loss can effectively reduce the risk of overfitting of the model and help improve the stability and reliability of the model in real scenarios. Training the model in this way can learn richer semantic information, thereby improving the generalization ability of the model. This enables the model to better adapt to different contexts and semantic relationships when processing new query texts and matching texts.

[0034] In some embodiments of the present disclosure, the recall model includes a dual encoder model and an implicit interaction module. The dual encoder model is a neural network structure, usually composed of two encoders, which are used to encode query text and matching text respectively. During the training process, the dual encoder model converts the training query text and the training matching text into corresponding feature vector sequences. These feature vector sequences capture the semantic and grammatical information of the text, and convert the text content into a high-dimensional vector representation for subsequent processing. The implicit interaction module is a key component in the recall model, and an interaction matrix is ​​generated by processing the feature vector sequence output by the dual encoder model. This interaction matrix describes the interaction relationship between the training query text and the training matching text, that is, it shows the correlation and matching degree between each feature vector between the texts. Through the implicit interaction module, the model can better understand the semantic relationship between the query text and the matching text, thereby improving the matching accuracy.

[0035] In some embodiments of the present disclosure, in a text retrieval task, the training sample data includes training query text and training matching text. The training query text is the historical query text entered by the user in various retrieval systems or question-answering systems. These query texts represent the description of the user's demand for information or questions. The role of the training query text is to enable the training model to understand the user's query intent, so that it can accurately match the relevant text content. The training matching text includes annotated positive training matching text and negative training matching text. The positive training matching text represents the text with a high degree of matching with the query text, while the negative training matching text represents the text with a low degree of matching with the query text. This distinction enables the model to learn to distinguish between relevant and irrelevant text content, thereby improving the accuracy of matching. The positive training matching text is the text related to the query text, and its matching degree is high. These texts are usually determined based on domain knowledge or manual annotations to ensure that they have obvious semantic or thematic relevance to the query text. The model learns how to correctly match the relevant content through the positive training matching text during the training process. The negative training matching text is the text that is irrelevant to the query text, and its matching degree is low. These texts are usually randomly selected from texts that have never been matched, or generated by some negative sampling techniques. During the training process, the model needs to learn how to distinguish negative training matching text from query text in order to correctly filter out relevant content. By training query text and training matching text in the training sample data, the model can learn how to accurately match query and text content, thereby improving the accuracy and effectiveness of text retrieval.

[0036] In some embodiments of the present invention, in text processing tasks, before using the dual encoder model, a series of preprocessing operations are usually performed on the text to reduce noise, extract key information, and provide better input for the model. These preprocessing operations include word segmentation, stop word removal, and stemming. Word segmentation is the process of dividing text into words or subwords. In natural language processing, text is usually represented by a continuous sequence of characters, which needs to be converted into words or subwords that the model can understand. For example, after the sentence "I love natural language processing" is segmented, ["I", "love", "natural", "language", "processing"] may be obtained. Stop words refer to words that appear frequently in the text but usually do not carry much information, such as "the", "is", "and", etc. Removing these stop words can reduce noise and improve the effect of the model. For example, after removing stop words in a sentence, it may become ["love", "natural", "language", "processing"]. Stemming is the process of converting words into their stems or original forms, which helps to merge words into the same form and reduce the interference caused by vocabulary variants to the model. For example, different forms of words (such as "running", "ran", "runs") are converted to their stems (such as "run") or original forms. Through the above preprocessing operations, the text can be more standardized, clean, and more in line with the input requirements of the model. These operations help improve the training effect of the model, reduce noise interference, and also improve the generalization ability of the model, so that it can better handle various text data.

[0037] In some embodiments of the present disclosure, the training query text and the training matching text are respectively subjected to vector representation processing to obtain the feature vector sequence of the training query text and the feature vector sequence of the training matching text, including: encoding each word in the training query text by a first encoder to obtain the feature vector sequence of the training query text; encoding each word in the training matching text by a second encoder to obtain the feature vector sequence of the training matching text. For example, for the training query text, the first encoder is used to encode each word therein. This process can be to obtain the vector representation of each word through a pre-trained word embedding model (such as Word2Vec, GloVe, BERT, etc.), or to learn the representation of each word through a neural network structure (such as LSTM, Transformer, etc.). After being processed by the first encoder, each word is converted into a vector representation. The vector representations of these words are combined together to obtain a feature vector sequence of the training query text, wherein each vector represents a word or subsequence in the text. For the training matching text, the second encoder is also used to encode each word therein. This encoder can be the same as the encoder for processing the query text, or it can be a different structure, depending on the specific task and model design. After being processed by the second encoder, each word is converted into a vector representation. Combining the vector representations of these words together, a feature vector sequence of the training matching text can be obtained, where each vector represents a word or subsequence in the text. Through the above process, the training query text and the training matching text are respectively converted into feature vector sequences, which capture the semantic and grammatical information of the text and provide input for the subsequent matching and interaction modules. This vector representation processing enables the model to better understand the text content and perform effective semantic matching in the matching process.

[0038] In some embodiments of the present disclosure, self-attention processing is performed on each feature vector in the feature vector sequence of the training query text and each feature vector in the feature vector sequence of the training matching text through the self-attention mechanism in the implicit interaction module. For example, self-attention processing is performed on each feature vector in the feature vector sequence of the training query text and each feature vector in the feature vector sequence of the training matching text to obtain the interaction matrix between the training query text and the training matching text, including: calculating the correlation between the feature vector sequence of the training query text and each feature vector in the feature vector sequence of the training matching text based on the self-attention algorithm; determining the interaction matrix between the training query text and the training matching text according to the correlation between the feature vector sequence of the training query text and each feature vector in the feature vector sequence of the training matching text. Specifically, the self-attention mechanism allows the model to perform weighted attention on elements at different positions in the sequence in order to better capture the dependency between the elements. Here, each feature vector can be regarded as a position in the sequence. Based on the self-attention algorithm, the correlation is calculated for each feature vector in the feature vector sequence of the training query text and the feature vector sequence of the training matching text. By calculating the correlation between each pair of feature vectors in the two sequences, an interaction matrix can be obtained, in which each element represents the degree of correlation between the feature vectors at corresponding positions. According to the correlation between the feature vector sequence of the training query text and the corresponding position of each feature vector in the feature vector sequence of the training matching text, the interaction matrix between the training query text and the training matching text is determined. This interaction matrix reflects the degree of semantic association between the training query text and the training matching text, which helps the model better understand the matching relationship between the texts. In this way, the self-attention mechanism can be used to interactively process the feature vector sequences of the training query text and the training matching text, so as to obtain the interactive information between them. This interactive information can help the model to match text more accurately and improve the accuracy and effect of matching.

[0039] Based on the above embodiment, first, the dual encoder model is used to encode the training query text and the training matching text to obtain their vector representations, which are denoted as q and p, where and n and m are the number of words in the training query text and the training matching text, respectively, and d is the dimension of the vector. The vector representation of the training query text and the training matching text in this embodiment uses the output of all positions of the last layer of the encoder, that is, each word or phrase has a corresponding vector representation. Then, the self-attention mechanism is used to calculate the implicit interaction matrix between the training query text and the training matching text, denoted as M, where

[0040] Figure 3FIG. 1 is a flow chart of another text data processing method provided by an embodiment of the present disclosure. Figure 3 The above step S240 may specifically include steps S310 to S330.

[0041] In step S310, a first average value of the interaction matrix between the training query text and the training matching text is calculated.

[0042] In step S320, a second average value of the interaction matrix between the training query text and the negative training matching text is calculated.

[0043] In step S330, the maximum marginal loss is determined according to the first average value, the second average value, and the hyper parameters of the recall model.

[0044] This method can maximize the marginal loss, and the recall model can learn a better text matching representation, making positive samples closer and negative samples farther away, thereby improving the performance of the text matching task.

[0045] In some embodiments, a first average value of the interaction matrix between the training query text and the positive training matching text is calculated. A second average value of the interaction matrix between the training query text and the negative training matching text is calculated. Based on the first average value, the second average value, and the hyperparameters of the recall model, the maximum edge loss is determined. For example, first, the first average value of the interaction matrix between the training query text and the positive training matching text is calculated. This represents the average degree of interaction between the positive samples. Then, the second average value of the interaction matrix between the training query text and the negative training matching text is calculated. This represents the average degree of interaction between the negative samples. Maximizing edge loss is a commonly used training method that aims to ensure that the similarity between positive samples is higher than the similarity between negative samples by a certain margin. This helps the model better distinguish between matching and unmatched text pairs. The difference between the positive sample score and the negative sample score, plus a margin value. If the difference is less than the margin value, the model will be penalized. By comparing the first average value and the second average value, and combining the hyperparameters of the recall model (such as the margin value), the specific form and calculation method of the edge loss can be determined. During the training process, the goal of the model is to maximize the edge loss so that the similarity between the positive samples is as high as possible and the similarity between the negative samples is as low as possible. Through the back-propagation algorithm, the model can update parameters according to the gradient of edge loss, thereby continuously optimizing the performance of the model and improving the accuracy and generalization ability of text matching.

[0046] Based on the above embodiment, in the training process of the recall model, the interaction matrix is ​​used as an auxiliary supervisory signal to optimize the parameters of the dual encoder model. The interaction matrix can reflect the correlation between the query and the matching text, that is, the value of the interaction matrix of the positive sample (text related to the query) should be greater than the value of the interaction matrix of the negative sample (text irrelevant to the query). Therefore, the maximum edge loss is used to achieve this goal, and its calculation formula is:

[0047] L MML =max(0,γ-M pos +M neg )

[0048] Among them, the maximum edge loss is L MML , M pos is the first average value, M neg is the second mean, γ is a hyperparameter, and represents the margin.

[0049] In some embodiments, the method further includes: generating a pseudo query text based on a feature vector sequence of a training matching text; performing vector representation processing on the pseudo query text to obtain a feature vector sequence of the pseudo query text; calculating a first similarity between a feature vector sequence of the training query text and a feature vector sequence of the training matching text; calculating a second similarity between a feature vector sequence of the pseudo query text and a feature vector sequence of the training matching text; and calculating the minimum cross entropy loss based on the first similarity and the second similarity. In this embodiment, the recall model further includes a pseudo query generation module for generating a pseudo query text based on the training matching text, that is, a text with a low correlation with the training matching text. For example, first, a pseudo query text is generated based on a feature vector sequence of the training matching text. Specifically, similar query texts can be constructed using information from matching texts to expand the training data set. The generated pseudo query text is vector represented to obtain a feature vector sequence of the pseudo query text. The first similarity between the feature vector sequence of the training query text and the feature vector sequence of the training matching text is calculated. This indicates the similarity between the real query text and the matching text in the training data. Then, the second similarity between the feature vector sequence of the pseudo query text and the feature vector sequence of the training matching text is calculated. This indicates the similarity between the generated pseudo query text and the matching text. Minimizing the cross entropy loss is a commonly used loss function that measures the difference between the probability distribution of the model output and the true label. In this context, the first similarity and the second similarity can be used to calculate the cross entropy loss between the model's prediction results and the actual label. Through this process, the model can learn a more effective text matching representation. At the same time, by introducing pseudo query text, the diversity of training data can be increased and the generalization ability of the model can be improved.

[0050] Based on the above embodiment, a pre-trained language model is used as a pseudo query generation module, i.e., a generator, to generate pseudo query text based on the matching text. The generator can be any Transformer-based model, such as BERT. Finally, the pseudo query text is used as an auxiliary supervisory signal to optimize the parameters of the dual encoder model. This goal is achieved by minimizing the cross entropy loss, which is calculated as follows:

[0051]

[0052] Among them, L MCEL To minimize the cross entropy loss, s pos is the first similarity, s neg is the second similarity.

[0053] In some embodiments of the present disclosure, the above method further includes: determining the target loss according to maximizing the edge loss and minimizing the cross entropy loss; and updating the parameters in the recall model according to the target loss. In this embodiment, updating the parameters in the recall model can be understood as updating the parameters in the dual encoder model. For example, a comprehensive target loss function can be defined by combining maximizing the edge loss and minimizing the cross entropy loss. This target loss function can be composed of a weighted sum of the edge loss term and the cross entropy loss term, and the weight can be adjusted according to the task requirements and model performance. During the training process, the parameters in the recall model (i.e., the parameters in the dual encoder model in this example) can be updated by minimizing the defined target loss function. The process of updating the parameters is usually implemented by a back propagation algorithm, which adjusts the model parameters according to the gradient information of the loss function so that the target loss function value is continuously reduced, thereby improving the performance of the model. The dual encoder model is an architecture commonly used in text matching tasks, consisting of two encoders, which encode the query text and the matching text respectively, and then make matching judgments by comparing the encoded representations. When updating the parameters of the dual encoder model, by optimizing the target loss function, the model will learn better text representations and matching strategies to improve matching accuracy and generalization ability. By comprehensively considering the marginal loss and the cross entropy loss and updating the parameters of the dual encoder model, the model can better learn the feature representation in the text matching task, thereby improving the performance and effect of the model.

[0054] Based on the above embodiment, a method of optimizing the parameters of the model by maximizing the edge loss and minimizing the cross entropy loss is used. The calculation formula of the loss function is:

[0055] L=L MML +λL MCEL

[0056] Among them, L is the target loss, L MML To maximize the marginal loss, L MCELTo minimize the cross entropy loss, λ is a hyperparameter that represents the weight of the two losses.

[0057] Figure 4 FIG. 1 is a flowchart of another text data processing method provided by an embodiment of the present disclosure. Figure 4 As shown, the above method may further include steps S410 to S430.

[0058] In step S410, the query text currently input by the terminal device is obtained.

[0059] In step S420, the query text is input into the recall model, the query text is recognized by the recall model, and scores of multiple matching texts are output.

[0060] In step S430, a target text corresponding to the query text is determined according to the scores of the multiple matching texts, and the target text is sent to the terminal device.

[0061] The method can obtain the query text currently input by the terminal device. The query text is input into the recall model, the query text is recognized and processed by the recall model, and the scores of multiple matching texts are output. The target text corresponding to the query text is determined according to the scores of multiple matching texts, and the target text is sent to the terminal device, which can help the system to effectively process the query text and output the text result that best matches it, thereby improving the accuracy and efficiency of the retrieval system or question-answering system.

[0062] In some embodiments of the present disclosure, after the recall model is trained by the above training method, the query text of the retrieval system or the question-answering system can be processed by the trained recall model. For example, the query text currently input by the terminal device is obtained. The query text is input into the recall model, the query text is recognized and processed by the recall model, and the scores of multiple matching texts are output. The target text corresponding to the query text is determined according to the scores of the multiple matching texts, and the target text is sent to the terminal device. Specifically, first, the query text currently input by the terminal device is obtained, for example, it can be the text input by the user in the search engine or the question-answering system. The query text is input into the trained recall model. In this example, the query text and the matching text in the database are encoded by the dual encoder to obtain the corresponding feature vector sequence. The dual encoder model encodes the query text and the matching text in the database into feature vector sequences respectively, so as to perform subsequent matching and scoring calculations. According to the encoded feature vector sequence, the recall model calculates the similarity or matching score between the query text and the matching text in the database. The output multiple matching texts will be scored, and the score usually indicates the matching degree of the query text with each matching text. Based on the scores of multiple matching texts, the recall model determines the best matching target text, that is, the text that is most relevant to the query text. Finally, the determined target text will be sent back to the terminal device to be displayed to the user or used for subsequent information processing or retrieval tasks. Through this process, the recall model can help the system effectively process the query text and output the text results that best match it, thereby improving the accuracy and efficiency of the retrieval system or question-answering system.

[0063] Based on the above embodiment, two pre-trained language models can be used as encoders to encode the word sequences of the query text and the matching text respectively to obtain their vector representations, which are denoted as q and p, where and d is the dimension of the vector. The pre-trained language model can be any Transformer-based model, such as BERT. Generally speaking, the vector representation of the query text and the matching text can use the output of the first position of the last layer of the encoder, or the average or maximum value of the output of all positions of the last layer of the encoder. Then, the similarity between the query text and the matching text is calculated using indicators such as dot product or cosine similarity, denoted as s, which is the above score.

[0064] Figure 5 It is a block diagram of a text data processing device provided by an embodiment of the present disclosure.

[0065] like Figure 5 As shown, the text data processing device 500 includes an acquisition module 510, a vector representation module 520, a self-attention processing module 530, a loss determination module 540 and a parameter updating module 550.

[0066] Specifically, the acquisition module 510 is used to acquire training sample data and input the training sample data into the recall model. The training sample data includes training query text and training matching text.

[0067] The vector representation module 520 is used to perform vector representation processing on the training query text and the training matching text respectively to obtain a feature vector sequence of the training query text and a feature vector sequence of the training matching text.

[0068] The self-attention processing module 530 is used to perform self-attention processing on each feature vector in the feature vector sequence of the training query text and each feature vector in the feature vector sequence of the training matching text to obtain an interaction matrix between the training query text and the training matching text.

[0069] The loss determination module 540 is used to determine the maximum edge loss according to the interaction matrix between the training query text and the training matching text.

[0070] The parameter updating module 550 is used to update the parameters in the recall model based on the maximized marginal loss.

[0071] The text data processing device 500 can effectively capture the interactive information of each feature vector between the query and the text by performing vector representation processing on the training query text and the training matching text and applying the self-attention mechanism. This helps the model to more comprehensively understand the semantic and grammatical relationship between the query and the text, helps to improve the accuracy and efficiency of text retrieval, and thus improves the user experience. Updating the model parameters by maximizing the edge loss can effectively reduce the risk of overfitting of the model, and help improve the stability and reliability of the model in real scenarios. Training the model in this way can learn richer semantic information, thereby improving the generalization ability of the model. The model can better adapt to different contexts and semantic relationships when processing new query texts and matching texts.

[0072] In some embodiments of the present disclosure, the vector representation module 520 is configured to: encode each word in the training query text through a first encoder to obtain a feature vector sequence of the training query text; encode each word in the training matching text through a second encoder to obtain a feature vector sequence of the training matching text.

[0073] In some embodiments of the present disclosure, the self-attention processing module 530 is configured to: calculate the correlation between the feature vector sequence of the training query text and the corresponding position of each feature vector in the feature vector sequence of the training matching text based on the self-attention algorithm; determine the interaction matrix between the training query text and the training matching text according to the correlation between the feature vector sequence of the training query text and the corresponding position of each feature vector in the feature vector sequence of the training matching text.

[0074] In some embodiments of the present disclosure, the training matching text includes positive training matching text and negative training matching text; the loss determination module 540 is configured to: calculate a first average value of the interaction matrix between the training query text and the positive training matching text; calculate a second average value of the interaction matrix between the training query text and the negative training matching text; and determine the maximum marginal loss based on the first average value, the second average value, and the hyperparameters of the recall model.

[0075] In some embodiments of the present disclosure, the text data processing device 500 is also used to: generate a pseudo query text based on a feature vector sequence of a training matching text; perform vector representation processing on the pseudo query text to obtain a feature vector sequence of the pseudo query text; calculate a first similarity between the feature vector sequence of the training query text and the feature vector sequence of the training matching text; calculate a second similarity between the feature vector sequence of the pseudo query text and the feature vector sequence of the training matching text; and calculate the minimized cross entropy loss based on the first similarity and the second similarity.

[0076] In some embodiments of the present disclosure, the text data processing apparatus 500 is further used to: determine the target loss according to maximizing the edge loss and minimizing the cross entropy loss; and update the parameters in the recall model according to the target loss.

[0077] In some embodiments of the present disclosure, the text data processing device 500 is also used to obtain the query text currently input by the terminal device; input the query text into the recall model, identify and process the query text through the recall model, and output scores of multiple matching texts; determine the target text corresponding to the query text according to the scores of the multiple matching texts, and send the target text to the terminal device.

[0078] Figure 6 Schematic diagram of an electronic device 6 provided in an embodiment of the present disclosure. Figure 6 As shown, the electronic device 6 of this embodiment includes: a processor 601, a memory 602, and a computer program 603 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the computer program 603, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 601 executes the computer program 603, the functions of the modules in the above-mentioned device embodiments are implemented.

[0079] The electronic device 6 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 6 may include, but is not limited to, a processor 601 and a memory 602. Those skilled in the art will appreciate that Figure 6 The electronic device 6 is merely an example and does not limit the electronic device 6 . The electronic device 6 may include more or less components than those shown in the figure, or different components.

[0080] The processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0081] The memory 602 may be an internal storage unit of the electronic device 6, for example, a hard disk or memory of the electronic device 6. The memory 602 may also be an external storage device of the electronic device 6, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 6. The memory 602 may also include both an internal storage unit of the electronic device 6 and an external storage device. The memory 602 is used to store computer programs and other programs and data required by the electronic device.

[0082] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0083] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0084] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. A text data processing method, characterized in that: include: Acquire training sample data, and input the training sample data into a recall model, wherein the training sample data includes a training query text and a training matching text; Performing vector representation processing on the training query text and the training matching text respectively to obtain a feature vector sequence of the training query text and a feature vector sequence of the training matching text; Performing self-attention processing on each feature vector in the feature vector sequence of the training query text and each feature vector in the feature vector sequence of the training matching text to obtain an interaction matrix between the training query text and the training matching text; Determining a maximum edge loss according to an interaction matrix between the training query text and the training matching text; Parameters in the recall model are updated based on the maximized margin loss.

2. The method according to claim 1, characterized in that The performing vector representation processing on the training query text and the training matching text respectively to obtain a feature vector sequence of the training query text and a feature vector sequence of the training matching text comprises: Encoding each word in the training query text by a first encoder to obtain a feature vector sequence of the training query text; Each word in the training matching text is encoded by a second encoder to obtain a feature vector sequence of the training matching text.

3. The method according to claim 1, characterized in that The performing self-attention processing on each feature vector in the feature vector sequence of the training query text and each feature vector in the feature vector sequence of the training matching text to obtain an interaction matrix between the training query text and the training matching text comprises: Calculate the correlation between the feature vector sequence of the training query text and the corresponding position of each feature vector in the feature vector sequence of the training matching text based on the self-attention algorithm; An interaction matrix between the training query text and the training matching text is determined according to the correlation between the feature vector sequence of the training query text and the corresponding position of each feature vector in the feature vector sequence of the training matching text.

4. The method according to claim 1, characterized in that The training matching text includes positive training matching text and negative training matching text; Determining the maximum edge loss according to the interaction matrix between the training query text and the training matching text includes: Calculating a first average value of an interaction matrix between the training query text and the positive training matching text; Calculating a second average value of the interaction matrix between the training query text and the negative training matching text; The maximized margin loss is determined according to the first average value, the second average value, and a hyperparameter of the recall model.

5. The method according to claim 4, characterized in that The method further comprises: Generate a pseudo query text based on the feature vector sequence of the training matching text; Performing vector representation processing on the pseudo query text to obtain a feature vector sequence of the pseudo query text; Calculating a first similarity between a feature vector sequence of the training query text and a feature vector sequence of the training matching text; Calculating a second similarity between the feature vector sequence of the pseudo query text and the feature vector sequence of the training matching text; According to the first similarity and the second similarity, a cross entropy loss is minimized.

6. The method according to claim 5, characterized in that The method further comprises: Determining a target loss according to the maximized edge loss and the minimized cross entropy loss; Parameters in the recall model are updated according to the target loss.

7. The method according to claim 1, characterized in that The method further comprises: Get the query text currently input by the terminal device; Inputting the query text into the recall model, performing recognition processing on the query text through the recall model, and outputting scores of multiple matching texts; A target text corresponding to the query text is determined according to the scores of the plurality of matching texts, and the target text is sent to the terminal device.

8. A text data processing device, characterized in that: include: An acquisition module, used for acquiring training sample data and inputting the training sample data into a recall model, wherein the training sample data includes a training query text and a training matching text; A vector representation module, used to perform vector representation processing on the training query text and the training matching text respectively, to obtain a feature vector sequence of the training query text and a feature vector sequence of the training matching text; A self-attention processing module, used for performing self-attention processing on each feature vector in the feature vector sequence of the training query text and each feature vector in the feature vector sequence of the training matching text, to obtain an interaction matrix between the training query text and the training matching text; A loss determination module, used for determining the maximum edge loss according to the interaction matrix between the training query text and the training matching text; A parameter updating module is used to update the parameters in the recall model based on the maximized marginal loss.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images