Text retrieval matching method and device, storage medium and computer program product
By obtaining the subject words of the search request text and using the target language model for feature extraction, combining data augmentation technology and adversarial training, the performance problem of manual labeled data training model in large-scale search matching scenarios is solved, and the search speed and accuracy are improved .
Patent Information
- Application Number
- CN202410925785.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-12
AI Technical Summary
In the prior art, models trained with labeled data using manual annotation have poor performance in search matching scenarios with large text volumes, and feature representations are difficult to migrate to other tasks, resulting in slow search rates and low validity of matching results.
By obtaining the subject words of the search request text, using the preset target language model to extract features of it and the alternative text, combining data augmentation technology to generate positive and negative samples of self-supervised learning, combat training models, and improve the generalization ability of the model. And filter the target text by vector similarity calculation.
Without a large number of manual annotations, the accuracy and efficiency of text retrieval matching are improved, and the performance problem of the model in large-scale retrieval matching scenarios is solved.
Smart Images

Figure CN118939754A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a text retrieval and matching method, device, storage medium and computer program product. Background Art
[0002] With the rapid development of artificial intelligence technology in recent years, especially the continuous updating of natural language processing, text retrieval and matching has become a core task in natural language processing. Many natural language processing tasks can be abstracted into text matching problems. Whether in dialogue systems, recommendation systems, or search engines, text retrieval and matching are indispensable.
[0003] At present, relevant technicians usually apply deep learning methods to text retrieval and matching tasks, among which the most common method is supervised learning. However, this method requires a large amount of labeled data that can be used for training, and the labeled data is obtained through manual annotation, so it requires a lot of manual annotation costs. In addition, the model obtained through supervised learning can often only learn some specific knowledge, but cannot learn general knowledge, which makes it difficult to transfer the feature representation learned to other tasks, resulting in a slow retrieval rate and low effectiveness of matching results.
[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0005] The embodiments of the present application provide a text retrieval and matching method, device, storage medium and computer program product to at least solve the technical problem that the model trained with manually annotated labeled data has poor performance in retrieval and matching scenarios with large amounts of text.
[0006] According to one aspect of an embodiment of the present application, a text retrieval and matching method is provided, comprising: obtaining a first search request text and extracting a subject word of the first search request text; obtaining at least one candidate text with the same subject word from multiple texts in a preset database; performing feature extraction on the first search request text and each candidate text using a preset target language model to obtain a first embedded representation vector corresponding to the first search request text and a second embedded representation vector corresponding to each candidate text, wherein the target language model is obtained by iterative adversarial training using multiple groups of target sample data after data enhancement, and each group of target sample data includes at least one positive sample and at least one negative sample; determining the vector similarity between the first embedded representation vector and each second embedded representation vector, and determining a target text whose vector similarity is not less than a preset vector similarity threshold from at least one candidate text.
[0007] Optionally, extracting the subject words of the first search request text includes: segmenting the first search request text to obtain multiple first participles; obtaining the part of speech of each first participle, and determining at least one key participle from the multiple first participles based on the part of speech; judging whether there are repeated key participles in at least one key participle; if so, determining to deduplicate the at least one key participle to obtain the subject words; if not, determining at least one key participle to be the subject word.
[0008] Optionally, the training process of the target language model includes: constructing an initial language model; obtaining multiple sets of initial sample data and preprocessing the multiple sets of initial sample data, wherein the initial sample data includes: a second search request text and a corresponding third embedded representation vector; using a text data enhancement method to process the preprocessed multiple sets of initial sample data to obtain multiple sets of target sample data; using the multiple sets of target sample data to iteratively adversarially train the initial language model to obtain a target language model.
[0009] Optionally, a text data enhancement method is used to process multiple groups of preprocessed initial sample data to obtain multiple groups of target sample data, including: for each group of preprocessed initial sample data, the second search request text in the initial sample data is divided into multiple second word segments, and at least one second word segment is shuffled, and the third search request text is composed of the shuffled second word segments, and the fourth embedded representation vector corresponding to each third search request text is determined; each second search request text and its corresponding third embedded representation vector, each third search request text and its corresponding fourth embedded representation vector are composed of positive samples; and each second search request text and the fifth embedded representation vector, each third search request text and the sixth embedded representation vector are composed of negative samples, wherein the fifth embedded representation vector is an embedded representation vector other than the third embedded representation vector corresponding to the second search request text, and the sixth embedded representation vector is an embedded representation vector other than the fourth embedded representation vector corresponding to the third search request text; and multiple groups of target sample data are composed of at least one positive sample and at least one negative sample.
[0010] Optionally, the preprocessed multiple groups of initial sample data are processed using a text data enhancement method to obtain multiple groups of target sample data, and also include: for each group of initial sample data after preprocessing, the second search request text and / or the third search request text in the initial sample data are segmented and deleted according to a preset probability to obtain a fourth search request text and / or a fifth search request text, and the seventh embedded representation vector corresponding to each fourth search request text and / or the eighth embedded representation vector corresponding to each fifth search request text are determined; each fourth search request text and its corresponding seventh embedded representation vector and / or each fifth search request text and its corresponding eighth embedded representation vector constitute a positive sample, and each fourth search request text and the ninth embedded representation vector and / or each fifth search request text and the tenth embedded representation vector constitute a positive sample, wherein the ninth embedded representation vector is an embedded representation vector other than the seventh embedded representation sub-vector corresponding to the fourth search request text, and the tenth embedded representation vector is an embedded representation vector other than the eighth embedded representation sub-vector corresponding to the fifth search request text; and the multiple groups of target sample data are composed of at least one positive sample and at least one negative sample.
[0011] Optionally, the target sample data includes a target search request text and a target embedded representation vector, wherein the initial language model is iteratively adversarially trained using multiple groups of target sample data to obtain a target language model, including: for each training batch in the iterative adversarial training process, the target search request text in each group of target sample data in the training batch is input into the initial language model to obtain a predicted embedded representation vector output by the initial language model; a target loss function is constructed using the target embedded representation vector and the predicted embedded representation vector of each group of target text data; and the model parameters of the initial language model are adjusted according to the target loss function.
[0012] Optionally, adjusting the model parameters of the initial language model according to the target loss function includes: for each training batch in the iteration process, based on multiple groups of target sample data of the current training batch, using a back propagation algorithm to determine the first gradient of the target loss function on the model parameters of the current iteration round; using a preset perturbation parameter, an adversarial perturbation value of the previous iteration round and the first gradient, calculating the current adversarial perturbation value of the current iteration round, wherein the adversarial perturbation value of the initial iteration round is zero; adding the current adversarial perturbation value to the model parameters of the initial language model corresponding to the target sample data, and updating the target loss function; using multiple groups of target sample data of the current training batch, calculating the second gradient of the updated target loss function on the model parameters; determining the adversarial gradient based on the first gradient and the second gradient, and processing the adversarial gradient using a preset gradient descent algorithm to adjust the model parameters of the initial language model, wherein the gradient descent algorithm includes at least one of the following: an adaptive learning rate gradient descent algorithm and a momentum gradient descent algorithm.
[0013] According to another aspect of an embodiment of the present application, a text retrieval and matching device is also provided, including: a first acquisition module, used to acquire a first search request text and extract a subject word of the first search request text; a second acquisition module, used to acquire at least one alternative text with the same subject word from multiple texts in a preset database; a feature extraction module, used to perform feature extraction on the first search request text and each alternative text using a preset target language model, and obtain a first embedded representation vector corresponding to the first search request text and a second embedded representation vector corresponding to each alternative text, wherein the target language model is obtained by iterative adversarial training using multiple groups of target sample data after data enhancement, and each group of target sample data includes at least one positive sample and at least one negative sample; a determination module, used to determine the vector similarity between the first embedded representation vector and each second embedded representation vector, and determine a target text whose vector similarity is not less than a preset vector similarity threshold from at least one alternative text.
[0014] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned text retrieval and matching method by running the computer program.
[0015] According to another aspect of an embodiment of the present application, a computer program product is further provided. The computer program product includes a stored computer program, wherein the computer program implements the above-mentioned text retrieval and matching method when executed by a processor.
[0016] In the embodiment of the present application, the first search request text is obtained and its subject words are extracted, and at least one candidate text with the same subject words is preliminarily screened out from multiple texts in the preset knowledge database; the target language model is used to extract features of the first search request text and each candidate text, and the first embedded representation vector corresponding to the first search request text and the second embedded representation vector corresponding to each candidate text are obtained; finally, the similarity of the first embedded representation vector and each second embedded representation vector is analyzed to accurately screen out the target text corresponding to the first search request text from at least one candidate text. Among them, the data enhancement technology is used to generate positive and negative sample pairs for self-supervised learning, and based on the generated positive and negative sample pairs, adversarial training is added in the deep learning training stage to increase the perturbation consistent with the gradient direction change to improve the generalization ability of the final model. Finally, the target text matching the first search request text is retrieved by the vector similarity calculation method, so as to achieve the purpose of improving the retrieval speed, so that in the absence of a large amount of manually annotated labeled data, the accuracy and efficiency of text retrieval matching can be effectively improved, thereby solving the technical problem that the model trained by manually annotated labeled data has poor performance in the retrieval matching scenario with a large amount of text. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0018] Figure 1 It is a hardware structure block diagram of a computer terminal for implementing a text retrieval and matching method according to an embodiment of the present application;
[0019] Figure 2 is a flowchart of an optional text retrieval and matching method according to an embodiment of the present application;
[0020] Figure 3 is a schematic structural diagram of an optional text retrieval and matching device according to an embodiment of the present application;
[0021] Figure 4 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0024] In addition, the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.
[0025] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:
[0026] Similarity vector search (ScaNN), also known as nearest neighbor search, is most commonly used by using the maximum inner product search (MIPS) of the inner product distance, where MIPS can be defined as follows: Consider a database X = x with n data points i ,i=1,2,…,n. Each data point is in a dimension In the vector space of The task is to determine the data point x∈X that has the largest inner product with q, that is:
[0027] Adversarial training is a training method that introduces noise and regularizes parameters to improve model robustness and generalization ability.
[0028] Back Propagation (BP) is short for "error back propagation". It is a common method for training artificial neural networks in combination with optimization algorithms (such as gradient optimization descent). This method can calculate the gradient of the loss function for all weights in the network. This gradient will be fed back to the optimization method to update the weights to minimize the loss function.
[0029] The gradient indicates that the directional derivative of a function at this point reaches its maximum value along this direction, that is, the function changes fastest along the direction (the direction of this gradient) at this point and the rate of change is the largest.
[0030] Example 1
[0031] According to an embodiment of the present application, a method embodiment of a text retrieval and matching method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0032] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a text search and matching method. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.
[0033] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
[0034] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the text retrieval matching method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the text retrieval matching method of the above-mentioned application. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0035] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0036] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).
[0037] In the above operating environment, Figure 2 is a flow chart of an optional text search and matching method according to an embodiment of the present application, such as Figure 2 As shown, the method at least includes steps S202-S208, wherein:
[0038] Step S202: Obtain a first search request text, and extract a subject word of the first search request text.
[0039] In the technical solution provided in step S202, when the user's search instruction is executed and the request question currently input in the search platform is extracted, the user's first search request text can be obtained. In order to preliminarily screen out a set of alternative texts related to the first search request from the many types of texts stored in the knowledge database to narrow the text search scope, the present application can also perform subject word extraction on the first search request text to obtain corresponding keywords.
[0040] Step S204: obtaining at least one candidate text having the same subject word from a plurality of texts in a preset database.
[0041] In the technical solution provided in step S204, the topics of multiple text tags stored in the associated knowledge database in the retrieval platform are compared with the subject words extracted from the first search request text, and at least one text that meets the subject words of the first search request text is selected as an alternative text.
[0042] For example, the first search request text entered by the user in the search platform is "search for the network structure of a convolutional neural network". By performing keyword extraction on the first search request text, the corresponding keywords obtained are "convolutional neural network" and "network structure". Therefore, multiple texts related to the keywords "convolutional neural network" and "network structure" can be screened out from the knowledge database and used as alternative texts.
[0043] Step S206, using a preset target language model to perform feature extraction on the first search request text and each candidate text, to obtain a first embedded representation vector corresponding to the first search request text and a second embedded representation vector corresponding to each candidate text.
[0044] In the technical solution provided in step S206, the first search request text and the above-screened backup texts are used as inputs of the target language model, and the first embedded representation vector corresponding to the first search request text and the second embedded representation vector corresponding to the alternative texts output by the target language model are obtained. Among them, considering that the training corpus of the relevant language model requires a large amount of manually annotated label training sample data, it requires high labor costs. To this end, an embodiment of the present application proposes that iterative adversarial training can be performed using multiple groups of target sample data after data enhancement to obtain a target language model, wherein each group of target sample data includes at least one positive sample and at least one negative sample.
[0045] Step S208, determining the vector similarity between the first embedded representation vector and each second embedded representation vector, and determining a target text whose vector similarity is not less than a preset vector similarity threshold from at least one candidate text.
[0046] In the technical solution provided in step S208, a vector similarity calculation formula is used to determine the vector similarity between the first embedded representation vector output by the target language model and each second embedded representation vector, and based on the first similarity, a target text whose similarity to the first search request text is not less than a preset vector similarity threshold is determined from the at least one candidate text. The vector similarity threshold can be set according to the actual application scenario, and the present application does not impose any specific restrictions on the setting of the threshold.
[0047] The above method of this embodiment is further introduced below.
[0048] As an optional implementation, in the technical solution provided in the above step S202, the process of refining the subject words may include:
[0049] First, the first search request text is segmented to obtain a plurality of first segmented words;
[0050] Next, the part of speech of each first participle is obtained, and at least one key participle is determined from the plurality of first participles according to the part of speech;
[0051] Then, determining whether there is a repeated key participle in at least one key participle;
[0052] If so, determine to perform a deduplication operation on at least one key word to obtain the subject word;
[0053] If not, determine at least one key word as the subject word.
[0054] In the above embodiment, the first search request text is segmented, and stop words are removed from each segmented word to obtain multiple first segmentations (also called valid segmentations). The above key segmentations refer to the analysis that can express the search request subject. Generally speaking, the common parts of speech used to express the subject are nouns, verbs, proper nouns, etc. By analyzing the part of speech of each segmentation word and matching it with the common part of speech used to express the subject, the key segmentations can be screened out from each first segmentation word. In addition, in order to avoid duplication of the key segmentations finally extracted, each key segmentation word can also be compared to identify whether there are repeated key segmentations. If so, the repeated key segmentations are deduplicated to finally extract the subject words of the first search request text.
[0055] As an optional implementation, the training process of the target language model includes the following steps S1-S4:
[0056] in:
[0057] Step S1, constructing an initial language model.
[0058] Specifically, the above initial language model can be any pre-trained model in Hugging Face, such as BERT, RoBERTa, GPT, T5, etc. Hugging Face is an open source machine learning model library that focuses on natural language processing and is used to provide a large number of pre-trained models. These models are open source and used in various NLP tasks, such as text analysis, sentiment analysis, question-answering systems, machine translation, summary generation, etc. Hugging Face's model library includes but is not limited to the following: Transformers, Datasets, Tokenizers, Trainer, Pipelines.
[0059] Step S2, obtaining multiple groups of initial sample data, and preprocessing the multiple groups of initial sample data.
[0060] The above step S2 can be understood as pre-acquiring initial sample data with a small amount of labels that have been manually annotated, wherein these initial sample data include: a second search request text and a corresponding third embedded representation vector. However, since the second search request text in the initial sample data may contain special characters, spaces, emoticons, and other messy characters, and these messy characters may affect the accuracy of subsequent model training, in the embodiment of the present application, regular expressions can be used to pre-process the second search request text to remove these messy characters.
[0061] Step S3, using a text data enhancement method to process the preprocessed multiple groups of initial sample data to obtain multiple groups of target sample data.
[0062] If a small amount of sample data is used to train the model, the model will over-learn the characteristics of these sample data, resulting in the final model performing well on the training sample data, but performing poorly on new, unseen data, causing the model to overfit and generalize poorly, and the trained model to have poor stability. Therefore, in this case, the present application proposes that the initial sample data can be processed using a text data enhancement method to obtain multiple sets of target sample data with data enhancement.
[0063] Step S4, using multiple groups of target sample data to perform iterative adversarial training on the initial language model to obtain a target language model.
[0064] The above step S4 can be understood as inputting multiple groups of target sample data into the initial language model in sequence for iterative training. In order to increase the generalization ability and stability of the model, the embodiment of the present application also adopts free adversarial training in the model training truncation to perform multiple iterative adversarial training to introduce disturbance (i.e., adversarial noise).
[0065] Optionally, the above-mentioned text data enhancement method includes but is not limited to: shuffle, cutoff, etc. Among them, shuffle refers to disrupting the data set of the training model so that the appearance of any sample data satisfies "randomness". Therefore, shuffle can effectively prevent model jitter during training and prevent model overfitting, so that the model learns more correct features. The cutoff probability refers to deleting the corresponding word segmentation in the text data according to the preset word segmentation cutoff probability.
[0066] Therefore, in the technical solution provided in the above step S3, the embodiment of the present application proposes that multiple groups of target sample data after data enhancement can be obtained by word segmentation shuffle, including the following steps:
[0067] Step 1: For each group of initial sample data after preprocessing, the second search request text in the initial sample data is divided into multiple second participles, and at least one second participle is shuffled, and the third search request text is composed of the shuffled second participles, and the fourth embedded representation vector corresponding to each third search request text is determined.
[0068] Step 2: each second search request text and its corresponding third embedded representation vector, each third search request text and its corresponding fourth embedded representation vector form a positive sample; and each second search request text and the fifth embedded representation vector (i.e., the second search request text does not correspond to the fifth embedded representation vector), each third search request text and the sixth embedded representation vector (i.e., the third search request text does not correspond to the sixth embedded representation vector) form a negative sample, wherein the fifth embedded representation vector is an embedded representation vector other than the third embedded representation vector corresponding to the second search request text, and the sixth embedded representation vector is an embedded representation vector other than the fourth embedded representation vector corresponding to the third search request text;
[0069] Step 3: Form multiple groups of target sample data from at least one positive sample and at least one negative sample.
[0070] In addition, in addition to performing data enhancement on text data through the above method, the embodiment of the present application also proposes that multiple groups of target sample data after data enhancement can be obtained by setting the word segmentation cutoff probability, which specifically includes the following steps:
[0071] Step 1: for each group of initial sample data after preprocessing, perform word segmentation and deletion on the second search request text and / or the third search request text in the initial sample data according to a preset probability to obtain a fourth search request text and / or a fifth search request text, and determine the seventh embedded representation vector corresponding to each fourth search request text and / or the eighth embedded representation vector corresponding to each fifth search request text;
[0072] Step 2: each fourth search request text and its corresponding seventh embedded representation vector and / or each fifth search request text and its corresponding eighth embedded representation vector constitute a positive sample, and each fourth search request text and its ninth embedded representation vector and / or each fifth search request text and its tenth embedded representation vector constitute a positive sample, wherein the ninth embedded representation vector is an embedded representation vector other than the seventh embedded representation sub-vector corresponding to the fourth search request text, and the tenth embedded representation vector is an embedded representation vector other than the eighth embedded representation sub-vector corresponding to the fifth search request text;
[0073] Step 3: Form multiple groups of target sample data from at least one positive sample and at least one negative sample.
[0074] It should be noted that in the embodiment of the present application, the above-mentioned cutoff probability, word segmentation shuffle method or a combination of the two can be used to perform data enhancement on each initial sample data in the initial sample set to form multiple groups of positive and negative samples under self-supervised learning. Among them, in the embodiment of the present application, the cutoff probability is preferably set to 0.15.
[0075] In the technical solution provided in the above step S4, the method may include the following steps:
[0076] Step S41, for each training batch in the iterative adversarial training process, the target search request text in each group of target sample data in the training batch is input into the initial language model to obtain the predicted embedded representation vector output by the initial language model.
[0077] Step S42, constructing a target loss function using the target embedded representation vector and the predicted embedded representation vector of each group of target text data.
[0078] Specifically, the above objective loss function can be written as:
[0079]
[0080] Where N represents the number of target samples in the training batch, i represents the sequence number of the target sample data, and P(pos i |que i ) indicates that in the second search request text quei In the case of occurrence, the third embedded representation vector pos i The probability of occurrence (i.e. conditional probability), and when i=j, P(pos i |que i ) represents the probability of a positive sample. When i≠j, P(pos i |que i ) represents the probability of negative samples.
[0081] In addition, the above objective loss function can also be written as:
[0082]
[0083] Among them, S(que i ,pos i ) represents the predicted embedded representation vector output by the i-th target training sample data input into the initial language model.
[0084] Step S43, adjusting the model parameters of the initial language model according to the target loss function.
[0085] Optionally, in the technical solution provided in the above step S43, the method can be implemented by the following steps:
[0086] Step S431, for each training batch in the iteration process, based on multiple groups of target sample data of the current training batch, a back propagation algorithm is used to determine the first gradient of the target loss function on the model parameters of the current iteration round;
[0087] Step S432, using the preset disturbance parameter, the counter-disturbance value of the previous iteration round and the first gradient, calculating the current counter-disturbance value of the current iteration round, wherein the counter-disturbance value of the initial iteration round is zero;
[0088] Step S433, adding the current adversarial disturbance value to the model parameters of the initial language model corresponding to the target sample data, and updating the target loss function;
[0089] Step S434, using multiple groups of target sample data of the current training batch, calculating the second gradient of the updated target loss function on the model parameters;
[0090] Step S435, determining an adversarial gradient based on the first gradient and the second gradient, and processing the adversarial gradient using a preset gradient descent algorithm to adjust model parameters of the initial language model, wherein the gradient descent algorithm includes at least one of the following: an adaptive learning rate gradient descent algorithm, a momentum gradient descent algorithm.
[0091] In the technical solution provided by the above steps S431-S435, the above iterative adversarial training to adjust the model parameters of the initial language model can be specifically refined into the following steps:
[0092] Step 1: Calculate the loss function value obtained by inputting the target sample data into the initial language model through the forward propagation method, then use the back propagation method to calculate the gradient value of the target loss function on each model parameter of the initial language model, and back up the gradient value.
[0093] Step 2: Set the number of steps of adversarial perturbation accumulation to m, and then execute an m-step for loop for the target sample data in each training batch, including:
[0094] Step 2.1: For the tth step (t≤m-1), the current adversarial perturbation value of the current iteration round can be calculated based on the first gradient value of the objective loss function on the model parameters in the previous iteration round, the preset perturbation parameters and the adversarial perturbation value of the previous iteration round according to the following formula:
[0095] r t+1 =r t +ε*sign(g)
[0096] Among them, r t+1 represents the adversarial perturbation value at step t+1, r t represents the adversarial perturbation value of the tth step, the adversarial perturbation value in the initial iteration round is 0, ε represents the perturbation parameter, g represents the first gradient value, and sign(g) represents the result of taking the sign function of the first gradient value. Therefore, while using sign(g) to ensure that the perturbation direction is consistent with the original first gradient direction, its current adversarial perturbation value "attacks" the initial language model (that is, the current adversarial perturbation obtained is added to the current model parameters of the initial language model), and the initial language model after the "attack" is obtained.
[0097] Step 2.2: For the step t≠m-1, use the multiple sets of target sample data of the current training batch to calculate the second gradient of the updated target loss function on the model parameters; and for the step t=m-1, restore to the previously backed-up gradient value.
[0098] Step 2.3: Superimpose the second gradient obtained in the above t steps with the first gradient to obtain the adversarial gradient after multiple iterative adversarial attacks.
[0099] Step 3: Use the preset gradient descent algorithm to process the adversarial gradient calculated after accumulating m times of adversarial perturbations to adjust the model parameters of the initial language model.
[0100] Among them, the above-mentioned gradient descent algorithm includes but is not limited to: adaptive learning rate gradient descent algorithm, momentum gradient descent algorithm, etc.
[0101] Specifically, the embodiment of the present application takes the projection-based momentum gradient descent algorithm as an example to illustrate the technical solution provided by the above step S435.
[0102] According to the current adversarial gradient g t Calculate the first-order moving average momentum, the expression can be written as:
[0103] m t =β1*m t-1 +(1-β1)*g t
[0104] Among them, m t represents the exponential moving average of the t-step gradient (m0 is initially 0), β1 represents the exponential decay rate of the first-order momentum (its value is between 0 and 1, usually approximately 1), m t-1 represents the exponential moving average of the t-1 step gradient, t represents the time step, g t represents the gradient of step t.
[0105] According to the current adversarial gradient g t Calculate the second-order moving average momentum, the expression can be written as:
[0106]
[0107] Among them, V t represents the exponential moving average of the square of the t-step gradient (V0 is initially 0), β2 represents the exponential decay rate of the second-order momentum (its value is between 0 and 1, usually approximately 1), and v t- 1 is the exponential moving average of the gradient in step t-1.
[0108] Then, according to the first-order moving average momentum m t and the second-order moving average momentum V t Calculate the model parameter correction value P t , the formula is as follows:
[0109]
[0110] Here, ε is a weighted adjustment parameter between 0 and 1.
[0111] Finally, the model parameters θ of the t-step initial language model are t (θ0 is initially 0), the parameter θ for step t+1 t+1 Make adjustments, including:
[0112] When θ t *g t When it is less than the adjustment threshold parameter δ, the adjustment formula can be written as:
[0113] θt+1 =θ t -η∏P t
[0114] Among them, ∏P t Indicates P t The projection in the direction of the current parameters of the target loss function, η represents the learning rate.
[0115] When θ t *g t When it is not less than the adjustment threshold parameter δ, the adjustment formula can be written as:
[0116] θ t+1 =θ t -ηP t
[0117] According to the above-mentioned gradient descent method combining first-order momentum and second-order momentum, the optimal parameter model of the initial language model can be quickly obtained.
[0118] As an optional implementation, in the technical solution provided in the above step S208, a tree index can be first generated by a second embedded representation vector corresponding to at least one alternative text, and a second target embedded representation vector matching the first embedded representation vector can be determined in the tree index, and the text corresponding to the second target embedded representation vector can be determined as the target text matching the first search request text.
[0119] Specifically, a preset number of trees are generated using the second embedded representation vectors corresponding to each alternative text, and a tree index of the generated tree is given, wherein, if the number of trees in the tree index is smaller, the retrieval speed is faster and the time consumed is shorter, but the accuracy of the retrieval results will also be lower; if the number of trees in the tree index is larger, the retrieval results are more accurate, but the corresponding retrieval time is longer. Therefore, the embodiment of the present application can define the number of trees in the tree index according to the number of retrieved data. For example, when generating a tree index, multiple second embedded representation vectors can be regarded as multiple points in a plane. If the preset number is 2, the plane can be divided into two planes by a line. The two planes after division each correspond to a tree, that is, the number of trees in the tree index is the same as the number of planes obtained by division.
[0120] Furthermore, a matching path is determined in the tree index; the vector similarity (cosine value) between each second embedded representation vector and the first embedded representation vector in multiple second embedded representation vectors on the matching path is determined to obtain multiple vector similarities; a second target embedded representation vector corresponding to a preset vector similarity threshold value among the multiple vector similarities is determined, and the text corresponding to the second target embedded representation vector is determined as the target text matching the first search request text.
[0121] That is to say, after the entire plane where the multiple second embedded representation vectors are located is divided into two sub-planes, when determining the path, it is necessary to determine which sub-plane the first embedded representation vector to be matched falls on. If it falls on the first plane, the matching path is the path that matches the first embedded representation vector to be matched with all the second embedded representation vectors in the sub-plane in sequence; if the entire plane where the multiple second embedded representation vectors are located is divided into multiple sub-planes, then after initially determining which sub-plane the first embedded representation vector to be matched falls on, the previous plane in the plane can be determined in sequence according to the tree index, that is, the plane is a node of the tree index, and each root node of the node of the plane is searched one by one in the direction of the root node in the tree index, and the path composed of these nodes is the matching path. After determining the matching path, determine the vector similarity (cosine value) between each second embedded representation vector and the first embedded representation vector in the multiple second embedded representation vectors on the matching path, and obtain multiple vector similarities; determine the second target embedded representation vector corresponding to the multiple vector similarities that is not less than the preset vector similarity threshold, and determine the text corresponding to the second target embedded representation vector as the target text matching the first search request text.
[0122] It should be noted that the embodiments of the present application can also determine the validity of the matching result through a pre-set similarity threshold, wherein the closer the vector similarity value between the second embedded representation vector and the first embedded representation vector is to the similarity threshold, the higher the accuracy of the matching result corresponding to the second embedded representation vector.
[0123] Through the above steps, the embodiment of the present application uses data enhancement technology to generate positive and negative sample pairs for self-supervised learning. Based on these generated positive and negative sample pairs, multiple iterative adversarial training is performed in the training stage to increase disturbances and improve the generalization ability of the final model. In addition, based on the idea of knowledge transfer, the gradient is optimized using a gradient optimization algorithm (such as a momentum gradient algorithm) to adjust the model parameters, so that the final trained model has good performance. The following control experiment will illustrate the superiority of the text retrieval matching method provided in the embodiment of the present application:
[0124] The self-supervised ConSert method was compared with the text retrieval and matching method provided in the embodiment of the present application using the authoritative Chinese public data sets ocnli (native Chinese natural language matching and inference data set) and mnli (multi-type natural language matching and inference data set of the Chinese language understanding evaluation benchmark CLUE benchmark). The original classification matching task was changed into a retrieval matching and recall task, that is, the first text set of the matching data in the verification set data was used as the retrieval text set, and the second text set was used as the retrieved text set, and the matching accuracy (recall rate) comparison results shown in Table 1 were obtained.
[0125] Table 1
[0126]
[0127]
[0128] It can be seen from Table 1 above that the text retrieval and matching method provided in the embodiment of the present application has a better improvement in retrieval and matching accuracy (recall rate) than the self-supervised ConSert method.
[0129] Example 2
[0130] Based on Embodiment 1 of the present application, an embodiment of a text search and matching device is also provided, and the text search and matching method of the above embodiment is executed when the device is running. Figure 3 is a schematic diagram of the structure of an optional text retrieval and matching device according to an embodiment of the present application, such as Figure 3 As shown, the text retrieval and matching device at least includes a first acquisition module 31, a second acquisition module 32, a feature extraction module 33 and a determination module 34, wherein:
[0131] A first acquisition module 31 is used to acquire a first search request text and extract a subject word from the first search request text;
[0132] A second acquisition module 32 is used to acquire at least one candidate text that is the same as the subject word from multiple texts in a preset database;
[0133] A feature extraction module 33 is used to extract features from the first search request text and each candidate text using a preset target language model to obtain a first embedded representation vector corresponding to the first search request text and a second embedded representation vector corresponding to each candidate text, wherein the target language model is obtained by iterative adversarial training using multiple groups of target sample data after data enhancement, and each group of target sample data includes at least one positive sample and at least one negative sample;
[0134] The determination module 34 is used to determine the vector similarity between the first embedded representation vector and each second embedded representation vector, and determine a target text whose vector similarity is not less than a preset vector similarity threshold from at least one candidate text.
[0135] It should be noted that the various modules in the above-mentioned text retrieval and matching device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.
[0136] Example 3
[0137] According to an embodiment of the present application, a non-volatile storage medium is further provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the text retrieval and matching method in Example 1.
[0138] Optionally, the device where the non-volatile storage medium is located implements the following steps by running the program: obtaining a first search request text and extracting a subject word of the first search request text; obtaining at least one alternative text with the same subject word from multiple texts in a preset database; using a preset target language model to perform feature extraction on the first search request text and each alternative text to obtain a first embedded representation vector corresponding to the first search request text and a second embedded representation vector corresponding to each alternative text, wherein the target language model is obtained by iterative adversarial training using multiple groups of target sample data after data enhancement, and each group of target sample data includes at least one positive sample and at least one negative sample; determining the vector similarity between the first embedded representation vector and each second embedded representation vector, and determining a target text from at least one alternative text whose vector similarity is not less than a preset vector similarity threshold.
[0139] According to an embodiment of the present application, a computer program product is also provided, which includes a stored computer program, wherein when the computer program is executed by a processor, the text retrieval and matching method in Example 1 is implemented.
[0140] Optionally, the computer program executes the following steps: obtaining a first search request text and extracting a subject word of the first search request text; obtaining at least one alternative text with the same subject word from multiple texts in a preset database; performing feature extraction on the first search request text and each alternative text using a preset target language model to obtain a first embedded representation vector corresponding to the first search request text and a second embedded representation vector corresponding to each alternative text, wherein the target language model is obtained by iterative adversarial training using multiple groups of target sample data after data enhancement, and each group of target sample data includes at least one positive sample and at least one negative sample; determining the vector similarity between the first embedded representation vector and each second embedded representation vector, and determining a target text from at least one alternative text whose vector similarity is not less than a preset vector similarity threshold.
[0141] According to an embodiment of the present application, a processor is also provided, which is used to run a program, wherein the text retrieval and matching method in Example 1 is executed when the program is running.
[0142] Optionally, the following steps are executed when the program is running: obtaining a first search request text and extracting a subject word of the first search request text; obtaining at least one alternative text with the same subject word from multiple texts in a preset database; using a preset target language model to perform feature extraction on the first search request text and each alternative text to obtain a first embedded representation vector corresponding to the first search request text and a second embedded representation vector corresponding to each alternative text, wherein the target language model is obtained by iterative adversarial training using multiple groups of target sample data after data enhancement, and each group of target sample data includes at least one positive sample and at least one negative sample; determining the vector similarity between the first embedded representation vector and each second embedded representation vector, and determining a target text from at least one alternative text whose vector similarity is not less than a preset vector similarity threshold.
[0143] According to an embodiment of the present application, an electronic device is also provided, wherein: Figure 4 is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application, such as Figure 4 As shown, the electronic device includes one or more processors; a memory for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to run the programs, wherein the programs are configured to execute the text retrieval and matching method in the above-mentioned embodiment 1 when running.
[0144] Optionally, the processor is configured to execute the following steps through a computer program: obtaining a first search request text and extracting a subject word of the first search request text; obtaining at least one alternative text with the same subject word from multiple texts in a preset database; using a preset target language model to perform feature extraction on the first search request text and each alternative text to obtain a first embedded representation vector corresponding to the first search request text and a second embedded representation vector corresponding to each alternative text, wherein the target language model is obtained by iterative adversarial training using multiple groups of target sample data after data enhancement, and each group of target sample data includes at least one positive sample and at least one negative sample; determining the vector similarity between the first embedded representation vector and each second embedded representation vector, and determining a target text from at least one alternative text whose vector similarity is not less than a preset vector similarity threshold.
[0145] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0146] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0147] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0148] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0149] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0150] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.
[0151] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A text retrieval and matching method, characterized in that: include: Obtaining a first search request text, and extracting a subject word from the first search request text; Acquire at least one candidate text identical to the subject word from a plurality of texts in a preset database; Using a preset target language model to perform feature extraction on the first search request text and each candidate text, obtaining a first embedded representation vector corresponding to the first search request text and a second embedded representation vector corresponding to each candidate text, wherein the target language model is obtained by iterative adversarial training using multiple groups of target sample data after data enhancement, and each group of the target sample data includes at least one positive sample and at least one negative sample; The vector similarity between the first embedded representation vector and each of the second embedded representation vectors is determined, and a target text whose vector similarity is not less than a preset vector similarity threshold is determined from the at least one candidate text.
2. The method according to claim 1, characterized in that Extracting the subject words of the first search request text includes: Performing word segmentation on the first search request text to obtain a plurality of first word segments; Acquire the part of speech of each of the first participles, and determine at least one key participle from the plurality of the first participles according to the part of speech; Determining whether there are repeated key participles in the at least one key participle; If so, determine to perform a deduplication operation on the at least one key word to obtain the subject word; If not, determine the at least one key word as the subject word.
3. The method according to claim 1, characterized in that The training process of the target language model includes: Build an initial language model; Acquire multiple groups of initial sample data, and preprocess the multiple groups of the initial sample data, wherein the initial sample data includes: a second search request text and a corresponding third embedded representation vector; Using a text data enhancement method to process the preprocessed multiple groups of initial sample data to obtain multiple groups of target sample data; The initial language model is iteratively trained against each other using multiple groups of the target sample data to obtain the target language model.
4. The method according to claim 3, characterized in that The preprocessed multiple groups of initial sample data are processed using a text data enhancement method to obtain multiple groups of target sample data, including: For each group of the preprocessed initial sample data, split the second search request text in the initial sample data into a plurality of second participles, shuffle at least one of the second participles, form a third search request text from the shuffled second participles, and determine a fourth embedded representation vector corresponding to each of the third search request texts; A positive sample is composed of each of the second search request texts and its corresponding third embedded representation vector, and each of the third search request texts and its corresponding fourth embedded representation vector; and a negative sample is composed of each of the second search request texts and the fifth embedded representation vector, and each of the third search request texts and the sixth embedded representation vector, wherein the fifth embedded representation vector is an embedded representation vector other than the third embedded representation vector corresponding to the second search request text, and the sixth embedded representation vector is an embedded representation vector other than the fourth embedded representation vector corresponding to the third search request text; The target sample data consists of at least one positive sample and at least one negative sample.
5. The method according to claim 4, characterized in that The preprocessed multiple groups of initial sample data are processed using a text data enhancement method to obtain multiple groups of target sample data, and further comprising: For each group of the initial sample data after preprocessing, perform word segmentation and deletion on the second search request text and / or the third search request text in the initial sample data according to a preset probability to obtain a fourth search request text and / or a fifth search request text, and determine the seventh embedded representation vector corresponding to each of the fourth search request texts and / or the eighth embedded representation vector corresponding to each of the fifth search request texts; A positive sample is composed of each of the fourth search request texts and its corresponding seventh embedded representation vector and / or each of the fifth search request texts and its corresponding eighth embedded representation vector, and a positive sample is composed of each of the fourth search request texts and the ninth embedded representation vector and / or each of the fifth search request texts and the tenth embedded representation vector, wherein the ninth embedded representation vector is an embedded representation vector other than the seventh embedded representation sub-vector corresponding to the fourth search request text, and the tenth embedded representation vector is an embedded representation vector other than the eighth embedded representation sub-vector corresponding to the fifth search request text; The target sample data consists of at least one positive sample and at least one negative sample.
6. The method according to claim 3, characterized in that The target sample data includes a target search request text and a target embedded representation vector, wherein the initial language model is iteratively adversarially trained using multiple groups of the target sample data to obtain the target language model, including: For each training batch in the iterative adversarial training process, inputting the target search request text in each group of target sample data in the training batch into the initial language model to obtain a predicted embedded representation vector output by the initial language model; Constructing a target loss function using the target embedded representation vector of each group of target text data and the predicted embedded representation vector; The model parameters of the initial language model are adjusted according to the target loss function.
7. The method according to claim 6, characterized in that Adjusting the model parameters of the initial language model according to the target loss function includes: For each training batch in the iterative process, based on multiple groups of target sample data of the current training batch, a back propagation algorithm is used to determine the first gradient of the target loss function on the model parameters of the current iteration round; Calculating a current adversarial disturbance value of a current iteration round by using a preset disturbance parameter, an adversarial disturbance value of a previous iteration round, and the first gradient, wherein the adversarial disturbance value of an initial iteration round is zero; Adding the current adversarial disturbance value to the model parameters of the initial language model corresponding to the target sample data, and updating the target loss function; Using multiple groups of target sample data of the current training batch, calculating the second gradient of the updated target loss function on the model parameters; An adversarial gradient is determined based on the first gradient and the second gradient, and the adversarial gradient is processed using a preset gradient descent algorithm to adjust model parameters of the initial language model, wherein the gradient descent algorithm includes at least one of the following: an adaptive learning rate gradient descent algorithm and a momentum gradient descent algorithm.
8. A text retrieval and matching device, characterized in that: include: A first acquisition module, configured to acquire a first search request text and extract a subject word from the first search request text; A second acquisition module is used to acquire at least one candidate text that is the same as the subject word from a plurality of texts in a preset database; A feature extraction module, used to extract features from the first search request text and each candidate text using a preset target language model to obtain a first embedded representation vector corresponding to the first search request text and a second embedded representation vector corresponding to each candidate text, wherein the target language model is obtained by iterative adversarial training using multiple groups of target sample data after data enhancement, and each group of target sample data includes at least one positive sample and at least one negative sample; A determination module is used to determine the vector similarity between the first embedded representation vector and each of the second embedded representation vectors, and to determine a target text whose vector similarity is not less than a preset vector similarity threshold from the at least one candidate text.
9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, wherein the device where the non-volatile storage medium is located executes the text retrieval and matching method according to any one of claims 1 to 7 by running the computer program.
10. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, the text retrieval and matching method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Model adversarial training method, medium and terminal
CN112257851A
Image classification model training method and device, electronic equipment and storage medium
CN115346072A
Text retrieval matching method and model training method
CN116842129A
Text matching method and device and electronic equipment
CN117194601A
Mass text retrieval matching method based on semantic analysis
CN117972025A
