Methods, apparatus and electronic devices for text matching

Abstract

This application discloses a method, apparatus, and electronic device for text matching. The method includes: vectorizing a first data set corresponding to the text to be retrieved using a neural network model to obtain a first representation vector corresponding to the first data set. The parameters in the neural network model are determined at least by adversarial gradients, which are obtained by summing the original gradient of the original representation vector corresponding to the first data set and a second gradient after perturbation. Based on the first representation vector, a tree index corresponding to the text to be retrieved is determined. The second data set corresponding to the target text to be retrieved is vectorized using the neural network model to obtain a second representation vector corresponding to the second data set. The second representation vector is then matched with vectors in the tree index to obtain a matching result. This application solves the technical problem in related technologies where the matching degree is low when matching texts with a certain degree of similarity stored between different production domains.

Description

Technical Field

[0001] This application relates to the field of natural language processing, and more specifically, to a method, apparatus, and electronic device for text matching. Background Technology

[0002] With the rapid development of artificial intelligence technology in recent years, especially the ever-evolving natural language processing (NLP), text retrieval and matching has become a core task in the field of NLP. Text matching is indispensable in dialogue systems, recommendation systems, and search engines. Therefore, the quality of Chinese text retrieval and matching methods will significantly impact the healthy development of related businesses.

[0003] To achieve the strategic task of digital transformation, it is necessary to integrate and compare data from multiple production domains, such as the M domain, B domain, and O domain. Some informational data is stored in Chinese text format. Due to historical reasons and departmental management, these texts have a certain degree of similarity, but cannot achieve a completely accurate match (such as supplier names in the M domain and customer names in the B domain). Traditional SQL queries yield limited results, while rule-based similarity retrieval and matching methods (such as TF-IDF) often ignore semantic relevance, resulting in a low matching accuracy.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a method, apparatus, and electronic device for text matching, which at least solves the technical problem in the related art of low matching degree when matching texts with a certain degree of similarity stored between different production domains.

[0006] According to one aspect of the embodiments of this application, a text matching method is provided, comprising: using a neural network model to vectorize a first data set corresponding to a text to be retrieved, obtaining a first representation vector corresponding to the first data set, wherein the parameters in the neural network model are determined at least by an adversarial gradient, the adversarial gradient being obtained by accumulating the original gradient of the original representation vector corresponding to the first data set and a second gradient after perturbation; determining a tree index corresponding to the text to be retrieved based on the first representation vector, wherein the tree index is used to match text corresponding to the target retrieved text; using a neural network model to vectorize a second data set corresponding to the target retrieved text, obtaining a second representation vector corresponding to the second data set; and matching the second representation vector with the vector in the tree index to obtain a matching result.

[0007] Optionally, a neural network model is used to vectorize the first data set corresponding to the text to be retrieved, including: segmenting the first data set into words to obtain a first sub-word sequence; randomly repeating the sub-words in the first sub-word sequence according to a preset repetition rate to obtain a second sub-word sequence; randomly deleting the corresponding sub-words from the first sub-word sequence and the second sub-word sequence respectively through an encoder to obtain positive sample pairs; and generating negative sample pairs from the first sub-word sequence and the second sub-word sequence respectively through a momentum encoder.

[0008] Optionally, a neural network model is used to vectorize the first data set corresponding to the text to be retrieved, including: determining the loss function of the neural network model, wherein the loss function is determined by the size of the data in a batch and the conditional probability when batch processing the first data set, and the conditional probability is the conditional probability of the second sample appearing given that the first sample in the positive sample pair appears.

[0009] Optionally, after determining the loss function of the neural network model, the method further includes: determining the original gradient of a first matrix composed of positive sample pairs and negative sample pairs, wherein the first matrix is ​​determined by the original representation vector corresponding to the first data set; determining the perturbation value based on the original gradient and perturbation parameters, wherein the direction of the perturbation value is consistent with the direction of the original gradient; adding the perturbation value to the first matrix to obtain a second matrix, and determining the second gradient corresponding to the second matrix; and accumulating the second gradient to the original gradient to obtain the adversarial gradient.

[0010] Optionally, the parameters of the neural network model can be adjusted by: determining the number of momentum dimensions of the neural network model as a first number; determining the first parameter obtained from the previous iteration of the neural network model in the current iteration; when adjusting the parameters of the neural network model in the current iteration, aggregating the momentum of the neural network model in each momentum dimension to obtain the momentum sum; and determining the parameters of the neural network model in the current iteration based on the first number, the first parameter, and the momentum sum.

[0011] Optionally, the momentum of each momentum dimension when iterating the neural network model at the current iteration is determined by: determining the first momentum in the target momentum dimension when iterating the neural network model at the current iteration, wherein the target momentum dimension is any one of the first number of momentum dimensions; determining the damping coefficient of the neural network model in the target momentum dimension; determining the first adversarial gradient corresponding to the previous iteration of the neural network model at the current iteration; and determining the second momentum in the target momentum dimension when iterating the neural network model at the current iteration based on the first momentum, the damping coefficient, and the first adversarial gradient.

[0012] Optionally, determining the tree index corresponding to the text to be retrieved based on the first representation vector includes: randomly selecting two points from the space corresponding to the first representation vector as initial center nodes, wherein each vector in the first representation vector corresponds to a node in the space; performing clustering on the other nodes in the first representation vector based on the initial center nodes to obtain two converged cluster center points; determining the perpendicular line connecting the two cluster center points as a hyperplane, wherein the hyperplane divides the space corresponding to the first representation vector into two subspaces; repeatedly performing spatial partitioning in each subspace and determining the hyperplane in each subspace until the number of child nodes contained in the final partitioned subspace is less than or equal to a preset number and then stopping the spatial partitioning, wherein each subspace and the child nodes contained in that subspace are a branch of the tree index.

[0013] Optionally, the second representation vector is matched with the vectors in the tree index to obtain a matching result, including: calculating the similarity between the second representation vector and the representation vectors in the tree index; and outputting the vectors with a similarity greater than a preset threshold to obtain a matching result.

[0014] According to another aspect of the embodiments of this application, a text matching apparatus is also provided, comprising: a first processing module, configured to use a neural network model to vectorize a first data set corresponding to a text to be retrieved, thereby obtaining a first representation vector corresponding to the first data set, wherein the parameters in the neural network model are determined at least by an adversarial gradient, which is obtained by accumulating the original gradient of the original representation vector corresponding to the first data set and the second gradient after perturbation; a determining module, configured to determine a tree index corresponding to the text to be retrieved based on the first representation vector, wherein the tree index is used to match the text corresponding to the target retrieved text; a second processing module, configured to use a neural network model to vectorize a second data set corresponding to the target retrieved text, thereby obtaining a second representation vector corresponding to the second data set; and a matching module, configured to match the second representation vector with the vector in the tree index to obtain a matching result.

[0015] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory for storing program instructions; and a processor connected to the memory for executing program instructions to perform the following functions: using a neural network model to vectorize a first data set corresponding to a text to be retrieved, obtaining a first representation vector corresponding to the first data set, wherein the parameters in the neural network model are determined at least by adversarial gradients, and the adversarial gradient is obtained by accumulating the original gradient of the original representation vector corresponding to the first data set and the second gradient after perturbation; determining a tree index corresponding to the text to be retrieved based on the first representation vector, wherein the tree index is used to match the text corresponding to the target retrieved text; using a neural network model to vectorize a second data set corresponding to the target retrieved text, obtaining a second representation vector corresponding to the second data set; and matching the second representation vector with the vector in the tree index to obtain a matching result.

[0016] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-described text matching method by running the computer program.

[0017] In this embodiment, a neural network model is used to vectorize the first data set corresponding to the text to be retrieved, resulting in a first representation vector corresponding to the first data set. The parameters in the neural network model are determined at least by adversarial gradients, which are obtained by summing the original gradient of the original representation vector corresponding to the first data set and the second gradient after perturbation. Based on the first representation vector, a tree index corresponding to the text to be retrieved is determined, whereby the tree index is used to match the text corresponding to the target retrieved text. The neural network model is then used to vectorize the second data set corresponding to the target retrieved text, resulting in a second representation vector corresponding to the second data set. The second representation vector is matched with the vector in the tree index to obtain a matching result. This achieves the goal of optimizing the parameters of the neural network model through adversarial gradients, thereby improving the matching degree when matching the text to be retrieved with the target retrieved text. This solves the technical problem in related technologies where the matching degree is low when matching texts with a certain degree of similarity stored between different production domains. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing a text matching method according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of a text matching method according to an embodiment of this application;

[0021] Figure 3 This is a structural diagram of a text matching device according to an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] Related technologies apply deep learning to Chinese text retrieval and matching, with common supervised training methods (most notably the dual-tower model). This approach requires a large amount of labeled data for training, often necessitating extensive manual annotation, which incurs significant costs for companies. Unsupervised matching methods based on knowledge transfer and NLP employ dropout. Positive sample pairs generated from the same text are of similar length, while negative sample pairs from different texts vary in length. This difference in length information leads the model to favor sentences of similar length, resulting in lower generalization ability and limited accuracy (recall) for Chinese text retrieval. Furthermore, this approach requires excessive resources and GPU memory for datasets with tens of thousands or even millions of samples, making retrieval implementation difficult.

[0025] To address the aforementioned issues, this application provides a text matching method. While utilizing word repetition and momentum contrast for self-supervised contrastive learning, it introduces adversarial training, employs a convergent momentum optimization architecture, and utilizes the Annoy dense vector search framework (i.e., the Annoy tree index below) during the inference phase. A novel deep learning Chinese text retrieval and matching model is designed, effectively solving the problem of Chinese text retrieval and matching in the absence of a large amount of manually labeled training data. Furthermore, it improves matching accuracy (recall) and retrieval speed, thereby enhancing the retrieval and matching performance. The above text matching method can be used... Figure 1 The computer terminal in the process is used for execution. The following is an explanation of the computer terminal.

[0026] The text matching method embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal for implementing a text matching method is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0027] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be implemented wholly or partially as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element in the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0028] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the text matching method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned text matching method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0029] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0030] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0031] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0032] In the above operating environment, this application provides a text matching method embodiment. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than that shown here.

[0033] Figure 2 This is a flowchart of a text matching method according to an embodiment of this application, such as... Figure 2As shown, the method includes the following steps:

[0034] Step S202: A neural network model is used to vectorize the first data set corresponding to the text to be retrieved, so as to obtain the first representation vector corresponding to the first data set. The parameters in the neural network model are determined by at least the adversarial gradient, which is obtained by accumulating the original gradient of the original representation vector corresponding to the first data set and the second gradient after adding perturbation.

[0035] In step S202 above, before the neural network model is used to vectorize the text to be retrieved, since there may be messy characters in the text that affect the accuracy of the model, it is necessary to clean the first data set corresponding to the text to be retrieved. Specifically, regular expressions can be used to clean the text data in the first data set to remove messy characters such as English and special characters.

[0036] Step S204: Based on the first representation vector, determine the tree index corresponding to the text to be retrieved, wherein the tree index is used to match the text corresponding to the target retrieved text.

[0037] Step S206: The second data set corresponding to the target retrieval text is vectorized using a neural network model to obtain the second representation vector corresponding to the second data set.

[0038] Step S208: Match the second representation vector with the vector in the tree index to obtain the matching result.

[0039] In steps S204 to S208 above, the text to be retrieved (i.e., the text to be retrieved) is input to obtain its embedded representation vector. The word vector dimension and the number of index trees are set, and an annoy tree index is built for the embedded representation vector of the retrieved text. The embedded representation vector of the retrieved text (i.e., the target retrieved text) is searched in the annoy tree index of the retrieved text using the same method to obtain the value with the closest cosine distance and its corresponding text, thus obtaining the retrieval matching result.

[0040] In steps S202 to S208 above, a neural network model is used to vectorize the first data set corresponding to the text to be retrieved, obtaining a first representation vector corresponding to the first data set. The parameters in the neural network model are determined at least by the adversarial gradient, which is the sum of the original gradient of the original representation vector corresponding to the first data set and the second gradient after perturbation. Based on the first representation vector, a tree index corresponding to the text to be retrieved is determined, whereby the tree index is used to match the text corresponding to the target retrieved text. The neural network model is then used to vectorize the second data set corresponding to the target retrieved text, obtaining a second representation vector corresponding to the second data set. The second representation vector is matched with the vector in the tree index to obtain a matching result. This achieves the goal of optimizing the parameters of the neural network model through the adversarial gradient, thereby improving the matching degree when matching the text to be retrieved with the target retrieved text. This solves the technical problem in related technologies where the matching degree is low when matching texts with a certain degree of similarity stored between different production domains.

[0041] In step S202 of the above text matching method, a neural network model is used to vectorize the first data set corresponding to the text to be retrieved. Specifically, the steps include: segmenting the first data set to obtain a first sub-word sequence; randomly repeating the sub-words in the first sub-word sequence according to a preset repetition rate to obtain a second sub-word sequence; randomly deleting the corresponding sub-words from the first sub-word sequence and the second sub-word sequence respectively through an encoder to obtain positive sample pairs; and generating negative sample pairs from the first sub-word sequence and the second sub-word sequence respectively through a momentum encoder.

[0042] In this embodiment of the application, positive and negative sample pairs are formed through contrastive learning: a training model is established based on a pre-trained model of Hugging Face and CLS pooling, and positive and negative sample pairs are formed through the enhanced Simcse method, i.e., the Esimcse method. The specific steps include:

[0043] 1. Forming positive sample pairs: After the input text has been segmented, the word sequence x is randomly repeated at a certain repetition rate to generate a modified word sequence x. + Then, compare the original text x with the text x that repeats the same word. + Positive sample pairs are generated using dropout through the encoder.

[0044] 2. Forming negative sample pairs: Within a batch, the results generated by the encoder (i.e., the momentum encoder mentioned above) that updates the parameters of different texts from the previous batch are stored in the queue for generating negative sample pairs. From this queue, negative sample pairs formed using different texts in the current batch are generated.

[0045] In step S202 of the above text matching method, a neural network model is used to vectorize the first data set corresponding to the text to be retrieved. The method also includes the following steps: determining the loss function of the neural network model, wherein the loss function is determined by the size of the data in a batch and the conditional probability when batch processing the first data set, and the conditional probability is the conditional probability of the second sample appearing when the first sample appears in the positive sample pair.

[0046] In this embodiment of the application, when determining the loss function, in each batch, all positive and negative sample data pairs after dropout of the input text data are trained, and the set loss function is:

[0047]

[0048] Where N is the size of a batch, and the sample pairs in the batch are (queu) i ,pos j When i = j, it is a positive sample pair; when i ≠ j, it is a negative sample pair. P(pos) i |que i S(que) is the conditional probability that the second sample appears when the first sample in a positive sample pair appears. i ,pos i ) represents the output of the neural network.

[0049] In the above steps, after determining the loss function of the neural network model, the method further includes the following steps: determining the original gradient of the first matrix composed of positive sample pairs and negative sample pairs, wherein the first matrix is ​​determined by the original representation vector corresponding to the first data set; determining the perturbation value based on the original gradient and perturbation parameters, wherein the direction of the perturbation value is consistent with the direction of the original gradient; adding the perturbation value to the first matrix to obtain the second matrix, and determining the second gradient corresponding to the second matrix; accumulating the second gradient to the original gradient to obtain the adversarial gradient.

[0050] In this embodiment, the loss function value calculated based on the forward propagation result is used for adversarial training after the neural network obtains its gradient through backpropagation. Specifically, the perturbation r is derived from the gradient of the embedding matrix using the following formula for the sign function:

[0051] r = ε*sign(g)

[0052] Where ε is the perturbation parameter, g is the original gradient of the embedding matrix, and sign(g) is the result of taking the sign function of the original gradient, using the sign function to ensure that the direction of the perturbation is consistent with the direction of the gradient. The above perturbation r is added to the embedding, the forward loss function value is calculated for the modified embedding, and the updated gradient is obtained by backpropagation. The original gradient of the embedding is recovered, and the updated gradient is accumulated to the original gradient to obtain the final adversarial gradient gf.

[0053] In the above text matching method, the parameters of the neural network model are adjusted in the following way: the number of momentum dimensions of the neural network model is determined as a first number; the first parameter obtained from the previous iteration of the neural network model is determined for the current iteration; when adjusting the parameters of the neural network model for the current iteration, the momentum of the neural network model under each momentum dimension is aggregated to obtain the momentum sum; based on the first number, the first parameter, and the momentum sum, the parameters corresponding to the neural network model for the current iteration are determined.

[0054] In the above text matching method, the momentum of each momentum dimension when iterating the neural network model at the current number of iterations is determined as follows: the first momentum in the target momentum dimension is determined when it was the previous iteration of the neural network model at the current number of iterations, wherein the target momentum dimension is any one of the first number of momentum dimensions; the damping coefficient of the neural network model in the target momentum dimension is determined; the first adversarial gradient corresponding to the previous iteration of the neural network model at the current number of iterations is determined; and the second momentum in the target momentum dimension is determined when iterating the neural network model at the current number of iterations based on the first momentum, the damping coefficient, and the first adversarial gradient.

[0055] In this embodiment, when finding the optimal parameters of the model based on the adversarial gradient gf, a convergent momentum optimization architecture is used, setting K momentum dimensions. For the t-th model iteration step, the momentum of the loss function under different dimensions is calculated, as shown in the following formula:

[0056]

[0057] in, β is the momentum in the i-th dimension direction during the t-th model iteration (i.e., the second momentum mentioned above), with an initial value of 0. (i) It is the damping coefficient in the i-th dimension. It is the momentum in the i-th dimension direction during the (t-1)-th model iteration (i.e., the first momentum mentioned above), which is the momentum in the i-th dimension direction during the previous training step, gf t-1 This is the first adversarial gradient from the previous training step. Then, it applies to the model's current parameters θ. tBased on the first parameter θ from the previous training step t-1 Adaptive adjustment is performed using the following formula:

[0058]

[0059] Where, r t is the weight of the momentum optimization model parameters in the t-th iteration. As can be seen, according to the above formula, the momentum of K different dimensions is aggregated, that is, linear averaged, to iteratively learn and find the optimal parameters of the model, so that the deep learning model can stably converge to the optimal model.

[0060] In step S204 of the above text matching method, the tree index corresponding to the text to be retrieved is determined based on the first representation vector. Specifically, this includes the following steps: randomly selecting two points from the space corresponding to the first representation vector as initial center nodes, wherein each vector in the first representation vector corresponds to a node in the space; performing clustering on the other nodes in the first representation vector based on the initial center nodes to obtain two converged cluster center points; determining the perpendicular line connecting the two cluster center points as a hyperplane, wherein the hyperplane divides the space corresponding to the first representation vector into two subspaces; repeatedly performing space partitioning in each subspace and determining the hyperplane in each subspace until the number of child nodes contained in the final partitioned subspace is less than or equal to a preset number, at which point the space partitioning is stopped, wherein each subspace and the child nodes contained in that subspace serve as a branch of the tree index.

[0061] In step S208 of the above text matching method, the second representation vector is matched with the vector in the tree index to obtain the matching result. Specifically, the steps include: calculating the similarity between the second representation vector and the representation vector in the tree index; outputting the vector with a similarity greater than a preset threshold to obtain the matching result.

[0062] The following example illustrates the text matching method described above:

[0063] Step 1: Use regular expressions to clean the supplier names and BSS customer names.

[0064] Due to historical management issues across various business lines, the original data contained many irregularities. For example, "A Province B Technology Company RWSHI" was cleaned using regular expressions to retain Chinese characters in the name, resulting in "A Province B Technology Company".

[0065] Step 2: Build a model and form positive and negative sample pairs through comparative learning.

[0066] A pre-trained model, simcse-chinese-roberta-wwm-ext, was constructed with a word repetition rate of 0.32 and a queue length of 150. The original text and the texts with repeated words were processed by the encoder and dropout to form positive sample pairs. The results generated by the encoder from different texts in the previous batch were stored in the queue for generating negative sample pairs. From this queue, negative sample pairs formed by different texts in the current batch were generated. The positive and negative sample data pairs were formed by comparison and learning.

[0067] Step 3: Model Training

[0068] During the experiment, a preset GPU can be used for training, with the number of training generations set to 2, the size of each batch to 64, and 10% of the training data used as a warm-up. The warm-up is a method for optimizing the learning rate. The momentum dimension K is 3, and the damping coefficients for different dimensions are 0.0, 0.9, and 0.99, respectively. At the same time, the aggregated momentum optimizer is used as a convergence tool to find the optimal model. This involves multiplying multiple momentum in different dimensions by different damping coefficients and averaging them to achieve aggregation, so that the deep learning model is optimized along the direction of low curvature. The optimal model result is then saved locally.

[0069] Step 4: Retrieval Match Inference

[0070] The optimal model obtained from the above training is loaded to encode the customer name data in the BSS domain (the aforementioned B domain) to form an embedded text vector, and an annoy index is built on it. Then, the optimal model is used to encode the supplier name data in the MSS domain, and a cosine distance index query is performed on the representation vector. The matching result with the highest score is retained. For example, the name of the supplier in the MSS domain (the aforementioned M domain) is "A Province B Technology Company", and the name of the customer in the BSS domain is "A Province B Technology Co., Ltd." In this way, the semantic relevance is learned by the neural network model, and the correct result is obtained.

[0071] The text matching method provided in this application has the following advantages: 1. For training text pairs formed by contrastive learning, the gradient of the embedding matrix is ​​updated by perturbation calculated by the sign function, and the updated gradient is accumulated on the original gradient to obtain the final adversarial gradient, thereby improving the generalization ability of the model; 2. When searching for the optimal parameters of the model based on the adversarial gradient, multiple momentum in different dimensions are multiplied by different damping coefficients and averaged to achieve aggregation, so that the deep learning model is optimized along the direction of low curvature to improve convergence performance, accelerate the convergence speed and achieve stable training, so as to obtain the globally optimal improved aggregated momentum optimized contrastive learning model; 3. During retrieval inference, an annoy tree index is established for the embedded representation vector of the retrieved text. The system also uses the trained model to obtain embedded representation vectors for the retrieved text, and then performs a cosine distance search on the established annoy tree index of the retrieved text to obtain the best matching text, providing fast and efficient retrieval matching capabilities; 4. It utilizes word repetition to generate positive sample pairs for contrastive learning, ensuring that the length of the positive sample pairs does not affect their semantics, avoiding the use of length information as a learning basis by positive sample pairs of the same length, which would affect the model's generalization; and it keeps the outputs of adjacent mini-batches in a queue, using momentum contrast to generate negative sample pairs, thus expanding the generation of negative sample pairs; 5. The retrieval matching accuracy (recall rate) is higher; 6. The retrieval requires less GPU memory and is faster. Test results show that for a dataset of 790,498 retrieved texts, using the traditional cyclic semantic search method for retrieval inference, only 5,000 retrieved texts can be matched at a time, and the time required for 5,000 retrieved texts is 402.31 seconds. This method can achieve a near-indistinguishable full-scale search and matching inference for a data set of 270,574 search texts under the same hardware conditions, taking only 259.75 seconds, demonstrating a significant improvement in efficiency.

[0072] The original unsupervised SimCSE method was compared with our proposed method using authoritative public Chinese datasets: pku (Peking University Chinese Text Paraphrasing and Matching Dataset), ocnli (Native Chinese Natural Language Matching and Inference Dataset), and lcqmc (Harbin Institute of Technology's Chinese Text Similarity Dataset based on Baidu Knows). (The original classification matching task was changed to a retrieval matching recall task, i.e., the first set of matching data in the validation set was used as the retrieval text set, and the second set of matching data was used as the retrieved text set.) The results are shown in Table 1 below.

[0073] Table 1

[0074]

[0075] As can be seen, for the three authoritative Chinese public datasets, the retrieval matching accuracy (recall) of this method is significantly improved compared with the original method. Among them, the improvement of more than 7% is achieved on the lcqmc dataset, which is the best result.

[0076] Figure 3 This is a structural diagram of a text matching apparatus according to an embodiment of this application, such as... Figure 3 As shown, the device includes:

[0077] The first processing module 32 is used to perform vectorization processing on the first data set corresponding to the text to be retrieved using a neural network model to obtain the first representation vector corresponding to the first data set. The parameters in the neural network model are determined at least by the adversarial gradient, which is obtained by accumulating the original gradient of the original representation vector corresponding to the first data set and the second gradient after adding perturbation.

[0078] The determining module 34 is used to determine the tree index corresponding to the text to be retrieved based on the first representation vector, wherein the tree index is used to match the text corresponding to the target retrieved text;

[0079] The second processing module 36 is used to perform vectorization processing on the second data set corresponding to the target retrieval text using a neural network model to obtain the second representation vector corresponding to the second data set.

[0080] Matching module 38 is used to match the second representation vector with the vector in the tree index to obtain the matching result.

[0081] In the first processing module of the aforementioned text matching device, the first processing module is further configured to perform word segmentation on the first data set to obtain a first sub-word sequence; randomly repeat the sub-words in the first sub-word sequence according to a preset repetition rate to obtain a second sub-word sequence; randomly delete the corresponding sub-words from the first sub-word sequence and the second sub-word sequence respectively through an encoder to obtain positive sample pairs; and generate negative sample pairs from the first sub-word sequence and the second sub-word sequence respectively through a momentum encoder.

[0082] In the first processing module of the aforementioned text matching device, the first processing module is further used to determine the loss function of the neural network model, wherein the loss function is determined by the size of the data contained in a batch and the conditional probability when batch processing the first data set, and the conditional probability is the conditional probability of the second sample appearing given that the first sample in a positive sample pair appears.

[0083] In the first processing module of the aforementioned text matching device, the first processing module is further configured to determine the original gradient of a first matrix composed of positive sample pairs and negative sample pairs, wherein the first matrix is ​​determined by the original representation vector corresponding to the first data set; determine the perturbation value based on the original gradient and perturbation parameters, wherein the direction of the perturbation value is consistent with the direction of the original gradient; add the perturbation value to the first matrix to obtain a second matrix, and determine the second gradient corresponding to the second matrix; and accumulate the second gradient onto the original gradient to obtain the adversarial gradient.

[0084] In the aforementioned text matching device, the parameters of the neural network model are adjusted in the following manner: the number of momentum dimensions of the neural network model is determined as a first number; the first parameter obtained from the previous iteration of the neural network model is determined for the current iteration; when adjusting the parameters of the neural network model for the current iteration, the momentum of the neural network model under each momentum dimension is aggregated to obtain the momentum sum; based on the first number, the first parameter, and the momentum sum, the parameters corresponding to the neural network model for the current iteration are determined.

[0085] In the aforementioned text matching apparatus, the momentum of each momentum dimension during the current iteration of the neural network model is determined as follows: the first momentum in the target momentum dimension is determined during the previous iteration of the neural network model at the current iteration, wherein the target momentum dimension is any one of the first number of momentum dimensions; the damping coefficient of the neural network model in the target momentum dimension is determined; the first adversarial gradient corresponding to the previous iteration of the neural network model at the current iteration is determined; and the second momentum in the target momentum dimension is determined based on the first momentum, the damping coefficient, and the first adversarial gradient during the current iteration of the neural network model.

[0086] In the determination module of the aforementioned text matching device, the determination module is further configured to randomly select two points from the space corresponding to the first representation vector as initial center nodes, wherein each vector in the first representation vector corresponds to a node in the space; based on the initial center nodes, perform clustering on the other nodes in the first representation vector to obtain two converged cluster center points; determine the perpendicular line connecting the two cluster center points as a hyperplane, wherein the hyperplane divides the space corresponding to the first representation vector into two subspaces; repeatedly perform space partitioning in each subspace and determine the hyperplane in each subspace until the number of child nodes contained in the final partitioned subspace is less than or equal to a preset number and then stop performing space partitioning, wherein each subspace and the child nodes contained in the subspace are used as a branch of the tree index.

[0087] In the matching module of the aforementioned text matching device, the matching module is further used to calculate the similarity between the second representation vector and the representation vector in the tree index; output the vector with a similarity greater than a preset threshold to obtain the matching result.

[0088] It should be noted that, Figure 3 The text matching device shown is used to perform Figure 2 The text matching method shown above is also applicable to the text matching device, and will not be repeated here.

[0089] This application also provides an electronic device, including: a memory for storing program instructions; and a processor connected to the memory for executing program instructions that perform the following functions: using a neural network model to vectorize a first data set corresponding to the text to be retrieved, obtaining a first representation vector corresponding to the first data set, wherein the parameters in the neural network model are determined at least by adversarial gradients, and the adversarial gradient is obtained by accumulating the original gradient of the original representation vector corresponding to the first data set and the second gradient after perturbation; determining a tree index corresponding to the text to be retrieved based on the first representation vector, wherein the tree index is used to match the text corresponding to the target retrieved text; using a neural network model to vectorize a second data set corresponding to the target retrieved text, obtaining a second representation vector corresponding to the second data set; and matching the second representation vector with the vector in the tree index to obtain a matching result.

[0090] It should be noted that the aforementioned electronic equipment is used to perform Figure 2 The text matching method shown above also applies to this electronic device, and will not be repeated here.

[0091] This application embodiment also provides a non-volatile storage medium, which includes a stored computer program. The device containing the non-volatile storage medium executes the following text matching method by running the computer program: a neural network model is used to vectorize a first data set corresponding to the text to be retrieved, obtaining a first representation vector corresponding to the first data set. The parameters in the neural network model are determined at least by adversarial gradients, which are obtained by accumulating the original gradient of the original representation vector corresponding to the first data set and a second gradient after perturbation. Based on the first representation vector, a tree index corresponding to the text to be retrieved is determined, wherein the tree index is used to match text corresponding to the target retrieved text. A neural network model is used to vectorize a second data set corresponding to the target retrieved text, obtaining a second representation vector corresponding to the second data set. The second representation vector is matched with the vector in the tree index to obtain a matching result.

[0092] It should be noted that the aforementioned non-volatile storage media is used for execution. Figure 2 The text matching method shown above also applies to this non-volatile storage medium, and will not be repeated here.

[0093] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0094] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0095] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0096] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0097] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0098] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0099] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A text matching method, characterized in that, include: A neural network model is used to vectorize the first data set corresponding to the text to be retrieved, so as to obtain the first representation vector corresponding to the first data set. The parameters in the neural network model are determined by at least the adversarial gradient, which is obtained by accumulating the original gradient of the original representation vector corresponding to the first data set and the second gradient after adding perturbation. Based on the first representation vector, the tree index corresponding to the text to be retrieved is determined, wherein the tree index is used to match the text corresponding to the target retrieved text; The neural network model is used to vectorize the second data set corresponding to the target retrieval text to obtain the second representation vector corresponding to the second data set. The second representation vector is matched with the vector in the tree index to obtain the matching result; The method employs a neural network model to vectorize a first data set corresponding to the text to be retrieved, including: determining the loss function of the neural network model; after determining the loss function, the method further includes: determining the original gradient of a first matrix composed of positive and negative sample pairs, wherein the first matrix is ​​determined by the original representation vector corresponding to the first data set; determining a perturbation value based on the original gradient and perturbation parameters, wherein the direction of the perturbation value is consistent with the direction of the original gradient; adding the perturbation value to the first matrix to obtain a second matrix, and determining the second gradient corresponding to the second matrix; and accumulating the second gradient onto the original gradient to obtain the adversarial gradient.

2. The method according to claim 1, characterized in that, A neural network model is used to vectorize the first dataset corresponding to the text to be retrieved, including: The first data set is segmented to obtain the first sub-word sequence; The subwords in the first subword sequence are randomly repeated according to a preset repetition rate to obtain the second subword sequence; The first and second sub-word sequences are randomly deleted by the encoder to obtain positive sample pairs. The first sub-word sequence and the second sub-word sequence are respectively used to generate negative sample pairs through a momentum encoder.

3. The method according to claim 2, characterized in that, The loss function is determined by the size of the data in a batch and the conditional probability when batch processing the first data set, wherein the conditional probability is the conditional probability of the second sample appearing given that the first sample in the positive sample pair has appeared.

4. The method according to claim 1, characterized in that, The parameters of the neural network model can be adjusted in the following ways: The number of momentum dimensions in the neural network model is determined to be a first number; Determine the first parameter obtained from the previous iteration of the neural network model for the current iteration; When the parameters of the neural network model are adjusted at the current number of times, the momentum of the neural network model in each momentum dimension is aggregated to obtain the momentum sum; Based on the first quantity, the first parameter, and the momentum, the parameters corresponding to the neural network model at the current number of iterations are determined.

5. The method according to claim 4, characterized in that, The momentum of each momentum dimension is determined as follows when iterating the neural network model at the current number of iterations: Determine the first momentum in the target momentum dimension when the neural network model was iterated for the current number of iterations, wherein the target momentum dimension is any one of the momentum dimensions in the first number of iterations; Determine the damping coefficient of the neural network model in the target momentum dimension; Determine the first adversarial gradient corresponding to the previous iteration of the neural network model at the current iteration number; Based on the first momentum, the damping coefficient, and the first adversarial gradient, a second momentum in the target momentum dimension is determined when iterating the neural network model at the current number of iterations.

6. The method according to claim 1, characterized in that, Based on the first representation vector, the tree index corresponding to the text to be retrieved is determined, including: Two points are randomly selected from the space corresponding to the first representation vector as initial center nodes, wherein each vector in the first representation vector corresponds to a node in the space; Based on the initial center node, clustering is performed on the other nodes in the first representation vector to obtain two converged cluster center points; The perpendicular line connecting the two cluster centers is defined as the hyperplane, which divides the space corresponding to the first representation vector into two subspaces. After repeatedly performing spatial partitioning within each subspace, a hyperplane is determined within each subspace. The spatial partitioning is stopped when the number of child nodes contained in the final partitioned subspace is less than or equal to a preset number. Each subspace and the child nodes contained in that subspace serve as a branch of the tree index.

7. The method according to claim 1, characterized in that, The second representation vector is matched with the vectors in the tree index to obtain the matching results, including: Calculate the similarity between the second representation vector and the representation vector in the tree index; Output vectors with a similarity greater than a preset threshold to obtain the matching results.

8. A text matching apparatus, characterized in that, include: The first processing module is configured to use a neural network model to vectorize a first data set corresponding to the text to be retrieved, thereby obtaining a first representation vector corresponding to the first data set. The parameters in the neural network model are determined at least by adversarial gradients, which are obtained by accumulating the original gradient of the original representation vector corresponding to the first data set and a second gradient after perturbation. The first processing module is further configured to determine the loss function of the neural network model, determine the original gradient of a first matrix composed of positive and negative sample pairs, wherein the first matrix is ​​determined by the original representation vector corresponding to the first data set; determine a perturbation value based on the original gradient and perturbation parameters, wherein the direction of the perturbation value is consistent with the direction of the original gradient; add the perturbation value to the first matrix to obtain a second matrix, and determine the second gradient corresponding to the second matrix; and accumulate the second gradient onto the original gradient to obtain the adversarial gradient. The determining module is used to determine the tree index corresponding to the text to be retrieved based on the first representation vector, wherein the tree index is used to match the text corresponding to the target retrieved text; The second processing module is used to perform vectorization processing on the second data set corresponding to the target retrieval text using the neural network model to obtain the second representation vector corresponding to the second data set. The matching module is used to match the second representation vector with the vectors in the tree index to obtain the matching result.

9. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor, connected to the memory, is configured to execute program instructions that perform the following functions: vectorizing a first data set corresponding to the text to be retrieved using a neural network model to obtain a first representation vector corresponding to the first data set, wherein the parameters in the neural network model are determined at least by adversarial gradients, the adversarial gradients being the sum of the original gradient of the original representation vector corresponding to the first data set and the second gradient after perturbation; determining a tree index corresponding to the text to be retrieved based on the first representation vector, wherein the tree index is used to match text corresponding to the target retrieved text; vectorizing a second data set corresponding to the target retrieved text using the neural network model to obtain a second representation vector corresponding to the second data set; and converting the first data set into a second representation vector. The two representation vectors are matched with the vectors in the tree index to obtain a matching result; a neural network model is used to vectorize the first data set corresponding to the text to be retrieved, including: determining the loss function of the neural network model; after determining the loss function of the neural network model, the model further includes: determining the original gradient of a first matrix composed of positive sample pairs and negative sample pairs, wherein the first matrix is ​​determined by the original representation vector corresponding to the first data set; determining a perturbation value based on the original gradient and perturbation parameters, wherein the direction of the perturbation value is consistent with the direction of the original gradient; adding the perturbation value to the first matrix to obtain a second matrix, and determining the second gradient corresponding to the second matrix; and accumulating the second gradient to the original gradient to obtain the adversarial gradient.

10. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the text matching method according to any one of claims 1 to 7 by running the computer program.

Images