Text retrieval method and device and electronic equipment

Abstract

The invention discloses a text retrieval method and device and electronic equipment. The method comprises the following steps: acquiring a retrieved text and a to-be-retrieved text; the retrieved text and the to-be-retrieved text are processed through a text characterization model, a first characterization vector and a second characterization vector are obtained, the text characterization model is trained according to the positive and negative sample pairs after data enhancement, and model parameters are optimized through a target gradient optimizer according to the adversarial gradient of the positive and negative sample pairs; a target index is determined according to the first representation vector, query is carried out in the target index, a text retrieval result corresponding to the second representation vector is obtained, and the target index is a multi-level graph structure index which is connected according to the similarity among all text vectors in the first representation vector. According to the method and the device, the technical problems of relatively low text retrieval matching accuracy and relatively slow retrieval matching speed under the condition of lack of large-scale manually-labeled label data in related technologies are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of natural language processing, and in particular, to a text retrieval method and device and electronic equipment. BACKGROUND

[0002] In recent years, with the rapid progress of artificial intelligence technology, especially the breakthroughs in the field of NLP (Natural Language Processing), text retrieval and matching have become the core of many technical applications. Whether it is the interactive understanding of dialogue systems, personalized services of recommendation systems, or accurate information acquisition of search engines, text retrieval and matching are indispensable. Currently, the industry mainly adopts two methods to solve the problem of text retrieval and matching: rule-based retrieval and matching method and supervised learning method based on neural networks.

[0003] Specifically, the rule-based retrieval and matching method usually relies on word frequency, inverse document frequency (TF-IDF) or n-gram and other statistical features to calculate the similarity of text. Although this rule-based method is simple and easy to implement, it often ignores the context relationship and semantic association of the text when dealing with semantic-related matching tasks, especially in the face of changes in text order or grammatical structure, the performance is significantly reduced, and the matching accuracy is greatly reduced. In addition, the rule-based method is often difficult to capture deep semantic information, resulting in poor matching effect in complex context.

[0004] The supervised learning method based on neural networks usually adopts deep learning models such as convolutional neural networks, recurrent neural networks and transformers, etc., to automatically extract semantic features of text, realizing higher level of text matching. However, a key bottleneck of these methods is the need for a large amount of labeled training data to guide the learning process of the model. For enterprises, collecting and labeling such a large amount of text data is not only time-consuming and laborious, but also costly. Therefore, although this type of method can provide more accurate matching results in theory, it is often limited by the cost barrier of data collection when deployed in practice.

[0005] In addition, in order to solve the above-mentioned limitations of the supervised learning method, an unsupervised learning method is also proposed in the related art, which attempts to train the model through the internal structure of the data itself without relying on manually labeled labels. For example, the SimCSE method introduces the idea of contrast learning into the NLP field, trying to improve the performance of text matching through knowledge transfer in the absence of labels. However, the accuracy and recall rate of the SimCSE method in Chinese text matching are still limited, especially when dealing with long texts or variable semantic scenarios. At the same time, the retrieval efficiency and resource consumption of the unsupervised learning method such as SimCSE in the inference stage are not fully optimized, and the processing efficiency of large-scale data sets is low, consuming a lot of computing resources such as GPU memory.

[0006] At present, there is no effective solution to the above problems. SUMMARY

[0007] The embodiments of the present application provide a text retrieval method, device and electronic equipment to at least solve the technical problems of low accuracy and slow retrieval speed of text retrieval matching in the related art in the absence of large-scale manually labeled label data.

[0008] According to an aspect of an embodiment of the present application, a text retrieval method is provided, comprising: obtaining a searched text and a to-be-searched text; processing the searched text and the to-be-searched text by a text representation model to obtain a first representation vector and a second representation vector, wherein the text representation model is trained according to positive and negative sample pairs after data augmentation, and a target gradient optimizer is used to optimize the model parameters according to the adversarial gradient of the positive and negative sample pairs, and the target gradient optimizer includes multiple gradient optimization and parameter management components; determining a target index according to the first representation vector, and querying in the target index to obtain a text retrieval result corresponding to the second representation vector, wherein the target index is a multi-level graph structure index connected according to the similarity between all text vectors in the first representation vector.

[0009] Optionally, the text representation model is trained by the following method: obtaining original data for training the text representation model, adding noise in the original data according to a preset probability to obtain augmented data, merging the original data and the augmented data to obtain training data; determining positive and negative sample pairs according to the training data, and performing adversarial learning on the positive and negative sample pairs according to a preset number of iterations to obtain an adversarial gradient of the positive and negative sample pairs; determining optimal model parameters by the target gradient optimizer according to the adversarial gradient, and determining the text representation model according to the optimal model parameters.

[0010] Optionally, the determining the positive and negative sample pair according to the training data comprises: determining a random regularization probability in a full connection layer and an attention mechanism in the initial model, wherein the random regularization probability is used to represent a probability of neurons being randomly discarded in a model training process; performing twice encoding operations on a first text by an encoder in the initial model according to the random regularization probability to obtain a first encoding result, and determining the first encoding result as a positive sample, wherein the first text is any one of text data in the training data; performing an encoding operation on a second text by the encoder to obtain a second encoding result, and determining the second encoding result as a negative sample, wherein the second text is other text data in the training data except the first text; and combining the positive sample and the negative sample to obtain the positive and negative sample pair.

[0011] Optionally, before the performing the adversarial learning on the positive and negative sample pair according to the preset number of iterations, the method further comprises: performing a forward propagation on the positive and negative sample pair, and determining an initial loss value of the positive and negative sample pair by a target loss function; performing a backward propagation on the positive and negative sample pair, and determining an initial gradient value of the positive and negative sample pair.

[0012] Optionally, the performing the adversarial learning on the positive and negative sample pair according to the preset number of iterations to obtain an adversarial gradient of the positive and negative sample pair comprises: determining a disturbance accumulation step number and an initial adversarial disturbance for the adversarial training of the positive and negative sample pair; determining an adversarial disturbance of the positive and negative sample pair in a next round of adversarial training according to the initial adversarial disturbance, the initial loss value and the initial gradient value; generating an adversarial sample according to the adversarial disturbance and the positive and negative sample pair, performing an adversarial training on the adversarial sample, and performing a forward propagation and a backward propagation on the adversarial sample respectively to update the initial loss value and the initial gradient value, until the training is stopped when the disturbance accumulation step number is reached, and the adversarial gradient of the positive and negative sample pair is obtained, wherein in the last round of training of the adversarial sample, the updated gradient value is restored to the initial gradient value.

[0013] Optionally, the gradient optimization and parameter management component at least comprises: an adaptive gradient clipping component, a gradient centering component, a positive and negative momentum component, a weight soft regularization and stable decay component, and a weight moving average component, and the method further comprises: clipping a gradient component exceeding a preset gradient threshold in the adversarial gradient by the adaptive gradient clipping component; performing zero-mean processing on the adversarial gradient by the gradient centering component; determining a momentum for positive and negative weight distribution of initial model parameters of the text representation model by the positive and negative momentum component, and updating the initial model parameters according to the momentum; performing soft regularization in the updating process of the initial model parameters by the weight soft regularization and stable decay component, and clipping the updated model parameters until stable decay is achieved; and maintaining an exponential moving average of the initial model parameters in the updating process by the weight moving average component.

[0014] Optionally, determining the target index according to the first representation vector comprises: initializing an index structure parameter, wherein the index structure parameter comprises at least one of the following: a maximum number of connections of each node when determining the target index, a maximum number of neighbors of each node when determining the target index, and a maximum number of neighbors of each node when querying the index; performing normalization processing on the first representation vector to obtain a normalized vector; determining the similarity between all text vectors in the normalized vector, and determining the target index according to the similarity and the index structure parameter, wherein the closer the similarity between the text vectors, the closer the distance between the text vectors in the target index.

[0015] Optionally, querying in the target index to obtain a text retrieval result corresponding to the second representation vector comprises: determining a distance sequence of the second representation vector and all text vectors in the target index, and determining the shortest distance in the distance sequence and the corresponding text vector as the text retrieval result of the second representation vector; in a case where the shortest distance is less than a preset distance threshold, determining that the text retrieval result is valid retrieval; and in a case where the shortest distance is greater than or equal to the preset distance threshold, determining that the text retrieval result is invalid retrieval.

[0016] According to another aspect of the embodiments of the present application, a text retrieval device is further provided, comprising: an acquisition module configured to acquire a searched text and a to-be-searched text; a processing module configured to process the searched text and the to-be-searched text by a text representation model to obtain a first representation vector and a second representation vector, wherein the text representation model is trained according to a positive and negative sample pair after data enhancement, and a target gradient optimizer is used to optimize model parameters according to an adversarial gradient of the positive and negative sample pair, and the target gradient optimizer comprises a plurality of gradient optimization and parameter management components; and a retrieval module configured to determine a target index according to the first representation vector, and query in the target index to obtain a text retrieval result corresponding to the second representation vector, wherein the target index is a multi-level graph structure index connected according to the similarity between all text vectors in the first representation vector.

[0017] According to still another aspect of the embodiments of the present application, an electronic device is further provided, comprising: a memory and a processor, wherein the memory is configured to store program instructions, and the processor is connected with the memory and configured to execute the above-mentioned text retrieval method.

[0018] According to yet another aspect of the embodiments of the present application, a non-volatile storage medium is further provided, comprising a stored computer program, wherein a device in which the non-volatile storage medium is located executes the above-mentioned text retrieval method by running the computer program.

[0019] According to still another aspect of the embodiments of the present application, a computer program product is further provided, comprising computer instructions, which, when executed by a processor, implement the above-mentioned text retrieval method.

[0020] In the embodiment of the present application, the searched text and the text to be searched are obtained, the searched text and the text to be searched are processed by a text representation model respectively to obtain a first representation vector and a second representation vector, wherein the text representation model is trained according to positive and negative sample pairs after data augmentation, and a target gradient optimizer is used to optimize the model parameters according to the adversarial gradient of the positive and negative sample pairs, the target gradient optimizer includes multiple gradient optimization and parameter management components; a target index is determined according to the first representation vector, and a query is performed in the target index to obtain a text retrieval result corresponding to the second representation vector, wherein the target index is a multi-level graph structure index connected according to the similarity between all text vectors in the first representation vector, achieving the purpose of efficient text retrieval and matching without a large amount of manually labeled data, thereby realizing the technical effect of improving the accuracy and speed of text retrieval and matching, and reducing the training resource demand, and further solving the technical problems of low accuracy and slow speed of text retrieval and matching due to the lack of large-scale manually labeled label data in related technologies. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of this application and help to explain the present application, but do not limit the present application in any way. In the drawings:

[0022] Figure 1 is a hardware structure diagram of a computer terminal for implementing a text retrieval method according to an embodiment of the present application;

[0023] Figure 2 is a flowchart of a text retrieval method according to an embodiment of the present application;

[0024] Figure 3 is a principle schematic diagram of a text retrieval process according to an embodiment of the present application;

[0025] Figure 4 is a structure diagram of a text retrieval device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and in the above drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] First, some of the nouns or terms that appear in the process of explaining the embodiments of the present application are applicable to the following explanations:

[0029] NLP (Natural Language Processing): an important direction in the field of computer science and artificial intelligence, which studies various theories and methods that can realize effective communication between man and computer using natural language.

[0030] TF-IDF (Term Frequency-Inverse Document Frequency): a weighting technique used in information retrieval and data mining. The higher the TF-IDF value, the greater the importance of the word to the document, which is commonly used for feature extraction and similarity calculation of text.

[0031] SimCSE: an unsupervised learning method for generating high-quality vector representations of sentences, which captures the semantic information of sentences by using contrastive learning techniques in sentence embedding, and is suitable for text matching and retrieval tasks.

[0032] In order to solve the problem of poor text retrieval efficiency in the related art, the embodiments of the present application provide a text retrieval method, which can be run in Figure 1 The computer terminal is described below.

[0033] The text retrieval method embodiments provided by the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 A hardware structure block diagram of a computer terminal for implementing the text retrieval method is shown. As Figure 1As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0034] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be implemented wholly or partially as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element in the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0035] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the text retrieval method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned text retrieval method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0036] The transmission module 106 is configured to receive or send data via a network. The network can include, for example, a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network interface controller (NIC) that can connect to other network devices through a base station to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module that is configured to communicate with the Internet via wireless communication.

[0037] The display can be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10.

[0038] It is noted that in some alternative embodiments, the above Figure 1 The computer terminal can include hardware elements (including circuitry), software elements (including computer code stored on a computer readable medium), or a combination of both hardware and software elements. It should be noted that in some embodiments, the computer terminal can be comprised of a plurality of computer terminals that are connected together via a network. Figure 1 is merely one example of a particular implementation and is intended to illustrate the types of components that can be present in the computer terminal.

[0039] In the above operating environment, the embodiments of the present application provide a text retrieval method. It is noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0040] Figure 2 is a flowchart of a text retrieval method according to an embodiment of the present application. As shown in the flowchart, the method includes the following steps: Figure 2

[0041] In step S202, the text to be searched and the text to be searched are obtained.

[0042] In step S202, the text to be searched and the text to be searched are obtained.

[0043] ​In step S204, the retrieved text and the text to be retrieved are processed by the text representation model respectively to obtain a first representation vector and a second representation vector, wherein the text representation model is trained according to the positive and negative sample pairs after data augmentation, and the model parameters are optimized according to the adversarial gradient of the positive and negative sample pairs by using a target gradient optimizer, and the target gradient optimizer includes multiple gradient optimization and parameter management components.

[0044] In step S204, the text representation model is improved based on a pre-trained model such as Bert. Overall, first, data augmentation can be performed on the input original data, that is, noise such as missing information “[MASK]” is added to the original text according to a certain probability outside the neural network model to form augmented data. Subsequently, the positive and negative sample pairs required for contrastive learning are generated inside the model. In the training process, the Bert encoder is optimized and trained by using the Free Adversarial Training adversarial training strategy and a target gradient optimizer (such as the momentum adaptive double average gradient optimizer Madgrad) including multiple gradient optimization and parameter management components (such as adaptive gradient clipping, gradient centering, positive and negative momentum, weight soft regularization and stable decay, and weight moving average), to obtain a first representation vector (embedded vector of the retrieved text) and a second representation vector (embedded vector of the text to be retrieved) that can effectively represent the semantic of the text.

[0045] In step S206, a target index is determined according to the first representation vector, and a query is performed in the target index to obtain a text retrieval result corresponding to the second representation vector, wherein the target index is a multi-level graph structure index connected according to the similarity between all text vectors in the first representation vector.

[0046] In step S206, the first representation vector can be normalized first, and then the HNSW index (i.e., the target index) of the multi-level graph structure is constructed based on the similarity between all text vectors. When retrieval is needed, the second representation vector processed in the same way is input into the constructed HNSW index for search query. By performing approximate nearest neighbor search in the HNSW index, the most similar vector to the retrieval vector can be efficiently found, thereby obtaining the text retrieval result that best matches the second representation vector.

[0047] Through the above steps S202 to S206, the purpose of efficient text retrieval and matching without the need for a large amount of manually labeled data is achieved, thereby realizing the technical effect of improving the accuracy and speed of text retrieval and matching while reducing the demand for training resources, and further solving the technical problems of low accuracy and slow speed of text retrieval and matching due to the lack of large-scale manually labeled label data in related technologies.

[0048] Figure 3 is a principle schematic diagram of a text retrieval process according to an embodiment of the present application. As shown in the figure, the process mainly includes three key parts of a text representation model training, HNSW index construction and text retrieval matching. The above text retrieval method is described in detail below. Figure 3 Figure 3 The above text retrieval method is described in detail below.

[0049] I. Text representation model training.

[0050] Optionally, the text representation model is obtained by the following method: obtaining original data for training the text representation model, adding noise in the original data according to a preset probability to obtain enhanced data, merging the original data and the enhanced data to obtain training data; determining a positive-negative sample pair according to the training data, and performing adversarial learning on the positive-negative sample pair according to a preset iteration number to obtain an adversarial gradient of the positive-negative sample pair; determining optimal model parameters using a target gradient optimizer according to the adversarial gradient, and determining the text representation model according to the optimal model parameters.

[0051] In the embodiments of the present application, the training of the text representation model is crucial in that the understanding ability and matching performance of the model for Chinese text are enhanced through data enhancement and deep adversarial learning mechanism, and the use of a target gradient optimizer and the fusion of multiple gradient optimization and parameter management components. The specific steps are analyzed as follows:

[0052] 1. Data enhancement.

[0053] Noise is added to the original data participating in model training according to a preset probability outside the neural network model to obtain data enhanced data, and the enhanced data and the original data are merged to form new training data.

[0054] 2. Model building, forming a positive-negative sample pair through contrastive learning.

[0055] Optionally, the positive-negative sample pair is determined according to the training data, including: determining a random regularization probability in the full connection layer and the attention mechanism in the initial model, wherein the random regularization probability is used to represent the probability of neurons being randomly discarded in the model training process; performing twice encoding operation on a first text through the encoder in the initial model according to the random regularization probability to obtain a first encoding result, and determining the first encoding result as a positive sample, wherein the first text is any one of the text data in the training data; performing encoding operation on a second text through the encoder to obtain a second encoding result, and determining the second encoding result as a negative sample, wherein the second text is other text data in the training data except the first text; combining the positive sample and the negative sample to obtain the positive-negative sample pair.

[0056] ​In the embodiments of the present application, an initial model can be established based on a pre-training model of Hugging Face and CLS pooling, and a full connection layer for constructing positive samples in contrastive learning and a random dropout probability (random regularization probability) in attention mechanism are respectively set in the model. Subsequently, similar data is pulled and dissimilar data is pushed away, a first encoding result obtained by inputting the same text (i.e., a first text) twice through an encoder is taken as a positive sample, a second encoding result obtained by inputting other texts (i.e., second texts) through the encoder is taken as a negative sample, and a positive-negative sample pair is obtained by combination.

[0057] 3. Model training.

[0058] In the embodiments of the present application, the token embedding of the positive-negative sample pair after data enhancement can be combined with the dropout mask mechanism, and Free Adversarial Training (free adversarial training) adversarial learning is added, and a multi-component momentum adaptive double average gradient optimizer is used to perform model coding training on the Bert neural network. The specific steps can be as follows:

[0059] S1: In each batch, a target loss function is set for the input positive-negative sample pair.

[0060] In the embodiments of the present application, before the adversarial learning of the positive-negative sample pair according to the preset iteration number, the positive-negative sample pair is further subjected to forward propagation, and an initial loss value of the positive-negative sample pair is determined through the target loss function; the positive-negative sample pair is further subjected to backward propagation, and an initial gradient value of the positive-negative sample pair is determined.

[0061] The target loss function is as follows:

[0062]

[0063] In the formula, N represents the size of a batch, the sample pair in the batch is que i ,pos i , when i=j, it is a positive sample pair, and when i≠j, it is a negative sample pair, P(pos i |que i ) is the conditional probability of the occurrence of the second sample when the first sample in the positive sample pair occurs, S(que i ,pos i ) represents the output of the neural network.

[0064] S2: After the gradient value (i.e., the initial gradient value) of the neural network is obtained through the backward propagation of the loss function value calculated according to the forward propagation result, the adversarial learning is performed in a multi-step iteration manner.

[0065] Optionally, the positive-negative sample pair is subjected to adversarial learning according to a preset number of iterations to obtain an adversarial gradient of the positive-negative sample pair, comprising: determining a disturbance accumulation step number and an initial adversarial disturbance for adversarial training of the positive-negative sample pair; determining an adversarial disturbance of the positive-negative sample pair in a next round of adversarial training process according to the initial adversarial disturbance, an initial loss value and an initial gradient value; generating an adversarial sample from the adversarial disturbance and the positive-negative sample pair, performing adversarial training on the adversarial sample, and performing forward propagation and backward propagation on the adversarial sample respectively to update the initial loss value and the initial gradient value, until the disturbance accumulation step number is reached to stop training, and an adversarial gradient of the positive-negative sample pair is obtained, wherein in the last round of training of the adversarial sample, the updated gradient value is restored to the initial gradient value.

[0066] In the embodiments of the present application, adversarial learning is introduced. Unlike traditional model training methods, by adding carefully designed adversarial disturbances in the training process, the model learns more robust and more general text representations, thereby improving the accuracy and recall rate of Chinese text retrieval matching. The specific step analysis can be as follows:

[0067] S2.1: Calculate the initial loss value loss of the positive-negative sample pair through the target loss function, and determine the initial gradient value grad of the positive-negative sample pair, that is, perform forward propagation and backward propagation in turn, and backup all initial gradient values.

[0068] S2.2: Set the disturbance accumulation step number of the adversarial sample to m steps, and divide the total training iteration epoch by m, so as to ensure that the calculation number of the total adversarial gradient is consistent with the ordinary model training. Then, for each generation of adversarial sample, m steps of for loop are performed:

[0069] a. Adversarial attack: For the first step, save the initial model parameters and initial gradient values without attack.

[0070] For t steps (t <= m-1), the disturbance of the next t+1 step is calculated according to the following formula and the gradient value of the current embedding layer:

[0071] r t+1 = r t + ε·sign(g)

[0072] In the formula, r t+1 represents the adversarial disturbance of t+1 step; r t+1 represents the adversarial disturbance of t step, that is, the initial adversarial disturbance, the initial value is 0; ε represents the disturbance parameter; g represents the initial gradient value of the embedding layer; sign(g) represents the sign function result of the initial gradient value; according to the obtained adversarial disturbance r t+1 and the positive-negative sample pair, a new round of adversarial sample can be obtained.

[0073] b. if-else branch: if it is not the m-1th step, the current gradient of the model is cleared, and the new round of adversarial samples is forward propagated and backward propagated to update the loss value and gradient value; when it is the m-1th step, i.e., the last round of iteration, the updated gradient value is restored to the initial gradient value saved in the first step to ensure the consistency of the model parameter update and the stability of the gradient calculation.

[0074] S2.3: After completing all steps of the adversarial training, the model parameters backed up in the first step are restored to ensure that the robustness learned by the model during the adversarial training does not affect its performance on standard non-adversarial inputs, and the generalization ability of the model is maintained.

[0075] S2.4: Gradient descent update is performed to adjust the model parameters. In this process, the effects of adversarial perturbations of m iterations are accumulated to obtain adversarial gradients containing adversarial learning information, which are used to reflect the learning direction of the model when processing adversarial perturbations, and help to improve the robustness and generalization ability of the model.

[0076] S3: When finding the optimal parameters of the model according to the adversarial gradient, the Madgrad (momentum adaptive double average gradient optimizer) is used as the target gradient optimizer to optimize the training text representation model.

[0077] Optionally, the gradient optimization and parameter management component at least includes: an adaptive gradient clipping component, a gradient centering component, a positive and negative momentum component, a weight soft regularization and stable decay component, and a weight moving average component. Further, the above method further includes: clipping the gradient component in the adversarial gradient that exceeds the preset gradient threshold through the adaptive gradient clipping component; zero-mean processing of the adversarial gradient through the gradient centering component; determining the momentum of the initial model parameters of the text representation model for positive and negative weight allocation through the positive and negative momentum component, and updating the initial model parameters according to the momentum; soft regularization in the updating process of the initial model parameters through the weight soft regularization and stable decay component, and clipping the updated model parameters until stable decay; maintaining the exponential moving average of the initial model parameters in the updating process through the weight moving average component. The components are analyzed in detail as follows:

[0078] (1) Adaptive gradient clipping: to solve the stability destruction of the random gradient descent caused by the backward propagation of too many gradients and the small batch of sporadic "high loss", reduce the manual adjustment of the model depth, and adaptively ensure that the gradient is below the given preset gradient threshold, the following formula is used for gradient clipping:

[0079]

[0080] wherein, is the gradient of the tth iteration, ε ,Γ is a set initialization parameter constant to avoid freezing zero, r is the dimension being processed, is the model parameter of t iterations.

[0081] Through the above gradient clipping, the unit ratio of the gradient norm and the parameter norm is ensured to be proportional, so that the clipping threshold of the gradient is adaptively and dynamically updated.

[0082] (2) Gradient centering: while adaptively clipping the gradient, the weight gradient is zero-meaned (i.e. centered) using the following formula:

[0083] g t =▽L t (θ t-1 )-m ean(▽L t (θ t-1 ))

[0084] Where g t is the gradient of t iterations, which does not involve the dimension here,▽ is the partial derivative, L t is the loss function of t iterations, θ t-1 is the model parameter of t-1 iterations.

[0085] (3) Positive and negative momentum: when the centered adversarial gradient according to the foregoing adaptive clipping is used to find the optimal parameter through back propagation, the average of the respective moment estimates of odd and even iterations is determined as the current momentum and the previous momentum estimate, respectively, and the assigned momentum is used for parameter weight update, ensuring smoother finding of the optimal value of the loss function, i.e. the optimal parameter of the model.

[0086] (4) Weight soft regularization and stable decay: when updating the model parameters, the L2 norm of the model parameters is used to soft regularize the parameter update weight of the model, thereby limiting the loss function, and at the same time, at the end of each step of training, the parameter value weight in the neural network is directly clipped by a certain proportion, achieving stable decay.

[0087] (5) Weight moving average: keep the exponential moving average of the parameter weight, update the parameter weight every k steps and replace it with the current weight.

[0088] II. HNSW index (i.e. target index) construction.

[0089] Optionally, the HNSW index can be constructed according to the similarity of all text vectors in the first feature vectors, comprising: initializing index structure parameters, wherein the index structure parameters comprise at least one of the following: the maximum number of connections of each node when determining the target index, the maximum number of neighbors of each node when determining the target index, the maximum number of neighbors of each node when querying the index; normalizing the first feature vectors to obtain normalized vectors; determining the similarity between all text vectors in the normalized vectors, and determining the target index according to the similarity and the index structure parameters, wherein the closer the similarity of the text vectors, the closer the distance of the text vectors in the target index.

[0090] In the embodiments of the present application, the HNSW index is used to accelerate the inference phase of text retrieval matching, and in particular when a large number of text vectors are processed, the HNSW can significantly reduce the required computing resources and time. The specific construction process can be as follows:

[0091] 1. Initialize index structure parameters.

[0092] Before constructing the HNSW index, a series of index structure parameters need to be set, including but not limited to:

[0093] Maximum number of connections of each node: used to control the upper limit of the number of other nodes that each node in the index can be connected to, affecting the complexity and query performance of the index;

[0094] Maximum number of neighbors of each node when constructing the index: used to guide how to construct the index, and a higher value often means a more accurate index, and the construction time is also longer;

[0095] Maximum number of neighbors of each node when querying the index: when querying the index, it determines the number of neighbors considered in the query process, affecting the retrieval speed and accuracy.

[0096] 2. Normalization processing.

[0097] When adding data, the first feature vectors generated by the above text representation mode are added to the HNSW index structure one by one after normalization processing. Among them, normalization is to ensure that the length of the vector is consistent, to avoid the influence of vector length on similarity calculation, so that the similarity measurement is more fair and accurate.

[0098] 3. Determine the similarity between text vectors.

[0099] By calculating the similarity (such as cosine similarity) between the normalized text vectors, the similarity relationship of the text vectors is formed. The higher the similarity of the text vectors, the closer the distance of the nodes representing the text vectors in the HNSW index.

[0100] 4. Determine the target index according to the similarity and the index structure parameters.

[0101] Based on the similarity of the text vector and the preset index structure parameter, the HNSW index is constructed. In the construction process, the text vectors with high similarity are placed in adjacent positions in the index structure, forming a multi-layer small world network structure with navigation properties. This enables navigation from the top layer of the index to quickly locate the bottom layer node where the text vector with high similarity is located during retrieval, greatly improving the retrieval efficiency.

[0102] III. Text retrieval and matching.

[0103] Optionally, the query is performed in the target index to obtain a text retrieval result corresponding to the second representation vector, including: determining a distance sequence of the second representation vector and all text vectors in the target index, and determining the shortest distance in the distance sequence and the corresponding text vector as the text retrieval result of the second representation vector; in the case that the shortest distance is less than a preset distance threshold, determining that the text retrieval result is valid retrieval; in the case that the shortest distance is greater than or equal to the preset distance threshold, determining that the text retrieval result is invalid retrieval.

[0104] In the embodiments of the present application, the HNSW index not only accelerates the retrieval speed, but also effectively judges the quality of the retrieval result by setting the preset distance threshold. Specifically, the system calculates the distance sequence of the second representation vector and all text vectors in the index, and selects the shortest distance and the corresponding text vector as the retrieval result. By introducing the distance threshold, when the shortest distance is lower than the preset value, the system determines that the retrieval result is valid, indicating that a highly relevant matching item is found; otherwise, if the shortest distance exceeds or equals the preset threshold, it is considered as invalid retrieval, which effectively filters the candidate matching with low similarity, ensuring the accuracy and practicality of the retrieval result. In particular, in large enterprise data processing scenarios, the efficiency and accuracy of text retrieval and matching can be significantly improved, resource waste is reduced, and the accuracy of business processing is improved.

[0105] In the embodiments of the present application, taking the balance carryover risk mining scene as an example, in order to find audit doubts, the supplier name data sent by the MSS side of a large enterprise data domain needs to be matched with the customer name data on the BSS side through Chinese retrieval. The specific text retrieval process can be as follows:

[0106] 1. Data enhancement.

[0107] For the original data participating in model training, set p_mask to 0.1, and add noise when the number generated by the random function is less than p_mask. Set the added noise to be missing information "[MASK]", that is, add noise outside the neural network model to the original text data with a probability of 0.1, increase the disturbance, and merge the enhanced data with the original data to form new training data.

[0108] 2. Build a model and form positive and negative sample pairs through contrastive learning.

[0109] The training model is established based on the simcse-chinese-roberta-wwm-ext pre-training model of Hugging Face, the random dropout probability of the attention layer and the hidden layer is set to 0.1, the same text input is encoded twice by the encoder using dropout to form a positive sample pair, and the encoding results of other texts are used as negative sample pairs, and the positive and negative sample data pairs are formed through contrastive learning.

[0110] 3. Model training.

[0111] The enterprise platform uses the GPU of TESLA T4 for training, sets the training iteration number to 3, the size of each batch to 64, the bert neural network parameter max_seq_length to 64, 10% of the training data as warm-up, adds Free Adversarial Training for adversarial training, sets the iteration step number m to 5, the perturbation parameter to 0.1, and uses the momentum adaptive double average gradient optimizer with multiple components such as adaptive gradient clipping, gradient centering, positive and negative momentum, weight soft regularization and stable decay, and weight moving average as the convergence tool for finding the optimal model, and saves the optimal text representation model result found in the local.

[0112] 4. Text retrieval and matching.

[0113] Load the text representation model obtained by the above training, encode the BSS domain customer name data to form an embedded vector, and build an HNSW index based on a multi-layer graph structure on this basis, set the maximum number of connections M of each node to 32, the maximum number of neighbors efConstruction of each node when building the index to 100, and the maximum number of neighbors efSearch of each node when searching for neighbors to 100, then encode the MSS domain supplier name data using the above optimal model, query the representation vector, obtain the nearest distance and its corresponding text vector, and obtain the text retrieval result.

[0114] In the embodiments of the present application, an improved data enhancement contrastive learning text retrieval method is proposed, which effectively improves the accuracy (recall rate) and inference speed of Chinese text retrieval and matching, and reduces the artificial cost and technical challenges that enterprises may face when implementing text matching. Specifically, the present application has the following advantages compared with the text retrieval method in the related art:

[0115] 1. The token embedding output by the pre-trained language model is used with dropout, a regularization technique, to form positive and negative sample pairs for contrastive learning within the neural network model. Multiple iterations are performed within a limited range during the deep learning training phase. Adversarial attacks are added based on the perturbations calculated by the sign function during the iterations, which improves the generalization ability of the model.

[0116] 2. By adding noise or missing information from real-world scenarios to the input data outside the neural network model, data augmentation is performed. This trains the model to adapt to noisy inputs, and it is expected that the model can recover the masked information from the context. This enhances the model's robustness to noise and the generation of more stable sentence representations, thereby improving the overall generalization ability of the model.

[0117] 3. When searching for the optimal parameters of the model based on the adversarial gradient, the Madgrad adaptive dual average gradient optimizer with momentum quantization is used as the core optimizer. At the same time, multiple components such as adaptive gradient clipping, gradient centering, positive and negative momentum, soft regularization and stable decay of weights, and moving average of weights are added to optimize the training of the Bert encoder, which can yield the optimal text representation model.

[0118] 4. During retrieval and inference, the embedded representation vector of the retrieved text is normalized, and then an HNSW index based on a multi-level graph structure is established. Similarly, the embedded representation vector of the retrieved text is obtained based on the trained model, and this vector is searched within the established HNSW index of the retrieved text, providing fast and efficient retrieval and matching capabilities.

[0119] 5. Higher retrieval accuracy (recall). Using authoritative public datasets lcqmc and pku, the SimCSE method in related technologies was compared with the text retrieval method in this application. The original classification matching task was changed to a retrieval matching recall task, where the first set of matching data in the validation set was used as the retrieval text set, and the second set of matching data was used as the retrieved text set. Experimental results show that the accuracy of this method on the authoritative lcqmc and pku datasets is improved by 7.36% and 10.19% respectively compared to the original method, with particularly significant improvements on the pku dataset, demonstrating the model's powerful performance in handling semantically complex variations.

[0120] 6、Inference retrieval requires less video memory and is faster. The test was conducted using the 16G video memory of the NVIDIA TESLA T4 GPU. The results show that, for data of 790498 pieces of text to be searched, using the traditional loop semantic search method for retrieval inference, only 5000 pieces of text data can be matched at a time, and the time required for 5000 pieces of text to be searched is 402.31 seconds. The present application can achieve substantially indistinguishable full-text retrieval matching for data of 270574 pieces of text to be searched under the same hardware conditions. In the case of a 50-fold increase in data, the time taken is 2876.35 seconds, and the efficiency is significantly improved.

[0121] In summary, the present application has significant advantages in improving the accuracy, speed and reducing the implementation cost of Chinese text retrieval matching, providing a cutting-edge solution for enterprise data processing and intelligent business, and showing important breakthroughs in natural language processing and artificial intelligence.

[0122] According to an embodiment of the present application, a text retrieval device is provided. It should be noted that the text retrieval device of the present application can be used to execute the text retrieval method provided by the present application. The text retrieval device provided by the present application is introduced as follows.

[0123] Figure 4 is a structural diagram of a text retrieval device according to an embodiment of the present application. As shown in Figure 4 , the device comprises:

[0124] The acquisition module 40 is configured to acquire the text to be searched and the text to be searched.

[0125] The processing module 42 is configured to process the text to be searched and the text to be searched by a text representation model to obtain a first representation vector and a second representation vector, wherein the text representation model is trained according to the positive and negative sample pairs after data augmentation, and the model parameters are optimized according to the adversarial gradient of the positive and negative sample pairs by using a target gradient optimizer, and the target gradient optimizer comprises a plurality of gradient optimization and parameter management components.

[0126] The retrieval module 44 is configured to determine a target index according to the first representation vector, and query in the target index to obtain a text retrieval result corresponding to the second representation vector, wherein the target index is a multi-level graph structure index connected according to the similarity between all text vectors in the first representation vector.

[0127] By the acquisition module, the processing module and the retrieval module in the text retrieval device, the purpose of efficient text retrieval and matching without a large amount of manual annotation data is achieved, thereby achieving the technical effects of improving the accuracy and speed of text retrieval and matching, and reducing the training resource requirement, thereby solving the technical problems of low accuracy and slow speed of text retrieval and matching in the related art in the case of lacking large-scale manually annotated label data.

[0128] In the text retrieval device provided by the embodiments of the present application, the retrieval module is further configured to initialize an index structure parameter, wherein the index structure parameter comprises at least one of the following: a maximum number of connections of each node when determining the target index, a maximum number of neighbors of each node when determining the target index, and a maximum number of neighbors of each node when querying the index; normalize the first feature vector to obtain a normalized vector; determine the similarity between all text vectors in the normalized vector, and determine the target index according to the similarity and the index structure parameter, wherein the closer the similarity between the text vectors, the closer the distance between the text vectors in the target index.

[0129] In the text retrieval device provided by the embodiments of the present application, the retrieval module is further configured to determine a distance sequence of the second feature vector and all text vectors in the target index, and determine the shortest distance in the distance sequence and the corresponding text vector as the text retrieval result of the second feature vector; in the case that the shortest distance is less than a preset distance threshold, determine that the text retrieval result is valid retrieval; in the case that the shortest distance is greater than or equal to the preset distance threshold, determine that the text retrieval result is invalid retrieval.

[0130] In the text retrieval device provided by the embodiments of the present application, the training module 46 is further configured to acquire original data for training the text representation model, add noise in the original data according to a preset probability to obtain enhanced data, merge the original data and the enhanced data to obtain training data; determine a positive-negative sample pair according to the training data, and perform adversarial learning on the positive-negative sample pair according to a preset number of iterations to obtain an adversarial gradient of the positive-negative sample pair; determine optimal model parameters by using a target gradient optimizer according to the adversarial gradient, and determine the text representation model according to the optimal model parameters.

[0131] In the text retrieval device provided in the embodiments of the present application, the training module is further configured to determine a random regularization probability in the full connection layer in the initial model and in the attention mechanism, where the random regularization probability is used to represent a probability of neurons being randomly discarded in the model training process; perform twice encoding operations on the first text according to the random regularization probability through the encoder in the initial model to obtain a first encoding result, and determine the first encoding result as a positive sample, where the first text is any one of the training data; perform an encoding operation on a second text through the encoder to obtain a second encoding result, and determine the second encoding result as a negative sample, where the second text is other text data in the training data except the first text; and combine the positive sample and the negative sample to obtain a positive-negative sample pair.

[0132] In the text retrieval device provided in the embodiments of the present application, the training module is further configured to perform forward propagation on the positive-negative sample pair, and determine an initial loss value of the positive-negative sample pair through a target loss function; perform backward propagation on the positive-negative sample pair, and determine an initial gradient value of the positive-negative sample pair.

[0133] In the text retrieval device provided in the embodiments of the present application, the training module is further configured to determine a disturbance accumulation step number and an initial adversarial disturbance for adversarial training of the positive-negative sample pair; determine an adversarial disturbance of the positive-negative sample pair in a next round of adversarial training process according to the initial adversarial disturbance, the initial loss value and the initial gradient value; generate an adversarial sample according to the adversarial disturbance and the positive-negative sample pair, perform adversarial training on the adversarial sample, and perform forward propagation and backward propagation on the adversarial sample respectively to update the initial loss value and the initial gradient value, until the training is stopped when the disturbance accumulation step number is reached, and an adversarial gradient of the positive-negative sample pair is obtained, where in the last round of training of the adversarial sample, the updated gradient value is restored to the initial gradient value.

[0134] The embodiments of the present application also provide an electronic device, including a memory and a processor, where the memory is used to store program instructions; the processor is connected with the memory and is used to execute the text retrieval method.

[0135] It should be noted that the electronic device is used to execute the text retrieval method as shown in the above method, and therefore the related explanations in the text retrieval method are also applicable to the electronic device, which will not be repeated here. Figure 2

[0136] The embodiments of the present application also provide a non-volatile storage medium, which includes a stored computer program, where a device in which the non-volatile storage medium is located executes the text retrieval method by running the computer program.

[0137] It should be noted that the non-volatile storage medium is used to execute the text retrieval method as shown in the above method, and therefore the related explanations in the text retrieval method are also applicable to the non-volatile storage medium, which will not be repeated here. Figure 2 ​The text retrieval method shown is thus applicable to the nonvolatile storage medium, and the relevant explanations in the above text retrieval method are also applicable to the nonvolatile storage medium, which will not be repeated here.

[0138] The embodiments of the present application further provide a computer program product comprising computer instructions, which, when executed by a processor, implement the above text retrieval method.

[0139] It should be noted that the above computer program product is used to execute Figure 2 The text retrieval method shown is thus applicable to the nonvolatile storage medium, and the relevant explanations in the above text retrieval method are also applicable to the nonvolatile storage medium, which will not be repeated here.

[0140] The above sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0141] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0142] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the device embodiment described above is only illustrative, and for example, the division of the units can be a logical function division, and actual implementation can be in another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other forms.

[0143] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0144] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0145] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0146] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A text retrieval method, characterized in that, include: Retrieve the text to be searched and the text to be retrieved; The text to be retrieved and the text to be retrieved are processed by a text representation model to obtain a first representation vector and a second representation vector. The text representation model is trained based on data-augmented positive and negative sample pairs and a target gradient optimizer is used to optimize the model parameters based on the adversarial gradient of the positive and negative sample pairs. The target gradient optimizer includes multiple gradient optimization and parameter management components. A target index is determined based on the first representation vector, and a query is performed in the target index to obtain the text retrieval result corresponding to the second representation vector. The target index is a multi-level graph structure index that connects all text vectors in the first representation vector based on their similarity.

2. The method according to claim 1, characterized in that, The text representation model is trained in the following way: Obtain the original data for training the text representation model, add noise to the original data according to a preset probability to obtain augmented data, and merge the original data and the augmented data to obtain training data; The positive and negative sample pairs are determined based on the training data, and adversarial learning is performed on the positive and negative sample pairs according to a preset number of iterations to obtain the adversarial gradient of the positive and negative sample pairs. The optimal model parameters are determined using the objective gradient optimizer based on the adversarial gradient, and the text representation model is determined based on the optimal model parameters.

3. The method according to claim 2, characterized in that, Determining the positive and negative sample pairs based on the training data includes: Determine the random regularization probability in the fully connected layer and attention mechanism in the initial model, wherein the random regularization probability is used to represent the probability that a neuron is randomly dropped during model training; The encoder in the initial model performs two encoding operations on the first text according to the random regularization probability to obtain the first encoding result, and the first encoding result is determined as a positive sample, wherein the first text is any text data in the training data; The second text is encoded by the encoder to obtain a second encoding result, and the second encoding result is determined as a negative sample. The second text is other text data in the training data other than the first text. The positive sample and the negative sample are combined to obtain the positive-negative sample pair.

4. The method according to claim 2, characterized in that, Before performing adversarial learning on the positive and negative sample pairs according to a preset number of iterations, the method further includes: The positive and negative sample pairs are forward propagated, and the initial loss value of the positive and negative sample pairs is determined by the target loss function; Backpropagation is performed on the positive and negative sample pairs to determine the initial gradient values ​​of the positive and negative sample pairs.

5. The method according to claim 4, characterized in that, Adversarial learning is performed on the positive and negative sample pairs according to a preset number of iterations to obtain the adversarial gradient of the positive and negative sample pairs, including: Determine the number of perturbation accumulation steps and the initial adversarial perturbation for adversarial training of the positive and negative sample pairs; The adversarial perturbation of the positive and negative sample pairs in the next round of adversarial training is determined based on the initial adversarial perturbation, the initial loss value, and the initial gradient value. Adversarial samples are generated based on the adversarial perturbation and the positive and negative sample pairs. Adversarial training is performed on the adversarial samples, and forward and backward propagation are performed on the adversarial samples respectively. The initial loss value and the initial gradient value are updated until the perturbation accumulation step is reached and training stops. The adversarial gradient of the positive and negative sample pairs is obtained. In the last round of training of the adversarial samples, the updated gradient value is restored to the initial gradient value.

6. The method according to claim 1, characterized in that, The gradient optimization and parameter management components include at least: an adaptive gradient clipping component, a gradient centering component, a positive and negative momentum component, a weight soft regularization and stable decay component, and a weight moving average component. The method further includes: The adaptive gradient clipping component clips gradient components that exceed a preset gradient threshold in the adversarial gradient. The adversarial gradient is zero-meaned by the gradient centering component. The positive and negative momentum components determine the momentum for assigning positive and negative weights to the initial model parameters of the text representation model, and update the initial model parameters based on the momentum. The weighted soft regularization and stable decay components perform soft regularization during the update of the initial model parameters and prune the updated model parameters until stable decay is achieved. The weighted moving average component maintains the exponential moving average of the initial model parameters during the update process.

7. The method according to claim 1, characterized in that, Determining the target index based on the first representation vector includes: Initialize index structure parameters, wherein the index structure parameters include at least one of the following: the maximum number of connections per node when determining the target index, the maximum number of neighbors per node when determining the target index, and the maximum number of neighbors per node during index query; The first representation vector is normalized to obtain a normalized vector; The similarity between all text vectors in the normalized vector is determined, and the target index is determined based on the similarity and the index structure parameters, wherein text vectors with closer similarity are closer in the target index.

8. The method according to claim 7, characterized in that, A query is performed in the target index to obtain text retrieval results corresponding to the second representation vector, including: Determine the distance sequence between the second representation vector and all text vectors in the target index, and determine the shortest distance in the distance sequence and the corresponding text vector as the text retrieval result of the second representation vector; If the shortest distance is less than a preset distance threshold, the text search result is determined to be a valid retrieval. If the shortest distance is greater than or equal to the preset distance threshold, the text search result is determined to be an invalid search.

9. A text retrieval device, characterized in that, include: The acquisition module is used to acquire the text to be searched and the text to be searched. The processing module is used to process the retrieved text and the text to be retrieved respectively through a text representation model to obtain a first representation vector and a second representation vector. The text representation model is trained based on data-augmented positive and negative sample pairs, and a target gradient optimizer is used to optimize the model parameters based on the adversarial gradient of the positive and negative sample pairs. The target gradient optimizer includes multiple gradient optimization and parameter management components. The retrieval module is used to determine a target index based on the first representation vector, and to perform a query in the target index to obtain a text retrieval result corresponding to the second representation vector. The target index is a multi-level graph structure index that connects all text vectors in the first representation vector based on their similarity.

10. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is used to store program instructions; The processor, connected to the memory, is used to execute the text retrieval method according to any one of claims 1 to 8.

11. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the text retrieval method according to any one of claims 1 to 8 by running the computer program.

12. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the text retrieval method according to any one of claims 1 to 8.

Images