Comparison learning training method and system for recalling military documents according to user questions

Abstract

The invention provides a comparative learning training method and system for recalling military documents according to user questions. The method comprises the following steps: preprocessing document data, extracting document keywords, and generating document features; extracting query keywords from the input question to generate query features; based on the input problem, constructing a positive sample pair and a negative sample pair, the positive sample pair comprising the input problem and a corresponding positive sample, and the negative sample pair comprising the input problem and a corresponding negative sample; respectively processing the query features and the document features by using a double-tower model to generate query feature vectors and document feature vectors; calculating the similarity between the query feature vector and the document feature vector, and distributing similarity labels for the positive samples and the negative samples; carrying out soft mining and hard mining processing on the negative sample; the batch size of contrast learning is increased by using a momentum contrast mechanism and a negative sample queue. The processing efficiency and the recall accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a comparative learning training method and system for recalling military documents based on user questions. Background Art

[0002] Information in the military field plays an important role in dealing with criminal investigation, military decision-making, etc. Due to the particularity of the military field, its data needs to be trained in an offline environment. With limited hardware resources, how to effectively and accurately recall the relevant military documents required by users is a big challenge. According to the user's short text problem, the large-model RAG (Retrieval-Augmented Generation) system is particularly critical to achieve accurate recall of long-text military documents. The existing vector embedding model works well in general fields, but its performance in the military field needs to be improved, especially when the document text is long, the processing efficiency and accuracy of the document recall system need to be improved. Summary of the invention

[0003] In view of this, the purpose of this application is to propose a comparative learning training method and system for recalling military documents based on user questions, aiming to improve processing efficiency and accuracy of document recall.

[0004] In order to achieve one of the above disclosed purposes, the present application provides a comparative learning training method for recalling military documents based on user questions, the method comprising:

[0005] Preprocess document data, extract document keywords, and generate document features;

[0006] Extract query keywords from input questions and generate query features;

[0007] Based on the input question, construct a positive sample pair and a negative sample pair, the positive sample pair includes the input question and a corresponding positive sample, and the negative sample pair includes the input question and a corresponding negative sample;

[0008] Using a double-tower model to process the query feature and the document feature respectively, generating a query feature vector and a document feature vector;

[0009] Calculating the similarity between the query feature vector and the document feature vector, and assigning similarity labels to the positive sample and the negative sample;

[0010] Performing soft mining and hard mining processing on the negative samples;

[0011] Use momentum contrast mechanism and negative sample queue to increase the batch size of contrastive learning.

[0012] As a further improvement of an implementation mode of the present application, the preprocessing of the document data, extracting document keywords, and generating document features includes:

[0013] uniformly processing the document data into unstructured document data;

[0014] Cleaning the document data to remove irrelevant information in the document data;

[0015] dividing the document data into manageable units;

[0016] The document keywords are extracted using a pre-trained model, and the weights of the document keywords are adjusted based on a proper noun library.

[0017] As a further improvement of an implementation method of the present application, the dual-tower model includes a query tower and a document tower. The query tower is used to process the input question and perform feature fusion on the input query keywords to generate the query feature vector; the document tower is used to process the document data and generate the document feature vector.

[0018] As a further improvement of an implementation mode of the present application, the calculating the similarity between the query feature vector and the document feature vector and assigning similarity labels to the positive sample and the negative sample includes:

[0019] The similarity between the query feature vector and the document feature vector is calculated using cosine similarity. The expression of the cosine similarity Sim is:

[0020] Sim=Normalize(E q )·Normalize(E d );

[0021] Among them, E q represents the query feature vector, E d represents the document feature vector, and NOrmalize represents the normalization operation of the vector;

[0022] The similarity label between the input question and the positive sample is 1, and the similarity label between the input question and the negative sample is 0.

[0023] As a further improvement of an implementation mode of the present application, the soft mining and hard mining processing on the negative samples includes:

[0024] Using the hard mining to filter out difficult samples and easy samples from the negative samples;

[0025] When the similarity between the difficult sample and the input question is greater than a first threshold, it is determined as a positive sample and is not added to the loss of the negative sample;

[0026] When the similarity between the easy sample and the input question is less than a second threshold, there is no need to continue training, and the negative sample is not added to the loss;

[0027] The soft mining adjusts the weight of the negative sample by adjusting the loss function of the negative sample.

[0028] As a further improvement of an implementation manner of the present application, the expression of the loss of the negative sample is:

[0029]

[0030] in, represents the loss of the i-th negative sample in batch N, BCE represents the binary cross entropy loss function, represents the similarity between the i-th negative sample in batch N and the input question; t hard represents the first threshold, t easy represents the second threshold;

[0031] The adjusting the weight of the negative sample comprises introducing a weight coefficient;

[0032] The weight coefficient W i The expression is:

[0033]

[0034] Here, γ represents the exponential factor.

[0035] As a further improvement of an implementation mode of the present application, the use of momentum contrast mechanism and negative sample queue to increase the batch size of contrastive learning includes:

[0036] The negative sample queue is used to store negative sample representations in a previous batch, and the negative sample representations are used as negative samples of a current batch;

[0037] The momentum comparison mechanism uses a momentum encoder and a query encoder; updating the parameters of the query encoder by back-propagation, and updating the parameters of the momentum encoder by a momentum update mechanism, so that the representation of the momentum encoder is more consistent with the representation of the query encoder;

[0038] The expression of the momentum update mechanism is:

[0039] θ k ←mθ k +(1-m)θ q ;

[0040] Among them, θ k represents the parameters of the momentum encoder, θ qrepresents the parameters of the query encoder; m represents the momentum factor, and its value range is [0,1).

[0041] Based on the same inventive concept, the present application also provides a comparative learning training system for recalling military documents according to user questions, including:

[0042] The document processing module is used to pre-process the document data, extract document keywords, and generate document features;

[0043] The query processing module is used to extract query keywords from the input question and generate query features;

[0044] A sample pair construction module, used to construct a positive sample pair and a negative sample pair based on the input question, wherein the positive sample pair includes the input question and a corresponding positive sample, and the negative sample pair includes the input question and a corresponding negative sample;

[0045] A dual-tower model, used to process the query feature and the document feature respectively to generate a query feature vector and a document feature vector;

[0046] A similarity calculation module, used to calculate the similarity between the query feature vector and the document feature vector, and assign similarity labels to the positive sample and the negative sample;

[0047] A negative sample processing module, used for performing soft mining and hard mining processing on the negative samples;

[0048] An optimization module for increasing the batch size of contrastive learning using momentum contrast mechanisms and negative sample queues.

[0049] Based on the same inventive concept, the present application also provides an electronic device, including: a processor and a memory; the memory stores a computer program, and when the computer program is executed by the processor, the processor executes any of the steps of the comparative learning training method for recalling military documents based on user questions as described above.

[0050] Based on the same inventive concept, the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute any of the above-mentioned comparative learning training methods for recalling military documents based on user questions.

[0051] Compared with the prior art, the technical effect of the present invention is as follows: the present application processes the input problems of short texts of different lengths and the document data of long texts respectively through the dual-tower model, extracts keywords from the input problems and fuses them into more accurate query feature vectors, improves the key information of the vectors, and makes the recall effect better; it also processes the influence of difficult samples in negative samples on the loss through soft mining and hard mining data mining methods, and improves the training stability; by optimizing the embedding vector and similarity calculation method, the accuracy and efficiency of recall can be improved. The present application can effectively carry out comparative learning training in specific fields under limited hardware resources, and improve the relevance of input problems of short texts and document data of long texts. The present application can not only improve processing efficiency and reduce the occupation of hardware resources, but also improve the accuracy of recall, and can meet the personalized needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the implementation methods or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A flowchart of a comparative learning training method for recalling military documents based on user questions provided in an embodiment of the present application;

[0054] Figure 2 A flowchart of document processing provided for another embodiment of the present application;

[0055] Figure 3 A flowchart of negative sample processing provided by another embodiment of the present application;

[0056] Figure 4 A schematic diagram of a comparative learning training system for recalling military documents based on user questions provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0059] Implementation methods It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the implementation methods of this application should be understood by people with ordinary skills in the field to which this application belongs. The "first", "second" and similar words used in the implementation methods of this application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprising" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0060] Related technologies propose a large model for question-answering in the legal field, including: extracting legal data into features, storing them in a vector library, retrieving and recalling legal texts, cases, regulations, etc. according to user input cases, and finally, the large model answers questions within the scope of legal knowledge based on the recalled content and input questions. In the retrieval and recall part, the large model trains a keyword extraction model, which can extract keywords from questions raised by users, and then uses the BERT (bidirectional encoder representation) model to perform vector fusion on the keywords to obtain an overall keyword feature. After that, the cosine similarity between the keyword features and the features in the knowledge base is calculated, and the keyword features are matched with the features of the relevant legal clauses in the legal database. Finally, the model uses the found legal clauses as input and generates answers using the large model for question-answering. However, the large model for question-answering in the legal field cannot be directly applied to the military field, and in the stage of obtaining feature similarity, comparative learning training is required, and general comparative learning training requires a lot of hardware resources.

[0061] The embodiment of the present application provides a comparative learning training method for recalling military documents based on user questions, and the training is performed in a network-free environment, such as Figure 1 As shown, the contrastive learning training method for recalling military documents based on user questions includes the following steps:

[0062] Step S100, pre-processing the document data, extracting document keywords, and generating document features.

[0063] Step S200, extracting query keywords from the input question and generating query features;

[0064] Step S300, constructing a positive sample pair and a negative sample pair based on the input question, the positive sample pair includes the input question and the corresponding positive sample, and the negative sample pair includes the input question and the corresponding negative sample;

[0065] Step S400, using the dual-tower model to process query features and document features respectively, generating a query feature vector and a document feature vector;

[0066] Step S500, calculating the similarity between the query feature vector and the document feature vector, and assigning similarity labels to the positive samples and the negative samples;

[0067] Step S600, performing soft mining and hard mining processing on negative samples;

[0068] Step S700, using momentum contrast mechanism and negative sample queue to increase the batch size of contrastive learning.

[0069] In this application, a dual-tower model is used to process input problems of short texts of different lengths and document data of long texts respectively. By extracting keywords from the input problems and fusing them into more accurate query feature vectors, the key information of the vectors is improved, making the recall effect better. The data mining methods of soft mining and hard mining are also used to process the impact of difficult samples in negative samples on the loss, thereby improving the training stability. By optimizing the embedding vector and similarity calculation method, the accuracy and efficiency of recall can be improved. This application can effectively carry out comparative learning training in specific fields under limited hardware resources, and improve the relevance of input problems of short texts and document data of long texts. This application can not only improve processing efficiency and reduce the occupation of hardware resources, but also improve the accuracy of document recall. It is suitable for solving related problems in the military field and can meet the personalized needs of users.

[0070] Specifically, during the contrastive learning training process, the keyword fusion model will be updated according to the training results. In particular, in step S400, through contrastive learning training, the parameters of the model will be adjusted according to the feedback of the positive and negative sample pairs to optimize the effect of keyword fusion. By optimizing the keyword fusion model, the generated input vector can more accurately capture the main information of the input question, thereby improving the relevance and accuracy of document recall.

[0071] like Figure 2 As shown, in one possible implementation of the present application, step S100 includes:

[0072] Step S110, uniformly processing the document data into unstructured document data;

[0073] Step S120, cleaning the document data to remove irrelevant information in the document data;

[0074] Step S130, dividing the document data into manageable units;

[0075] Step S140: extract document keywords using the pre-trained model, and adjust the weights of the document keywords based on the proper noun library.

[0076] Specifically, by performing data normalization in step S110, the document data can be uniformly processed into unstructured text data, such as using the PP-OCR algorithm to convert images into text, and the pdfMiner tool to convert PDF documents into text. By performing data cleaning in step S120, irrelevant information in the document can be removed, such as special symbols, numbers, unstructured data, etc. By performing document segmentation in step S130, the document can be divided into manageable units, which can be divided into chapters, paragraphs or sentences. By performing feature vector extraction in step S140, the pre-trained model can use the BERT model to extract keywords in the input document. The BERT model can obtain the deep meaning of each entry by understanding the context.

[0077] Specifically, after extracting the document keywords, the embedding model (Embedding) is used to fuse the document keywords into a new embedding vector (doc_feature). Taking the document data in the military field as an example, the military field data has the characteristics of strong domain and diversity. Diversity refers to data forms such as images, texts, structured and unstructured, and strong domain refers to having proper nouns, etc. The proper noun library in the military field can be used to adjust the weight of the document keywords in the embedding model. For example, when the extracted document keywords are in the proper noun library, the weight in the embedding model will be increased.

[0078] Specifically, in step S200, in the keyword extraction stage, the BERT model is used to extract the query keywords (query_keywords) of the input question (query). The BERT model can identify and extract words or phrases that can represent the main content of the question through understanding the context. The extracted query keywords are converted into numerical vectors through the embedding model. The keyword fusion model fuses the embedding vectors of multiple query keywords into an overall feature vector, which can be called an input vector (query_embed_in). This vector contains both the semantic information of the query keywords and the overall semantic content of the input question. In general, the BERT model is used to extract keywords, and then these keywords are fused into a comprehensive input vector. This vector is continuously optimized in subsequent contrastive learning training, and ultimately more accurate document recall is achieved, which helps the model understand the main information in the input question, thereby achieving better recall effects. This process not only depends on the pre-training ability of the BERT model, but also on dynamic optimization in contrastive learning training.

[0079] Specifically, in step S300, the positive sample pair refers to the input question and its corresponding positive sample with high correlation (pos_docs), and the negative sample pair refers to the input question and other negative samples in the selected batch (neg_docs). The positive sample pair helps the model learn which features indicate that the document is relevant to the query, so that documents related to the query can be identified in the recall stage. The negative sample pair helps the model learn which features indicate that the document is irrelevant to the query, so that documents irrelevant to the query can be excluded in the recall stage. By determining the positive and negative sample pairs, the model can be helped to better learn and capture the similarities and differences between the input question and document data.

[0080] In one implementation of the present application, the dual-tower model in step S400 includes a query tower and a document tower. The query tower is used to process input questions and perform feature fusion on input query keywords to generate a query feature vector; the document tower is used to process document data and generate a document feature vector.

[0081] Specifically, the dual-tower model is used to separate the questions of short texts and the reference materials of long texts, and different towers are used to extract and obtain representation vectors for the questions and reference materials. Since the input questions of short texts and the document data of long texts have significantly different characteristics, if the same encoder is used to process the features of both, the confusion of the model will increase and it cannot be distinguished well. The present application uses a dual-tower model to process the features of and the features of respectively, and loads two encoders on both sides of the tower at the same time, wherein the two encoders can be copies of the same model or different models while keeping the output dimensions consistent. The design of the dual-tower model can process the input questions and document data separately, and adopt appropriate processing and encoding methods for different characteristics. It is worth mentioning that the query tower is the tower that performs feature fusion on the query keywords in step 200, and the parameters can be updated in comparative learning training.

[0082] In one possible implementation of the present application, step S500 includes:

[0083] Step S510, using cosine similarity to calculate the similarity between the query feature vector and the document feature vector, the expression of cosine similarity Sim is:

[0084] Sim=NOrmalize(E q )·Normalize(E d );

[0085] Among them, E q represents the query feature vector, E d Represents the document feature vector, and Noemalize represents the normalization operation of the vector;

[0086] The similarity label between the input question and the positive sample is 1, and the similarity label between the input question and the negative sample is 0.

[0087] Specifically, the query keywords are converted into query feature vectors E through the embedding model q , positive samples and negative samples are also converted into document feature vectors E through the embedding model d The generated embedding vector is normalized to ensure that the length of the vector is 1, which helps in the calculation of cosine similarity. The normalized embedding vector is denoted as Normalize(E q ) and Normalize(E d ). The cosine similarity formula is used to calculate the similarity between the normalized query feature vector and the document feature vector. The value of cosine similarity is between -1 and 1, which represents the angle between the vectors. In the similarity label assignment, the similarity label between the input question and the positive sample is 1, indicating a high correlation. The similarity label between the input question and the negative sample is 0, indicating no correlation. Using the similarity calculated above and the preset labels, the model is trained through contrastive learning. The contrastive loss function encourages the model to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. By calculating similarity and assigning labels, it is ensured that the system can accurately recall relevant documents based on the user's query. In other words, by minimizing the contrastive loss, optimizing the model parameters, and recalling documents related to the user's query based on the results of model training and similarity calculations.

[0088] like Figure 3 As shown, in one possible implementation of the present application, step S600 includes:

[0089] Step S610, using hard mining to filter out difficult samples and easy samples from negative samples;

[0090] Step S620, when the similarity between the difficult sample and the input question is greater than the first threshold, it is determined as a positive sample and is not added to the loss of negative samples;

[0091] Step S630: when the similarity between the easy sample and the input question is less than the second threshold, there is no need to continue training and the loss of the negative sample is not added;

[0092] Step S640: soft mining adjusts the weight of negative samples by adjusting the loss function of negative samples.

[0093] Specifically, processing difficult samples is a key step to improve the effect of contrastive learning. Difficult samples refer to samples that are difficult for the model to correctly classify. When determining positive and negative samples in step 300, negative samples are directly selected from other documents in the batch, so positive samples may be mistakenly judged as negative samples. Soft mining and hard mining methods are usually used to process difficult samples. Hard mining refers to directly screening out difficult samples and easy samples from a large number of samples.

[0094] Taking the M3E (Moka Massive Mixed Embedding) embedding model as an example, it has been pre-trained with a large amount of Chinese corpus and has good capabilities in text similarity tasks. Therefore, when doing hard mining, for difficult samples, the first threshold t is set easy , when the similarity between the negative sample and the question is greater than the first threshold, it is considered a positive sample and is not included in the loss of the negative sample. For easy samples, set the second threshold t easy , when the similarity between the negative sample and the question is less than the second threshold, there is no need to continue training and it is not added to the negative sample loss. Compared with hard mining, which directly selects samples that do not need to be trained, soft mining adjusts the model's attention or weight to negative samples. Adjusting the weight of negative samples includes introducing the weight coefficient W i .

[0095] Among them, the expression of the loss of negative samples is:

[0096]

[0097] in, represents the loss of the i-th negative sample in batch N, BCE represents the binary cross entropy loss function, Indicates the similarity between the i-th negative sample and the input question in batch N; t hard represents the first threshold, t easy Indicates the second threshold.

[0098] Among them, the weight coefficient W i The expression is:

[0099]

[0100] Wherein, γ represents an exponential factor. In this embodiment, γ is set to 0.2.

[0101] Among them, hard mining can involve identifying negative samples with high similarity to the input question to distinguish different document categories. Soft mining can involve assigning weights to negative samples based on their similarity to the input question, so that the model pays more attention to negative samples that are more difficult to classify during training. The combination of hard mining and soft mining can achieve a more comprehensive negative sample processing strategy. First, hard mining is used to identify difficult samples, and then soft mining is used to assign appropriate weights to these samples and other negative samples. This combined method can help the model learn more effectively and improve the accuracy and robustness of recall.

[0102] In one possible implementation of the present application, step S700 includes:

[0103] The negative sample queue is used to store negative sample representations in previous batches, which are used as negative samples in the current batch;

[0104] The momentum comparison mechanism uses a momentum encoder and a query encoder; the parameters of the query encoder are updated through back-propagation, and the parameters of the momentum encoder are updated through the momentum update mechanism, so that the representation of the momentum encoder is more consistent with the representation of the query encoder.

[0105] Among them, the expression of momentum update mechanism is:

[0106] θ k ←mθ k +(1-m)θ q ;

[0107] Among them, θ k represents the parameters of the momentum encoder, θ q represents the parameters of the query encoder; m represents the momentum factor, and its value range is [0,1). In this embodiment, the value of m is 0.995.

[0108] Specifically, according to the positive and negative sample labels obtained in step S500, the data storage is updated by MoCo (Momentum Contrast) method. MoCo is a technology for unsupervised representation learning, which increases the batch size of contrastive learning through momentum contrast mechanism and negative sample queue. Among them, the momentum encoder (KeyEncoder) is represented as f k , whose parameter is denoted by θ k ; The momentum encoder is responsible for generating the representation of the data and maintaining the consistency of the representation through the momentum update mechanism. The query encoder is represented by f q , whose parameter is denoted by θ q; The query encoder is responsible for generating representations of the current mini-batch. The negative sample queue is used to store representations of previous mini-batches, gradually replacing the oldest representations in the queue as training progresses. In traditional contrastive learning, the size of the dictionary (i.e., the queue) is limited by the size of the mini-batch. MoCo decouples the dictionary size from the mini-batch size, allowing the use of larger dictionaries, thereby increasing the effective batch size. The parameter θ of the query encoder is q Directly updated through back-propagation. The momentum update makes the representation of the momentum encoder more consistent with the representation of the query encoder, improving the consistency of the dictionary. The contrastive loss is calculated using the similarity labels of the positive and negative sample pairs, and the model is trained by minimizing the loss. In each training batch, the model calculates the similarity between the query representation and the positive sample representation, as well as the dissimilarity between the query representation and the negative sample representation. In general, the MoCo method improves the quality of representation in unsupervised learning through these mechanisms, especially in resource-constrained situations, can effectively utilize historical information, enhance the generalization ability of the model, and can perform more effective training with limited hardware resources.

[0109] like Figure 4 As shown, another embodiment of the present application discloses a comparative learning training system for recalling military documents according to user questions, including:

[0110] The document processing module is used to pre-process the document data, extract document keywords, and generate document features;

[0111] The query processing module is used to extract query keywords from the input question and generate query features;

[0112] A sample pair construction module is used to construct positive sample pairs and negative sample pairs based on the input question. The positive sample pair includes the input question and the corresponding positive sample, and the negative sample pair includes the input question and the corresponding negative sample.

[0113] The dual-tower model is used to process query features and document features respectively to generate query feature vectors and document feature vectors;

[0114] A similarity calculation module is used to calculate the similarity between the query feature vector and the document feature vector, and assign similarity labels to positive samples and negative samples;

[0115] Negative sample processing module, used to perform soft mining and hard mining on negative samples;

[0116] An optimization module for increasing the batch size of contrastive learning using momentum contrast mechanisms and negative sample queues.

[0117] This system uses a dual-tower model to process short text input questions of varying lengths and long text document data, respectively. It extracts keywords from input questions and fuses them into more accurate query feature vectors, improving the key information of the vectors and making the recall effect better. It also uses soft and hard mining data mining methods to process the impact of difficult samples in negative samples on the loss, improving the training stability. By optimizing the embedding vector and similarity calculation method, the accuracy and efficiency of recall can be improved. This application can effectively perform comparative learning training in specific fields with limited hardware resources, and improve the relevance of short text input questions and long text document data.

[0118] In one possible implementation of the present application, the document processing module includes:

[0119] Data normalization module, which processes document data uniformly into unstructured document data;

[0120] The data cleaning module cleans the document data and removes irrelevant information from the document data;

[0121] Document segmentation module, which divides document data into manageable units;

[0122] The weight adjustment module uses the pre-trained model to extract document keywords and adjusts the weights of document keywords based on the proper noun library.

[0123] In one possible implementation of the present application, the dual-tower model includes a query tower and a document tower. The query tower is used to process input questions and perform feature fusion on input query keywords to generate a query feature vector; the document tower is used to process document data and generate a document feature vector.

[0124] In one possible implementation of the present application, the negative sample processing module includes:

[0125] The soft mining module adjusts the weight of negative samples by adjusting the loss function of negative samples;

[0126] The hard mining module selects hard and easy samples from negative samples.

[0127] In one implementable method of the present application, the optimization module includes a MoCo module, which uses a momentum contrast mechanism and a negative sample queue to increase the batch size of contrastive learning, updates the parameters of the query encoder through back propagation, and updates the parameters of the momentum encoder through a momentum update mechanism, so that the representation of the momentum encoder is more consistent with the representation of the query encoder.

[0128] In one possible implementation of the present application, the comparative learning training system for recalling military documents based on user questions may also include an evaluation and feedback module, which uses an independent test set to evaluate the performance of the model, such as recall rate, accuracy, etc.

[0129] Figure 5 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0130] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the implementation methods of this specification.

[0131] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0132] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0133] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0134] The bus 1050 includes pathways that transmit information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0135] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the implementation scheme of this specification, and does not necessarily include all the components shown in the figure.

[0136] The electronic device of the above-mentioned embodiment is used to implement the corresponding comparative learning training method for recalling military documents based on user questions in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method implementation method, which will not be repeated here.

[0137] Based on the same inventive concept, corresponding to any of the above-mentioned implementation methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the comparative learning training method for recalling military documents based on user questions as described in any of the above implementation methods.

[0138] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0139] The computer instructions stored in the storage medium of the above-mentioned implementation are used to enable the computer to execute the comparative learning training method for recalling military documents based on user questions as described in any of the above implementations, and have the beneficial effects of the corresponding method implementation methods, which will not be repeated here.

[0140] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. This narrative style of the specification is merely for the sake of clarity, and those skilled in the art should take the specification as a whole. Under the concept of the present application, the above embodiments or the technical features in different embodiments may be appropriately combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0141] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power / ground connections to the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present application will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0142] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0143] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A contrastive learning training method for recalling military documents based on user questions, characterized in that: include: Preprocess document data, extract document keywords, and generate document features; Extract query keywords from input questions and generate query features; Based on the input question, construct a positive sample pair and a negative sample pair, the positive sample pair includes the input question and a corresponding positive sample, and the negative sample pair includes the input question and a corresponding negative sample; Using a double-tower model to process the query feature and the document feature respectively, generating a query feature vector and a document feature vector; Calculating the similarity between the query feature vector and the document feature vector, and assigning similarity labels to the positive sample and the negative sample; Performing soft mining and hard mining processing on the negative samples; Use momentum contrast mechanism and negative sample queue to increase the batch size of contrastive learning.

2. The contrastive learning training method for recalling military documents based on user questions according to claim 1 is characterized in that: The preprocessing of the document data, extracting document keywords, and generating document features includes: uniformly processing the document data into unstructured document data; Cleaning the document data to remove irrelevant information in the document data; dividing the document data into manageable units; The document keywords are extracted using a pre-trained model, and the weights of the document keywords are adjusted based on a proper noun library.

3. The contrastive learning training method for recalling military documents based on user questions according to claim 1 is characterized in that: The dual-tower model includes a query tower and a document tower. The query tower is used to process the input question and perform feature fusion on the input query keywords to generate the query feature vector; the document tower is used to process the document data to generate the document feature vector.

4. The contrastive learning training method for recalling military documents based on user questions according to claim 1 is characterized in that: The calculating the similarity between the query feature vector and the document feature vector and assigning similarity labels to the positive sample and the negative sample includes: The similarity between the query feature vector and the document feature vector is calculated using cosine similarity. The expression of the cosine similarity Sim is: Sim=Normalize(E q )·Normalize(E d ); Among them, E q represents the query feature vector, E d represents the document feature vector, and Noemalize represents the normalization operation of the vector; The similarity label between the input question and the positive sample is 1, and the similarity label between the input question and the negative sample is 0.

5. The contrastive learning training method for recalling military documents based on user questions according to claim 1 is characterized in that: The performing soft mining and hard mining processing on the negative samples includes: Using the hard mining to filter out difficult samples and easy samples from the negative samples; When the similarity between the difficult sample and the input question is greater than a first threshold, it is determined as a positive sample and is not added to the loss of the negative sample; When the similarity between the easy sample and the input question is less than a second threshold, there is no need to continue training, and the negative sample is not added to the loss; The soft mining adjusts the weight of the negative sample by adjusting the loss function of the negative sample.

6. The contrastive learning training method for recalling military documents based on user questions according to claim 5 is characterized in that: The expression of the loss of the negative sample is: in, represents the loss of the i-th negative sample in batch N, BCE represents the binary cross entropy loss function, represents the similarity between the i-th negative sample in batch N and the input question; t hard represents the first threshold, t easy represents the second threshold; The adjusting the weight of the negative sample comprises introducing a weight coefficient; The weight coefficient W i The expression is: Here, γ represents the exponential factor.

7. The contrastive learning training method for recalling military documents based on user questions according to claim 1 is characterized in that: The use of momentum contrast mechanism and negative sample queue to increase the batch size of contrastive learning includes: The negative sample queue is used to store negative sample representations in a previous batch, and the negative sample representations are used as negative samples of a current batch; The momentum comparison mechanism uses a momentum encoder and a query encoder; updating the parameters of the query encoder by back-propagation, and updating the parameters of the momentum encoder by a momentum update mechanism, so that the representation of the momentum encoder is more consistent with the representation of the query encoder; The expression of the momentum update mechanism is: i k ←mθ k +(1-m)θ q ; Among them, θ k represents the parameters of the momentum encoder, θ q represents the parameters of the query encoder; m represents the momentum factor, and its value range is [0,1).

8. A contrastive learning training system for recalling military documents based on user questions, characterized in that: include: The document processing module is used to pre-process the document data, extract document keywords, and generate document features; A query processing module is used to extract query keywords from input questions and generate query features; A sample pair construction module, used to construct a positive sample pair and a negative sample pair based on the input question, wherein the positive sample pair includes the input question and a corresponding positive sample, and the negative sample pair includes the input question and a corresponding negative sample; A dual-tower model, used to process the query feature and the document feature respectively to generate a query feature vector and a document feature vector; A similarity calculation module, used to calculate the similarity between the query feature vector and the document feature vector, and assign similarity labels to the positive sample and the negative sample; A negative sample processing module, used for performing soft mining and hard mining processing on the negative samples; An optimization module for increasing the batch size of contrastive learning using momentum contrast mechanisms and negative sample queues.

9. An electronic device, characterized in that: include: Processor and memory; The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images