Recall model training method, data search method and related devices and program products

Abstract

The present disclosure provides a recall model training method, a data search method, and related devices and program products, and relates to the fields of artificial intelligence technology such as deep learning, model training, and data search. The method includes: performing random masking on the corpus content of the real business corpus, and using the obtained masked corpus and the corresponding real business corpus to construct a positive sample, and obtaining a first negative sample by random negative sampling; using the first training sample constructed based on the positive sample and the first negative sample to train the initial recall model, and performing multiple rounds of enhanced training operations on the recall model in training: determining the wrong results that are located after the last correct result and before a preset number in the recall result sequence returned by the recall model in training for the test corpus in the current round as the second negative sample in the current round; using the second training sample constructed based on the first training sample and the second negative sample to perform enhanced training on the recall model in training.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, specifically to the field of artificial intelligence technology such as deep learning, model training, and data search, and in particular to a recall model training method and a data search method, as well as corresponding devices, electronic devices, computer-readable storage media, and computer program products. Background Art

[0002] Enterprises and government agencies accumulate a large amount of structured data over the years. In the subsequent use process, they often hope to use search engines to achieve accurate searches conveniently and quickly, thereby improving business efficiency and quality.

[0003] The search process can usually be divided into two stages: recall and sorting. How to improve the accuracy of the recall results and return a suitable number of accurate results with good relevance to the user's search questions through subsequent sorting is an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0004] The embodiments of the present disclosure propose a recall model training method, a data search method, and corresponding supporting devices, electronic devices, computer-readable storage media, and computer program products.

[0005] In a first aspect, an embodiment of the present disclosure proposes a recall model training method, comprising: performing random masking processing on the corpus content of a real business corpus to obtain a masked corpus; constructing a positive sample using the masked corpus and the corresponding real business corpus, and obtaining a first negative sample by performing random negative sampling on the real business corpus; training an initial recall model using a first training sample constructed based on the positive sample and the first negative sample to obtain a training recall model; repeatedly performing multiple rounds of preset enhanced training operations on the training recall model until the training recall model that meets the preset training objectives is output as a target recall model: determining a preset number of erroneous recall results that are located after the last correct result in the recall result sequence returned by the training recall model in the current round for the test corpus as the second negative sample in the current round; and performing enhanced training on the training recall model in the current round using the second training sample in the current round constructed based on the first training sample and the second negative sample in the current round.

[0006] In a second aspect, an embodiment of the present disclosure proposes a recall model training device, including: a random mask processing unit, configured to obtain a masked corpus by performing random masking on the corpus content of a real business corpus; a positive sample and a first negative sample construction unit, configured to construct a positive sample using the masked corpus and the corresponding real business corpus, and obtain a first negative sample by performing random negative sampling on the real business corpus; a simple training unit, configured to train an initial recall model using a first training sample constructed based on the positive sample and the first negative sample, to obtain a recall model in training; The reinforcement training unit is configured to repeatedly perform multiple rounds of preset reinforcement training operations on the training recall model until the training recall model that meets the preset training objectives is output as a target recall model: the first preset number of erroneous recall results located after the last correct result in the recall result sequence returned by the training recall model in the current round for the test corpus are determined as the second negative samples in the current round; and the training recall model in the current round is reinforced with the second training sample in the current round constructed based on the first training sample and the second negative sample in the current round.

[0007] In the third aspect, an embodiment of the present disclosure proposes a data search method, including: obtaining an input question to be searched; inputting the question to be searched into a preset target recall model to obtain an output recall result sequence; wherein the target recall model is obtained according to the recall model training method described in the first aspect; sorting the recall result sequence, and using the sorted recall results as the search results corresponding to the question to be searched.

[0008] In the fourth aspect, an embodiment of the present disclosure proposes a data search device, comprising: a question-to-be-searched acquisition unit, configured to acquire an input question-to-be-searched; a result recall unit, configured to input the question-to-be-searched into a preset target recall model, and obtain an output recall result sequence; wherein the target recall model is obtained according to the recall model training device described in the second aspect; a sorting unit, configured to sort the recall result sequence, and use the sorted recall result as the search result corresponding to the question-to-be-searched.

[0009] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, it can implement the recall model training method described in the first aspect or the data search method described in the third aspect.

[0010] In a sixth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement the recall model training method described in the first aspect or the data search method described in the third aspect when executed.

[0011] In a seventh aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the steps of the recall model training method as described in the first aspect or the steps of the data search method as described in the third aspect.

[0012] The recall model training scheme provided by the disclosed embodiment first obtains masked corpus by randomly masking real business corpus, so as to construct training samples suitable for self-supervised training based on the masked corpus and the real business corpus without labeling. Secondly, the initial recall model is subjected to simple training in the first stage by first using a simpler first negative sample and a first training sample constructed by a positive sample, so that the obtained recall model in training can distinguish between correct results and obvious erroneous results. Then, the erroneous results before a preset number of erroneous results located after the last correct result in the recall result sequence output by the recall model in training are selected as more difficult second negative samples, and the second training sample is used together with the first training sample to obtain a second training sample to perform enhanced training on the recall model in training in the second stage, so as to further improve the recall model's ability to distinguish between correct results and suspected erroneous results, so that the multiple results recalled by the target recall model obtained in the end are more reasonably sorted and have fewer erroneous results.

[0013] The data search solution provided by the embodiment of the present disclosure, after obtaining the question to be searched input by the user, first uses the target recall model trained by the above-mentioned recall model training scheme to recall the recall result sequence corresponding to the question to be searched, that is, uses the ability of the target recall model to obtain accurate and reasonably sorted recall results, and then re-sorts the recall result sequence to obtain the final search result sequence that can be used to present to the user, thereby improving the accuracy of each search result in the search result sequence and reducing the number of incorrect answers.

[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0016] Figure 1is an exemplary system architecture in which the present disclosure may be applied;

[0017] Figure 2 A flowchart of a recall model training method provided in an embodiment of the present disclosure;

[0018] Figure 3 A flow chart of a method for replacing an old model with a new model provided in an embodiment of the present disclosure;

[0019] Figure 4 A flowchart of a data search method provided by an embodiment of the present disclosure;

[0020] Figure 5 A structural block diagram of a recall model training device provided in an embodiment of the present disclosure;

[0021] Figure 6 A structural block diagram of a data search device provided in an embodiment of the present disclosure;

[0022] Figure 7 A schematic diagram of the structure of an electronic device suitable for executing a recall model training method and / or a data search method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0024] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0025] Figure 1 An exemplary system architecture 100 is shown to which the recall model training method, data search method, and corresponding supporting apparatus, electronic device, computer-readable storage medium, and computer program product of the present application can be applied.

[0026] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0027] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various applications for realizing information communication between the terminal devices 101, 102, 103 and server 105 can be installed, such as model training applications, data search applications, instant messaging applications, etc.

[0028] Terminal devices 101, 102, 103 and server 105 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.; when terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.

[0029] The server 105 can provide various services through various built-in applications. For example, a data search application that can provide data search services to users can achieve the following effects when the server 105 runs the data search application: first, the search question transmitted by the user through the terminal device 101, 102, 103 is received through the network 104; then, the search question is input into the preset target recall model to obtain the output recall result sequence; finally, the recall result sequence is sorted and the sorted recall result is used as the search result corresponding to the search question. Furthermore, the server 105 can also return the search result to the corresponding terminal device 101, 102, 103 through the network 104.

[0030] Among them, the target recall model can be trained by the model training application built into the server 105 according to the following steps: first, the masked corpus is obtained by randomly masking the corpus content of the real business corpus; then, the positive sample is constructed using the masked corpus and the corresponding real business corpus, and the first negative sample is obtained by randomly negative sampling the real business corpus; then, the initial recall model is trained using the first training sample constructed based on the positive sample and the first negative sample to obtain the training recall model; next, the preset enhanced training operation is repeatedly performed on the training recall model for multiple rounds until the training recall model that meets the preset training target is output as the target recall model: the wrong recall results that are located after the last correct result and before a preset number in the recall result sequence returned by the training recall model in the current round for the test corpus are determined as the second negative sample in the current round; the training recall model in the current round is enhanced by using the second training sample in the current round constructed based on the first training sample and the second negative sample in the current round.

[0031] Since it takes more computing resources and stronger computing power to train the target recall model, the recall model training method provided in the subsequent embodiments of this application is generally performed by a server 105 with stronger computing power and more computing resources. Correspondingly, the recall model training device is generally also arranged in the server 105. However, it should also be pointed out that when the terminal devices 101, 102, and 103 also have computing power and computing resources that meet the requirements, the terminal devices 101, 102, and 103 can also complete the various operations assigned to the server 105 by the recall model training application installed thereon, and then output the same result as the server 105. Correspondingly, the recall model training device can also be arranged in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104.

[0032] Of course, the server used to train the target recall model can be different from the server that calls the trained target recall model for use. In particular, the target recall model trained by the server 105 can also be used to obtain a lightweight target recall model suitable for being placed in the terminal devices 101, 102, and 103 through model distillation, that is, it is possible to flexibly choose to use the lightweight target recall model in the terminal devices 101, 102, and 103 or to choose to use the more complex target recall model in the server 105 according to the recognition accuracy actually required.

[0033] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0034] Please refer to Figure 2 , Figure 2 A flowchart of a recall model training method provided in an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0035] Step 201: performing random masking on the content of the real business corpus to obtain masked corpus;

[0036] This step is intended to be performed by the execution subject of the recall model training method (e.g. Figure 1 The server 105 shown in the figure performs random masking processing on the corpus content of the real business corpus in the underlying database data from a real business scenario (such as an enterprise scenario or a government agency scenario), thereby obtaining a masked corpus with part of the content obscured after the random masking processing.

[0037] Considering that in actual business, corpus usually contains multiple fields, such as name, address, telephone number, etc., it is possible to obtain masked corpus by randomly masking the field content of at least one field constituting the real business corpus. That is, random masking can be applied to one or more of these fields to obtain masked corpus with different degrees of content masked, for example, it can be used to ensure the privacy and security of data, for example, only the telephone number and address can be masked, and the name can be retained for identification.

[0038] The real business corpus is usually composed of structured data in real business scenarios, and may also be unstructured data in some other cases.

[0039] Step 202: construct a positive sample using the masked corpus and the corresponding real business corpus, and obtain a first negative sample by performing random negative sampling on the real business corpus;

[0040] On the basis of step 201, this step aims to use the masked corpus processed by random masking and the corresponding real business corpus that has not been masked to construct positive samples in a corresponding manner (it can be understood that the masked corpus obtained due to the presence of some field content being masked is used as the problem, and the corresponding real business corpus that has not been masked is used as the corresponding problem result). Then, a simple negative sample is selected from the real business corpus as the first negative sample by random negative sampling.

[0041] The process of obtaining the first negative sample from the real business corpus by random negative sampling may include the following steps:

[0042] 1) Clearly define positive samples: First, the masked corpus and the corresponding real business corpus are clearly defined as positive samples;

[0043] 2) Randomly select text: Randomly select text from real business corpus to form negative samples. You can use random sampling methods and ensure that the negative sample set is large enough to provide diversity in training;

[0044] 3) Label negative samples: Label the selected text as negative samples and ensure that there is a corresponding negative sample for each positive sample. In this way, each positive sample has a corresponding negative sample of a different category.

[0045] 4) Balance positive and negative samples: Make sure your positive and negative samples are roughly balanced, so as to avoid the model being biased towards the majority category during training, which is conducive to improving the performance of the model;

[0046] Considering that simple sampling methods such as random negative sampling are relatively intuitive and easy to implement, they have some limitations. One of the main limitations is that it just randomly selects text from the original data as negative samples, which may cause some negative samples to have some similarities with positive samples, which may not be in line with the actual situation. Therefore, more advanced negative sampling techniques such as generative adversarial networks can also be used to generate more challenging negative samples to further improve the performance of the model.

[0047] Step 203: using the first training sample constructed based on the positive sample and the first negative sample, training the initial recall model to obtain a recall model in training;

[0048] On the basis of step 202, this step aims to perform the first stage of training on the initial recall model by the above-mentioned execution subject using the first training sample constructed based on the positive sample and the first negative sample, and then obtain the training recall model as the result of the first stage of training.

[0049] Step 204: Repeat multiple rounds of preset enhanced training operations on the training recall model until the training recall model that meets the preset training target is output as the target recall model:

[0050] On the basis of step 203, this step aims to have the above-mentioned execution subject train the training recall model in the first stage using the first training sample constructed based on positive samples and simple negative samples, and then perform the second stage of enhanced training by repeatedly executing multiple rounds of enhanced training operations until the training recall model that meets the preset training objectives is output as the target recall model as the second stage training result.

[0051] The condition that the preset training goal is met may include at least one of the following:

[0052] The actual number of iterative training rounds exceeds the preset number of rounds, the proportion of incorrect results in the recall result sequence returned by the recall model for the test corpus during the current training is less than the preset proportion, the actual training time exceeds the preset time, etc.

[0053] The following two steps will describe in detail what steps are performed in each round during the second phase of intensive training:

[0054] Step 2041: Determine a preset number of wrong recall results after the last correct result in the recall result sequence returned by the recall model in training for the test corpus in the current round as the second negative samples in the current round;

[0055] The purpose of this step is to have the above-mentioned execution entity input the test corpus into the latest training recall model in the current round, and determine the first preset number of erroneous recall results after the last correct result in the recall result sequence output by the current training recall model as the second negative sample in the current round.

[0056] It should be understood that the multiple recall results in the recall result sequence are usually arranged according to their relevance and correctness to the question, showing this arrangement trend: high-relevance correct result → low-relevance correct result → high-relevance erroneous result → low-relevance erroneous result, and the first negative sample selected by a simple negative sampling technique such as random negative sampling, often only enables the recall model in training to have the ability to distinguish between correct results and clear erroneous results (i.e., low-relevance erroneous results) after the first stage of training, while the ability to distinguish high-relevance erroneous results is still insufficient. Therefore, this step is to determine the first preset number of erroneous recall results (i.e., high-relevance erroneous results that are ranked high, which can also be understood as erroneous results that are more confusing or more difficult to accurately distinguish) that are located after the last correct result in the recall result sequence output by the recall model in training under the current training round as more difficult negative samples (relative to simple negative samples) or second negative samples (used to distinguish first negative samples).

[0057] It should be noted that the size of the pre-set number should usually gradually decrease with the increase in the number of training rounds. Accordingly, as the number of training rounds increases, the recall model has gradually strengthened its ability to distinguish between correct results and confusing or suspected erroneous results. Therefore, the number of confusing or suspected erroneous results in the recall result sequence output by the latest recall model in subsequent rounds will continue to decrease.

[0058] Step 2042: using the second training sample in the current round constructed based on the first training sample and the second negative sample in the current round, to perform enhanced training on the recall model in the training in the current round.

[0059] On the basis of step 2041, this step aims to have the above-mentioned execution subject use the second training sample in the current round constructed based on the first training sample and the second negative sample in the current round to strengthen the training of the recall model in the training of the current round, so as to gradually improve the recall model's ability to distinguish between correct results and confusing erroneous results through strengthened training.

[0060] Specifically, for large amounts of structured data for enterprises or government agencies, the initial recall model is preferably a semantic recall model based on semantics and using a dual-tower Pair-Wise structure, and uses the Margin Rank Loss, which is more suitable for this scenario, as the loss function to improve the training effect.

[0061] The main goal of Margin Rank Loss is to push the model to learn better rankings by explicitly controlling the margin between positive pairs (related pairs) and negative pairs (irrelevant pairs).

[0062] The recall model training method provided by the embodiment of the present disclosure first obtains a masked corpus by randomly masking the real business corpus, so as to construct a training sample suitable for self-supervised training based on the masked corpus and the real business corpus without labeling. Secondly, the initial recall model is subjected to a first stage of simple training by first using a simpler first negative sample and a first training sample constructed with a positive sample, so that the obtained recall model in training can distinguish between correct results and obvious erroneous results. Then, the erroneous results before a preset number of erroneous results located after the last correct result in the recall result sequence output by the recall model in training are selected as the more difficult second negative samples, and the second training sample is used together with the first training sample to obtain a second training sample to perform a second stage of enhanced training on the recall model in training, so as to further improve the recall model's ability to distinguish between correct results and suspected erroneous results, so that the multiple results recalled by the target recall model obtained in the end are more reasonably sorted and have fewer erroneous results.

[0063] exist Figure 2 Based on the steps in the process 200 shown in the figure, considering that the old target recall model may need to be retrained or fine-tuned due to factors such as changes in the business database data and adjustments to the distribution of the question corpus, in order to accurately evaluate whether the newly trained target recall model can replace the old target recall model, this embodiment also uses Figure 3 A specific implementation is provided, wherein process 300 includes the following steps:

[0064] Step 301: Randomly construct old test corpus based on the real business corpus of the old target recall model before retraining or fine-tuning training;

[0065] In the case where the target recall model needs to be retrained or fine-tuned, this step aims to randomly construct old test corpora based on the real business corpora of the old target recall model before retraining or fine-tuning. That is, the constructed old test corpora should match the performance of the old target recall model and the actual situation of the business data in the base database (such as the structure of each field in the business corpus, the distribution of the problem corpus, etc.).

[0066] Step 302: randomly construct new test corpus based on the real business corpus of the new target recall model after retraining or fine-tuning training;

[0067] On the basis of step 301, this step aims to randomly construct new test corpus based on the real business corpus of the new target recall model after retraining or fine-tuning training by the above-mentioned execution subject. That is, the constructed new test corpus should match the performance required by the new target recall model and the actual situation of the business data in the current base database (such as the structure of each field in the business corpus, the distribution of the problem corpus, etc.).

[0068] Step 303: In response to the recall results of the new target recall model for the old test corpus and the new test corpus being better than the recall results of the old target recall model for the old test corpus and the new test corpus on a preset comparison indicator, the new target recall model is used to replace the old target recall model.

[0069] On the basis of step 301 and step 302, this step aims to determine by the above-mentioned execution subject whether the recall result of the new target recall model for the old test corpus is better than the recall result of the old target recall model for the old test corpus in terms of preset comparison indicators, and determine whether the recall result of the new target recall model for the new test corpus is better than the recall result of the old target recall model for the new test corpus in terms of preset comparison indicators, and only when the recall results of the new target recall model for both the old test corpus and the new test corpus are better than the recall results of the old target recall model for both the old test corpus and the new test corpus in terms of preset comparison indicators, the new target recall model is considered to have better effect than the old target recall model, and therefore the new target recall model can be used to replace the old target recall model.

[0070] That is, this embodiment provides a preferred solution for determining whether the new model can be used to replace the old model through steps 301 to 303, that is, the new target recall model is used to replace the old target recall model only when the recall results of the new target recall model for the old test corpus and the new test corpus are better than the recall results of the old target recall model for the old test corpus and the new test corpus in terms of preset comparison indicators, thereby ensuring that the new target recall model has comprehensive performance leadership.

[0071] On the basis of any of the above embodiments, considering that in the recall stage of providing search services to enterprises and government agencies, the relevant information stored in the database has actually been sorted, so that the number of correct results searched out is often not large, which puts higher requirements on the relevance of the recall results, that is, even if there are fewer erroneous results at this time, they will still be more conspicuous.

[0072] To solve this problem, this embodiment may also utilize a pre-trained filtering model to filter the relevance between questions and results on the recall results included in the recall result sequence output by the target recall model.

[0073] Among them, the filtering model can be trained by using annotated samples that are annotated with information about the matching method of the question corpus and the recall results that constitute the second training sample. The annotation information contained in the annotated sample may include: first annotation information of the correct matching method of the field to which the question belongs in the question corpus, and second annotation information of the actual matching method corresponding to each recall result. The matching method includes: exact matching, fuzzy matching, or inclusion class matching. That is, by matching each recall result in the recall result sequence returned by the target recall model with the matching method of the field to which the question belongs in the corresponding input question, it is possible to filter based on relevance, and then filter out some wrong answers with low relevance. For example, in a search question for a name, since the matching method of the field to which the name belongs often adopts an exact matching method, some recall results that do not belong to an exact match in the recall result can be filtered out accordingly.

[0074] Specifically, for large amounts of structured data for enterprises or government agencies, the filtering model preferably adopts a deep learning model with a single-tower Point-Wise structure, and uses Dice Loss as the loss function for characterizing positive samples, which is more suitable for this scenario.

[0075] Dice Loss is a method commonly used in fields such as image segmentation and medical image processing to characterize the F1 score of positive samples. The goal of Dice Loss is to measure the degree of overlap between the predicted results and the actual labels in order to better evaluate the matching degree of the positive samples. This embodiment can better filter out erroneous results with low relevance by applying it to information filtering.

[0076] The above embodiments describe how to train a target recall model from various aspects. In order to highlight the effect of the trained target recall model from the actual usage scenario as much as possible, the present disclosure also specifically provides a solution for using the trained target recall model to solve practical problems. A data search method includes the following steps:

[0077] Step 401: Obtain input questions to be searched;

[0078] Step 402: input the question to be searched into a preset target recall model to obtain an output recall result sequence;

[0079] Step 403: sorting the recall result sequence, and using the sorted recall results as search results corresponding to the question to be searched.

[0080] An implementation method including but not limited to may be:

[0081] The recall result sequence is sorted using a pre-trained sorting model; wherein the sorting model can be trained based on labeled samples in which the degree of relevance between each recall result in the recall result sequence returned by the test question and the test question is labeled.

[0082] This embodiment provides a specific usage scheme through steps 401 to 403, which first applies the target recall model to obtain the recall result sequence of the question to be searched, and then finally obtains the corresponding search result by sorting the recall result sequence. That is, after obtaining the question to be searched input by the user, firstly use the target recall model trained by the above recall model training scheme to recall the recall result sequence corresponding to the question to be searched, that is, use the ability of the target recall model to obtain accurate and reasonably sorted recall results, and then re-sort the recall result sequence to obtain the final search result sequence that can be used to present to the user, thereby improving the accuracy of each search result in the search result sequence and reducing the number of wrong answers.

[0083] To deepen understanding, the present disclosure also summarizes the contents of the above embodiments and provides a relatively complete implementation scheme, which is divided into the following parts:

[0084] 1. Construction of training samples:

[0085] a) Direct mock (mask) of the base database corpus: According to the meaning of each field, the corpus is randomly mocked by field, without guaranteeing the authenticity of the corpus;

[0086] b)i. Positive samples: For each mock database data, randomly combine the fields as a query (question, query statement, query word), and the database data is used as a positive sample;

[0087] ii. Simple negative samples: random negative sampling;

[0088] iii. Difficult negative samples: During the training process, based on the TopK results of the previous model recall, negative samples are identified according to the term (field) matching degree and added to the next round of model iteration (this process requires attention to ensure the negativity of the samples).

[0089] c) Training samples of the filtering model: For queries constructed with random fields, the recall model is used to obtain the TopK results. The correctness of the recall results is determined based on the matching method of the fields that constitute the query (such as =, like, match, etc.), so as to distinguish between positive and negative samples (to ensure the accuracy of positive and negative samples).

[0090] 2. Model framework selection and loss function selection

[0091] a) The recall model uses the Pair-Wise dual-tower model based on large model pre-training, and the loss function uses the Margin Rank Loss used for sorting;

[0092] b) The filtering model uses a Point-wise single-tower model based on large model pre-training, and the loss function uses Dice Loss for characterizing the F1 of positive samples;

[0093] c) The pre-trained model can ensure a certain degree of generalization.

[0094] 3. Model self-training solution

[0095] a) Thanks to the fact that recall and filtering models can be trained in a self-supervised manner, the models can be self-sampled and retrained / fine-tuned at the customer site;

[0096] b) Model evaluation plan: last evaluation query + this new evaluation query; each query is randomly constructed, and the model effect is also evaluated based on the matching of the fields in the query. When the effects of the last evaluation query and the new evaluation query are better, the model is replaced;

[0097] c) Through comparative analysis of real queries and recall results, the fields in the query are identified, abstracted into templates and aggregated, and the query templates are used for overall effect iteration (the usage is similar to that of randomly constructed queries), making the model more inclined to solve user problems.

[0098] By applying the technical solution provided in this embodiment, compared with the prior art, the following technical effects are achieved:

[0099] 1) Based on the structured field definition, self-supervised semantic recall model training can be achieved, saving the cost of manual labeling and enabling fast cold start;

[0100] 2) The model can achieve self-learning by continuously sampling queries and underlying databases to adapt to changing scenarios and requirements;

[0101] 3) After adding the filtering model, the strong relevance between the search results and the questions can be further guaranteed.

[0102] Further references Figure 5 and Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides a recall model training device embodiment and a data search device embodiment, the recall model training device embodiment and Figure 2 The recall model training method embodiment shown in FIG. 1 corresponds to the data search device embodiment. Figure 4 The above device can be applied to various electronic devices.

[0103] like Figure 5 As shown, the recall model training device 500 of this embodiment may include: a random mask processing unit 501, a positive sample and first negative sample construction unit 502, a simple training unit 503, and an enhanced training unit 504. Among them, the random mask processing unit 501 is configured to obtain a masked corpus by performing random masking on the corpus content of the real business corpus; the positive sample and first negative sample construction unit 502 is configured to construct a positive sample using the masked corpus and the corresponding real business corpus, and obtain a first negative sample by performing random negative sampling on the real business corpus; the simple training unit 503 is configured to train the initial recall model using the first training sample constructed based on the positive sample and the first negative sample to obtain a recall model in training; the enhanced training unit 504 , configured to repeatedly perform multiple rounds of preset reinforcement training operations on the training recall model until the training recall model that meets the preset training objectives is output as a target recall model: the first preset number of erroneous recall results located after the last correct result in the recall result sequence returned by the training recall model in the current round for the test corpus are determined as the second negative samples in the current round; and the training recall model in the current round is reinforced with the second training sample in the current round constructed based on the first training sample and the second negative sample in the current round.

[0104] In this embodiment, in the recall model training device 500, the specific processing of the random mask processing unit 501, the positive sample and the first negative sample construction unit 502, the simple training unit 503, and the enhanced training unit 504 and the technical effects thereof can be referred to respectively. Figure 2 The relevant descriptions of steps 201 - 204 in the corresponding embodiment are not repeated here.

[0105] In some optional implementations of this embodiment, the random mask processing unit 501 may be further configured to:

[0106] By randomly masking the field content of at least one field constituting the real business corpus, a masked corpus is obtained.

[0107] In some optional implementations of this embodiment, the real business corpus includes corpus consisting of structured data in real business scenarios.

[0108] In some optional implementations of this embodiment, the condition that the preset training goal is met includes at least one of the following:

[0109] The actual number of rounds of iterative training exceeds the preset number of rounds, the proportion of incorrect results in the recall result sequence returned by the recall model for the test corpus during the current training is less than the preset proportion, and the actual training time exceeds the preset time.

[0110] In some optional implementations of this embodiment, the size of the pre-preset number gradually decreases as the number of training rounds increases.

[0111] In some optional implementations of this embodiment, the initial recall model includes a semantic recall model based on semantics and using a dual-tower Pair-Wise structure, and uses a margin rank loss as a loss function.

[0112] In some optional implementations of this embodiment, the recall model training device 500 may further include:

[0113] An old test corpus construction unit is configured to randomly construct old test corpus based on the real business corpus of the old target recall model before the retraining or fine-tuning training in response to retraining or fine-tuning the target recall model;

[0114] A new test corpus construction unit is configured to randomly construct a new test corpus based on the real business corpus of the new target recall model after retraining or fine-tuning training;

[0115] The model replacement unit is configured to replace the old target recall model with the new target recall model in response to the recall results of the new target recall model for the old test corpus and the new test corpus being better than the recall results of the old target recall model for the old test corpus and the new test corpus on a preset comparison indicator.

[0116] In some optional implementations of this embodiment, the recall model training device 500 may further include:

[0117] The filtering unit is configured to use a pre-trained filtering model to filter the correlation between questions and results of the recall results contained in the recall result sequence output by the target recall model; wherein the filtering model is trained using labeled samples that are annotated with information on the matching method of the question corpus and the recall results constituting the second training sample, and the labeled information contained in the labeled samples includes: first labeling information on the correct matching method of the field to which the question of the question corpus belongs, and second labeling information on the actual matching method corresponding to each recall result, and the matching methods include: exact matching, fuzzy matching or inclusion class matching.

[0118] In some optional implementations of this embodiment, the filtering model includes a deep learning model using a single-tower Point-Wise structure, and uses Dice Loss for characterizing positive samples as a loss function.

[0119] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The recall model training apparatus provided by this embodiment first obtains masked corpus by randomly masking the real business corpus, so as to construct training samples suitable for self-supervised training based on the masked corpus and the real business corpus without labeling. Secondly, the initial recall model is subjected to a first stage of simple training by first using a simpler first negative sample and a first training sample constructed by a positive sample, so that the obtained recall model in training can distinguish between correct results and obvious erroneous results. Then, the erroneous results before a preset number of erroneous results located after the last correct result in the recall result sequence output by the recall model in training are selected as the more difficult second negative samples, and the second training samples are constructed by using the second negative samples and the first training samples to perform the second stage of enhanced training on the recall model in training, so as to further improve the recall model's ability to distinguish between correct results and suspected erroneous results, so that the multiple results recalled by the target recall model obtained in the end are more reasonably sorted and have fewer erroneous results.

[0120] like Figure 6 As shown, the data search device 600 of this embodiment may include: a to-be-searched question acquisition unit 601, a result recall unit 602, and a sorting unit 603. The to-be-searched question acquisition unit 601 is configured to acquire an input to-be-searched question; the result recall unit 602 is configured to input the to-be-searched question into a preset target recall model to obtain an output recall result sequence; the target recall model is based on Figure 5 The recall model training device 500 shown is obtained; a sorting unit 603 is configured to sort the recall result sequence and use the sorted recall result as the search result corresponding to the question to be searched.

[0121] In this embodiment, in the data search device 600, the specific processing of the to-be-searched question acquisition unit 601, the result recall unit 602, and the sorting unit 603 and the technical effects thereof can be respectively Figure 4 The relevant descriptions of steps 401 to 403 in the method embodiment shown are not repeated here.

[0122] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The data search apparatus provided by this embodiment, after obtaining the question to be searched input by the user, first uses the target recall model trained by the above-mentioned recall model training scheme to recall the recall result sequence corresponding to the question to be searched, that is, uses the ability of the target recall model to obtain accurate and reasonably ordered recall results, and then reorders the recall result sequence to obtain the final search result sequence that can be used to present to the user, thereby improving the accuracy of each search result in the search result sequence and reducing the number of incorrect answers.

[0123] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the recall model training method and / or data search method described in any of the above embodiments when executing.

[0124] According to an embodiment of the present disclosure, the present disclosure also provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the recall model training method and / or data search method described in any of the above embodiments when executed.

[0125] An embodiment of the present disclosure provides a computer program product, which, when executed by a processor, can implement the steps of the recall model training method and / or the steps of the data search method described in any of the above embodiments.

[0126] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0127] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0128] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0129] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the recall model training method and / or the data search method. For example, in some embodiments, the recall model training method and / or the data search method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the recall model training method and / or the data search method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the recall model training method and / or the data search method in any other suitable manner (eg, by means of firmware).

[0130] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0132] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0133] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0134] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0135] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0136] The technical solution of the disclosed embodiment first obtains masked corpus by randomly masking the real business corpus, so as to construct training samples suitable for self-supervised training based on the masked corpus and the real business corpus without annotation. Secondly, a simple negative sampling method is first used to select a simpler first negative sample and a positive sample to construct a first training sample, and a first stage of simple training is performed on the initial recall model so that the recall model in training has the ability to distinguish correct results from obviously erroneous results. Then, a second negative sample is further selected as a more difficult second negative sample that is located after the last correct result in the recall result sequence output by the recall model in training and a preset number of erroneous results are constructed together with the first training sample to obtain a second training sample, and the second training sample is used to perform a second stage of intensive training on the recall model in training, so as to further improve the recall model's ability to distinguish correct results from suspected erroneous results, so that the multiple results recalled by the target recall model obtained in the end are more reasonably sorted and have fewer erroneous results.

[0137] Subsequently, after obtaining the question to be searched input by the user, the target recall model trained by the above-mentioned recall model training scheme is first used to recall the recall result sequence corresponding to the question to be searched, and then the capability of the target recall model is used to make each recall result in the recall result sequence accurate and reasonably sorted. Then, the recall result sequence is re-sorted to obtain the final search result sequence that can be presented to the user, thereby improving the accuracy of each search result in the search result sequence and the number of incorrect answers.

[0138] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0139] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A recall model training method, comprising: By randomly masking the content of the real business corpus, the masked corpus is obtained; Constructing a positive sample using the masked corpus and the corresponding real business corpus, and obtaining a first negative sample by performing random negative sampling on the real business corpus; Training an initial recall model using a first training sample constructed based on the positive sample and the first negative sample to obtain a training recall model; Repeating multiple rounds of preset reinforcement training operations on the training recall model until the training recall model that meets the preset training target is output as the target recall model: Determine the first preset number of incorrect recall results after the last correct result in the recall result sequence returned by the recall model in training for the test corpus in the current round as the second negative samples in the current round, and arrange the multiple recall results in the recall result sequence according to their relevance to the question and correctness, in the following order: correct results with high relevance, correct results with low relevance, incorrect results with high relevance, and incorrect results with low relevance; The recall model in training in the current round is enhanced by using the second training sample in the current round constructed based on the first training sample and the second negative sample in the current round.

2. The method according to claim 1, wherein The masked corpus is obtained by randomly masking the corpus content of the real business corpus, including: The masked corpus is obtained by randomly masking the field content of at least one field constituting the real business corpus.

3. The method according to claim 1, wherein The real business corpus includes corpus composed of structured data in real business scenarios.

4. The method according to claim 1, wherein The conditions for the preset training goal to be met include at least one of the following: The actual number of iterative training rounds exceeds the preset number of rounds, the proportion of incorrect results in the recall result sequence returned by the recall model for the test corpus during training is less than the preset proportion, and the actual training time exceeds the preset time.

5. The method according to claim 1, wherein The size of the pre-set number gradually decreases as the number of training rounds increases.

6. The method according to claim 1, wherein The initial recall model includes a semantic recall model based on semantics and adopting a dual-tower Pair-Wise structure, and adopts a margin rank loss as a loss function.

7. The method according to claim 1, further comprising: In response to retraining or fine-tuning the target recall model, randomly constructing old test corpus based on the real business corpus of the old target recall model before the retraining or fine-tuning; Randomly construct new test corpus based on the real business corpus of the new target recall model after the retraining or fine-tuning training; In response to the recall results of the new target recall model for the old test corpus and the new test corpus being better than the recall results of the old target recall model for the old test corpus and the new test corpus on a preset comparison indicator, the new target recall model is used to replace the old target recall model.

8. The method according to any one of claims 1 to 7, further comprising: A pre-trained filtering model is used to filter the correlation between questions and results of the recall results contained in the recall result sequence output by the target recall model; wherein the filtering model is trained using labeled samples that are obtained by annotating information on the matching method of the question corpus and the recall results constituting the second training sample, and the labeling information contained in the labeled samples includes: first labeling information on the correct matching method of the field to which the question of the question corpus belongs, and second labeling information on the actual matching method corresponding to each of the recall results, and the matching method includes: exact matching, fuzzy matching or inclusion class matching.

9. The method according to claim 8, wherein The filtering model includes a deep learning model using a single-tower Point-Wise structure, and adopts Dice Loss for characterizing positive samples as a loss function.

10. A data search method comprising: Get the input question to be searched; Inputting the question to be searched into a preset target recall model to obtain an output recall result sequence; wherein the target recall model is obtained based on the recall model training method according to any one of claims 1 to 9; The recall result sequence is sorted, and the sorted recall results are used as search results corresponding to the question to be searched.

11. The method according to claim 10, wherein: The sorting process of the recall result sequence includes: The recall result sequence is sorted using a pre-trained sorting model; wherein the sorting model is trained based on labeled samples in which the degree of relevance between each recall result in the recall result sequence returned by the test question and the test question is labeled.

12. A recall model training device comprising: The random mask processing unit is configured to perform random masking on the content of the real business corpus to obtain masked corpus; A positive sample and first negative sample construction unit is configured to construct a positive sample using the masked corpus and the corresponding real business corpus, and obtain a first negative sample by performing random negative sampling on the real business corpus; A simple training unit is configured to train an initial recall model using a first training sample constructed based on the positive sample and the first negative sample to obtain a training recall model; The reinforcement training unit is configured to repeatedly perform multiple rounds of preset reinforcement training operations on the training recall model until the training recall model that meets the preset training target is output as the target recall model: Determine the first preset number of incorrect recall results after the last correct result in the recall result sequence returned by the recall model in training for the test corpus in the current round as the second negative samples in the current round, and arrange the multiple recall results in the recall result sequence according to their relevance to the question and correctness, in the following order: correct results with high relevance, correct results with low relevance, incorrect results with high relevance, and incorrect results with low relevance; The recall model in training in the current round is enhanced by using the second training sample in the current round constructed based on the first training sample and the second negative sample in the current round.

13. The device according to claim 12, wherein The random mask processing unit is further configured to: The masked corpus is obtained by randomly masking the field content of at least one field constituting the real business corpus.

14. The device according to claim 12, wherein The real business corpus includes corpus composed of structured data in real business scenarios.

15. The device according to claim 12, wherein The conditions for the preset training goal to be met include at least one of the following: The actual number of iterative training rounds exceeds the preset number of rounds, the proportion of incorrect results in the recall result sequence returned by the recall model for the test corpus during training is less than the preset proportion, and the actual training time exceeds the preset time.

16. The device according to claim 12, wherein The size of the pre-set number gradually decreases as the number of training rounds increases.

17. The device according to claim 12, wherein The initial recall model includes a semantic recall model based on semantics and adopting a dual-tower Pair-Wise structure, and adopts a margin rank loss as a loss function.

18. The apparatus according to claim 12, further comprising: an old test corpus construction unit, configured to, in response to retraining or fine-tuning the target recall model, randomly construct old test corpus based on the real business corpus of the old target recall model before the retraining or fine-tuning; A new test corpus construction unit is configured to randomly construct new test corpus based on the real business corpus of the new target recall model after the retraining or fine-tuning training; The model replacement unit is configured to replace the old target recall model with the new target recall model in response to the recall results of the new target recall model for the old test corpus and the new test corpus being better than the recall results of the old target recall model for the old test corpus and the new test corpus on a preset comparison indicator.

19. The apparatus according to any one of claims 12 to 18, further comprising: The filtering unit is configured to use a pre-trained filtering model to filter the correlation between questions and results of the recall results contained in the recall result sequence output by the target recall model; wherein the filtering model is trained using labeled samples that have been annotated with information on the matching method of the question corpus and the recall results constituting the second training sample, and the labeling information contained in the labeled samples includes: first labeling information on the correct matching method of the field to which the question of the question corpus belongs, and second labeling information on the actual matching method corresponding to each of the recall results, and the matching method includes: exact matching, fuzzy matching or inclusion class matching.

20. The device according to claim 19, wherein The filtering model includes a deep learning model using a single-tower Point-Wise structure, and adopts Dice Loss for characterizing positive samples as a loss function.

21. A data search device comprising: A question-to-be-searched obtaining unit is configured to obtain an input question-to-be-searched; A result recall unit is configured to input the question to be searched into a preset target recall model to obtain an output recall result sequence; wherein the target recall model is obtained based on the recall model training device according to any one of claims 12 to 20; The sorting unit is configured to sort the recall result sequence and use the sorted recall result as the search result corresponding to the question to be searched.

22. The device according to claim 21, wherein The sorting unit includes a sorting processing subunit configured to sort the recall result sequence, and the sorting processing subunit is further configured to: The recall result sequence is sorted using a pre-trained sorting model; wherein the sorting model is trained based on labeled samples in which the degree of relevance between each recall result in the recall result sequence returned by the test question and the test question is labeled.

23. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the recall model training method described in any one of claims 1-9 and / or the data search method described in any one of claims 10-11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the recall model training method described in any one of claims 1 to 9 and / or the data search method described in any one of claims 10 to 11.

25. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the recall model training method according to any one of claims 1 to 9 and / or the steps of the data search method according to any one of claims 10 to 11.

Images