Model training method and device, data processing method and device, equipment, storage medium and program product

Abstract

The invention provides a model training method and device, a data processing method and device, equipment, a storage medium and a program product. The method comprises the steps of determining recall scores of a plurality of candidate data samples in a database relative to a first query data sample through a pre-trained recall model; determining a plurality of recall data samples from the plurality of candidate data samples based on the recall scores; determining a sorting score of each recall data sample relative to the first query data sample through a pre-trained sorting model; fusing the recall score and the sorting score to obtain a fusion score of each recall data sample; determining a first loss value based on the recall score and the fusion score; and based on the first loss value, updating parameters of the recall model to obtain a fine-tuned recall model. According to the method and the device, the retrieval accuracy of the recall model after fine tuning can be improved on the basis of keeping the advantages of structural simplicity and low delay of the original recall model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method, data processing method, device, equipment, storage medium and program product. Background Art

[0002] In the model training method in the related art, the recall model is trained by calculating the similarity score between the query data and the candidate data. Although the structure is simple and the reasoning is fast, there is a problem of inaccurate calculation of the similarity score (recall score), which makes it difficult to meet the high-precision retrieval requirements. Although the two-stage cascade structure of "recall + rerank" (that is, first preliminarily screening the candidate set through the recall model, and then using the ranking model for refined scoring) can improve the retrieval effect, the need to train and maintain two independent models (recall model and ranking model) increases the difficulty of deployment and optimization, and the two-stage cascade increases the consumption of computing resources, resulting in an increase in the overall reasoning time, making it difficult to adapt to application scenarios with high real-time requirements. Summary of the Invention

[0003] The embodiments of the present application provide a model training method, data processing method, apparatus, equipment, storage medium and program product, which can improve the retrieval accuracy of the fine-tuned recall model while maintaining the simplicity of the original recall model structure and the low latency advantages.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present invention provides a model training method, which includes:

[0006] Determining, by a pre-trained recall model, recall scores of a plurality of candidate data samples in the database relative to the first query data sample;

[0007] determining a plurality of recalled data samples from the plurality of candidate data samples based on the recall scores;

[0008] Determining a ranking score of each of the recalled data samples relative to the first query data sample using a pre-trained ranking model;

[0009] fusing the recall score and the ranking score to obtain a fusion score for each of the recalled data samples;

[0010] determining a first loss value based on the recall score and the fusion score;

[0011] Based on the first loss value, the parameters of the recall model are updated to obtain a fine-tuned recall model.

[0012] This embodiment of the present application provides a data processing method, the method comprising:

[0013] Get query data;

[0014] Determining screening scores of multiple candidate data relative to the query data using a distillation model, wherein the distillation model is a fine-tuned recall model obtained by the model training method provided in an embodiment of the present application;

[0015] Target data for responding to the query data is determined from the plurality of candidate data based on the screening scores.

[0016] The present invention provides a model training device, comprising:

[0017] a data processing module, configured to determine, by using a pre-trained recall model, recall scores of a plurality of candidate data samples in the database relative to the first query data sample;

[0018] The data processing module is further configured to determine a plurality of recalled data samples from the plurality of candidate data samples based on the recall scores;

[0019] The data processing module is further configured to determine a ranking score of each of the recalled data samples relative to the first query data sample using a pre-trained ranking model;

[0020] The data processing module is further configured to fuse the recall score and the ranking score to obtain a fusion score for each of the recalled data samples;

[0021] The data processing module is further configured to determine a first loss value based on the recall score and the fusion score;

[0022] A training module is used to update the parameters of the recall model based on the first loss value to obtain a fine-tuned recall model.

[0023] An embodiment of the present application provides a data processing device, the device comprising:

[0024] Data acquisition module, used to obtain query data;

[0025] a data processing module, configured to determine screening scores of a plurality of candidate data relative to the query data using a distillation model, wherein the distillation model is a fine-tuned recall model obtained by the model training method provided in an embodiment of the present application;

[0026] The data processing module is further configured to determine target data for responding to the query data from the plurality of candidate data based on the screening scores.

[0027] An embodiment of the present application provides an electronic device, comprising:

[0028] a memory for storing computer-executable instructions or computer programs;

[0029] The processor is used to implement the model training method provided in the embodiment of the present application, or implement the data processing method provided in the embodiment of the present application when executing the computer-executable instructions or computer programs stored in the memory.

[0030] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which is used to implement the model training method provided in the embodiment of the present application, or implement the data processing method provided in the embodiment of the present application when executed by a processor.

[0031] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the model training method provided in the embodiment of the present application is implemented, or the data processing method provided in the embodiment of the present application is implemented.

[0032] The embodiments of the present application have the following beneficial effects:

[0033] By fusing the ranking score generated by the ranking model with the recall score to obtain the fusion score, and calculating the first loss value based on the recall score and the fusion score, a more accurate judgment of the correlation between the first query data sample and the recall data sample (represented by the fusion score) obtained in the two stages of the recall model and the ranking model is achieved as a supervision signal, and the parameters of the recall model are reversely optimized, so that the fine-tuned recall model has the comprehensive capabilities of the two stages of recall and ranking. Therefore, the fine-tuned recall model can be used as the knowledge distillation result of the recall model and ranking model before fine-tuning. On the basis of maintaining the simplicity of the original recall model structure and the low latency advantage, the collaborative optimization path of the recall and ranking stages is opened up, and the end-to-end retrieval accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a structural diagram of the model training system architecture provided in an embodiment of the present application;

[0035] Figure 2A This is a first structural diagram of a server provided in an embodiment of the present application;

[0036] Figure 2B This is a second structural diagram of the server provided in an embodiment of the present application;

[0037] Figure 3 This is a schematic diagram of the principle of the model training method provided in the embodiment of the present application;

[0038] Figure 4A This is a first flow chart of the model training method provided in an embodiment of the present application;

[0039] Figure 4B This is a second flow chart of the model training method provided in an embodiment of the present application;

[0040] Figure 4C This is a third flow chart of the model training method provided in an embodiment of the present application;

[0041] Figure 4D This is a fourth flow chart of the model training method provided in an embodiment of the present application;

[0042] Figure 4E This is a fifth flow chart of the model training method provided in an embodiment of the present application;

[0043] Figure 4F This is a sixth flow chart of the model training method provided in an embodiment of the present application;

[0044] Figure 5 Schematic diagram of the data processing method provided in the embodiment of the present application;

[0045] Figure 6 This is a schematic diagram of the application flow of the data processing method provided in the embodiment of the present application;

[0046] Figure 7 It is a schematic diagram of the training principle of the model training method provided in the embodiment of the present application;

[0047] Figure 8A This is a first structural diagram of the recall model provided in an embodiment of the present application;

[0048] Figure 8B This is a second structural diagram of the recall model provided in an embodiment of the present application.

[0049] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of distinction between the advantages and disadvantages of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0051] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0052] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0053] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0054] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0055] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0056] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0057] 1) In response, it is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed can be real-time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0058] 2) Recall Model: This refers to a model that quickly filters out a subset of candidates related to the query data (such as the keywords entered by the user) from a massive candidate set (such as tens of millions of products, documents, or content).

[0059] 3. Rerank Model: This is a model that performs refined sorting on the candidate subsets output by the recall model. It aims to calculate the degree of match between the candidate items and the user's demand query data and optimize the sorting order.

[0060] 4) Large Language Models (LLMs), also known as large models, are large-scale language models designed to understand and generate human language. They are trained on large amounts of text data and can perform a wide range of tasks, including text summarization, translation, sentiment analysis, and more. Large language models are characterized by their massive size, containing billions of parameters to help them learn complex patterns in language data. They are typically based on deep learning architectures. Large language models are deep learning models trained using large amounts of text data, containing billions or even more parameters. They can be used to generate and understand the meaning of natural language text. Through training, the models can learn the statistical laws and semantic associations of language to build a massive language knowledge base, thereby simulating human language understanding and generation capabilities.

[0061] 5) Multimodal Large Language Models (MLLMs) are deep learning models that can simultaneously process and integrate data from multiple modalities (such as text, images, audio, and video). By integrating information from different modalities, these models can more comprehensively understand and generate content, resulting in excellent performance across a wide range of tasks. With the advancement of computing power and the emergence of large-scale datasets, multimodal large language models have made significant progress in fields such as natural language processing, computer vision, and speech recognition.

[0062] 6) Knowledge Distillation is a deep learning model compression and optimization technology. Its core idea is to transfer the knowledge of a complex large model (teacher model) to a lighter small model (student model) through a "Teacher-Student Framework". This allows the student model to maintain high performance while significantly reducing computational and storage requirements.

[0063] 7) Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) technique used to adapt a pre-trained large language model (LLM) or multimodal model (MLLM) to a new task without fully fine-tuning all parameters. The core idea is to adapt the model to new tasks while preserving the original knowledge by training only a small number of additional parameters through low-rank matrix factorization.

[0064] In the model training method in the related art, the recall model is trained by calculating the similarity score between the query data and the candidate data. Although the structure is simple and the reasoning is fast, there is a problem of inaccurate calculation of the similarity score (recall score), which makes it difficult to meet the high-precision retrieval requirements. Although the two-stage cascade structure of "recall + rerank" (that is, first preliminarily screening the candidate set through the recall model, and then using the ranking model for refined scoring) can improve the retrieval effect, the need to train and maintain two independent models (recall model and ranking model) increases the difficulty of deployment and optimization, and the two-stage cascade increases the consumption of computing resources, resulting in an increase in the overall reasoning time, making it difficult to adapt to application scenarios with high real-time requirements.

[0065] The embodiments of the present application provide a model training method, a data processing method, an apparatus, a device, a computer-readable storage medium, and a computer program product, which can improve the retrieval accuracy of the fine-tuned recall model while maintaining the simplicity of the original recall model structure and the low latency advantage. The exemplary application of the electronic device provided by the embodiments of the present application is described below. The device provided by the embodiments of the present application can be implemented as various types of terminals such as laptops, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, and car terminals, and can also be implemented as a server.

[0066] See also Figure 1 , Figure 1 This is a schematic diagram of the structure of the model training system architecture provided in the embodiment of the present application. Figure 1 The server 100, the terminal device 200 and the network 300 are involved. The terminal device 200 is connected to the server 100 via the network 300, wherein the network 300 can be a wide area network or a local area network, or a combination of the two.

[0067] In some embodiments, the model training method provided in the embodiments of the present application can be implemented collaboratively by a server and a terminal device. For example, the terminal device 200 sends multiple candidate data samples to the server 100, and the server 100 receives the multiple candidate data samples, trains the model training method provided in the embodiments of the present application to obtain a fine-tuned recall model, sends the fine-tuned recall model to the terminal device 200, and the terminal device 200 receives the fine-tuned recall model, and implements the data processing method provided in the embodiments of the present application based on the fine-tuned recall model.

[0068] The fine-tuned recall model provided in the embodiments of the present application can be applied to various scenarios requiring retrieval, such as product retrieval scenarios, educational resource retrieval scenarios, etc., as illustrated below with examples.

[0069] 1) Product retrieval scenario: For example, a user takes a picture of a product or enters a product description (such as "blue sneakers") on a terminal device. The terminal device transmits the product picture or description to the server. The server uses a fine-tuned recall model to perform cross-modal similarity calculations on multimodal data such as product images, specifications, and user reviews in the product database. It quickly recalls a set of product candidates that match the user input (such as similar styles) and returns them to the terminal device in a sorted order based on the degree of match (the recall score obtained by the fine-tuned recall model), providing users with accurate product recommendations and retrieval results.

[0070] 2) Educational resource retrieval scenarios: For example, students enter subject keywords (such as "high school physics mechanics formulas") on terminal devices (such as tablets and mobile phones). The terminal device transmits the text query (i.e., subject keywords) to the server. The server uses a fine-tuned recall model to perform cross-modal matching on multimodal data such as courseware documents, teaching videos, and example analysis in the educational resource library, recall relevant resources (such as video explanations of corresponding chapters and example problem sets), sort them by matching degree (the recall score obtained by the fine-tuned recall model), and return them to the terminal device to help students efficiently obtain learning materials.

[0071] 3) Intelligent customer service knowledge retrieval scenarios, for example, when a user enters a text question (such as "How to apply for a return or exchange") or sends a screenshot of the customer service conversation through the chat window on the terminal device, the terminal device extracts the key information in the natural language text or screenshot (such as order number, question type label) and uploads it to the server; the server runs a fine-tuned recall model to search multimodal data such as historical conversation records, policy documents, and operation guides in the customer service knowledge base, recalls relevant answer templates or solutions, and returns them to the terminal device according to the matching degree (the recall score obtained by the fine-tuned recall model), thereby realizing the automation and precision of customer service responses.

[0072] In other embodiments, the model training method provided in the embodiments of the present application can be implemented independently by a terminal device. The terminal device 200 calls multiple candidate data samples from a local database, obtains a fine-tuned recall model through the model training method provided in the embodiments of the present application, and implements the data processing method provided in the embodiments of the present application based on the fine-tuned recall model.

[0073] In other embodiments, the data processing methods provided in the embodiments of the present application can be implemented collaboratively by a server and a terminal device. For example, terminal device 200 sends query data to server 100. Server 100 receives the query data, obtains target data using the data processing methods provided in the embodiments of the present application, and sends the target data to terminal device 200. The data processing methods provided in the embodiments of the present application are implemented based on a distillation model, which is a fine-tuned recall model obtained using the model training methods provided in the embodiments of the present application.

[0074] In other embodiments, the data processing method provided by the embodiments of the present application can be implemented solely by a terminal device. The terminal device 200 invokes a local distillation model and obtains target data corresponding to the query data using the data processing method provided by the embodiments of the present application. The distillation model can be a fine-tuned recall model obtained by the terminal device 200 using the model training method provided by the embodiments of the present application, or a fine-tuned recall model obtained by the server 100 using the model training method provided by the embodiments of the present application and sent to the terminal device 200.

[0075] Here, server 100 can be a single server. In this case, the model training method and data processing method provided in the embodiments of the present application can be implemented by the same server. Server 100 can also be a cluster of servers. In the case where server 100 is a server cluster, the model training method and data processing method provided in the embodiments of the present application can be implemented by different servers, and the embodiments of the present application do not limit this.

[0076] Take the server for model training as an example, see Figure 2A , Figure 2A This is a first structural diagram of a server provided in an embodiment of the present application. Figure 2A The server 100-1 shown includes: at least one processor 110-1, a memory 130-1, and at least one network interface 120-1. The various components in the server 100-1 are coupled together via a bus system 140-1. It is understood that the bus system 140-1 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 140-1 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 140-1 is not described in detail. Figure 2A In FIG. 1 , various buses are labeled as bus system 140 - 1 .

[0077] The processor 110-1 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0078] Memory 130-1 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. Memory 130-1 may optionally include one or more storage devices physically located remotely from processor 110-1.

[0079] The memory 130-1 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 130-1 described in the embodiments of the present application is intended to include any suitable type of memory.

[0080] In some embodiments, the memory 130 - 1 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0081] Operating system 131-1, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0082] A network communication module 132-1, configured to communicate with other electronic devices via one or more (wired or wireless) network interfaces 120-1. Exemplary network interfaces 120-1 include Bluetooth, Wi-Fi, and USB.

[0083] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2A The model training device 133 stored in the memory 130-1 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a data processing module 1331 and a training module 1332. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0084] Take the server for data processing as an example, see Figure 2B , Figure 2B This is a second structural diagram of the server provided in an embodiment of the present application. Figure 2B The server 100-2 shown includes: at least one processor 110-2, a memory 130-2 and at least one network interface 120-2. The various components in the server 100-2 are coupled together via a bus system 140-2. It is understood that the bus system 140-2 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 140-2 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 140-2 is not described in detail. Figure 2BIn the figure, various buses are labeled as bus system 140-2. The detailed description of processor 110-2 and memory 130-2 is as above and will not be repeated here.

[0085] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2B The data processing device 134 stored in the memory 130-2 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a data acquisition module 1341 and a data processing module 1342. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0086] In some embodiments, the terminal device or server can implement the model training method and data processing method provided by the embodiment of the present application by running various computer executable instructions or computer programs.For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level.The computer program can be a native program or software module in the operating system; it can be a local (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run; it can also be a small program that can be embedded in any APP, that is, a program that only needs to be downloaded to a browser environment and can be run.In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0087] In other embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the model training method and data processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0088] The following will combine the exemplary application and implementation of the server provided in the embodiment of the present application, taking the server as the execution entity, to illustrate the model training method provided in the embodiment of the present application.

[0089] First, we introduce the basic principles of model training. Figure 3 , Figure 3 This is a schematic diagram of the principle of the model training method provided in an embodiment of the present application. First, a recall score of each candidate data sample in the database relative to the first query data sample is obtained through a pre-trained recall model, and multiple recall data samples are determined from multiple candidate data samples based on the recall score (corresponding to steps 101 to 102 below). Next, a ranking score of each recall data sample is obtained through a pre-trained ranking model (corresponding to step 103 below). Next, the recall score and ranking score corresponding to each recall data sample are fused to obtain a fusion score corresponding to each recall data sample (corresponding to step 104 below). Finally, a first loss value is determined based on the recall score and the fusion score, and the parameters of the recall model are updated based on the first loss value to obtain a fine-tuned recall model (or called a recall model after knowledge distillation, a distillation model) (corresponding to steps 105 to 106 below).

[0090] The following describes the specific method of model training. Figure 4A , Figure 4A This is a first flow chart of the model training method provided in the embodiment of the present application, which will be combined with Figure 4A The steps shown are explained.

[0091] In step 101 , a recall score of a plurality of candidate data samples in a database relative to a first query data sample is determined using a pre-trained recall model.

[0092] It should be noted that the first query data sample can be either unimodal or multimodal. For example, the first query data sample is an image modality, a text modality, or a multimodal of an image modality and a text modality, etc. The candidate data sample can also be either unimodal or multimodal. The embodiment of the present application does not limit this. The training of the recall model can refer to the description of steps 201 to 204 below.

[0093] For example, if the first query data sample is an image of "a woman in a red dress" (image modality), and the candidate data sample is a collection of product images in the database (single modality), the recall model can calculate the recall scores of each product image and the query image; if the first query data sample is "recommend a red dress suitable for summer" (text modality), and the candidate data sample is still a product image, then the recall scores of the text and image need to be calculated across modalities; if the first query data sample is "red dress" text + example image (multimodality), and the candidate data sample is product image + product description (multimodality), then the recall score needs to be calculated by integrating multiple modal information.

[0094] In some embodiments, for unimodal scenarios (such as the first query data sample of text modality and the candidate data sample of text modality, the first query data sample of text modality and the candidate data sample of image modality, etc.), the vector similarity (such as cosine similarity) between each candidate data sample and the first query data sample is calculated as the recall score through a pre-trained recall model; for multimodal scenarios (such as the first query data sample of text modality + image modality and the candidate data sample of text modality, etc.), the recall model first encodes the data of each modality of the first query data sample separately, and then fuses them into a unified representation through an attention mechanism or feature splicing to obtain a fused feature vector corresponding to the first query data sample, and calculates the similarity between the fused feature vector and the candidate data sample as the recall score.

[0095] For example, when the first query data sample is "new summer dress" (text modality) and the candidate data samples are products in the database (including product images and product description texts), the recall model first converts the query text (first query data sample) into a text vector through a text encoder (such as Transformer, Bidirectional Encoder Representations from Transformers (BERT)), etc.), and at the same time converts the product image into an image vector through an image encoder (such as ResNet, Vision Transformer (ViT), etc.), converts the product description text into a text vector, and fuses the image vector and text vector corresponding to the product through the attention mechanism to obtain the product vector of each product; finally, the cosine similarity between the text vector of the first query data sample and each product vector is calculated, and the similarity value is used as the recall score. The higher the recall score, the more relevant the candidate data sample is to the first query data sample.

[0096] Here, the recall score (RecallScore) is a quantitative indicator calculated by the recall model, which is used to measure the semantic relevance between the candidate data sample (single modality or multimodality) and the first query data sample (single modality or multimodality). The recall score is generated by the recall model by performing similarity calculation (such as cosine similarity, dot product operation) on the encoding features of the first query data sample and the encoding features of the candidate data sample (such as image features, text embedding). The higher the recall score, the closer the semantic association between the candidate data sample and the first query data sample.

[0097] In another embodiment, see Figure 4B Before determining the recall scores of multiple candidate data samples in the database relative to the first query data sample through the pre-trained recall model, the following steps 201 to 204 may also be performed, which are described in detail below.

[0098] In step 201, positive samples and negative samples corresponding to the second query data sample are obtained, wherein the positive sample is a sample having a semantic association with the second query data sample, and the negative sample is a sample having no semantic association with the second query data sample, and the second query data sample is different from the first query data sample.

[0099] Here, semantic association refers to the association characteristics of the semantic vectors of the sample and the second query data sample in the high-dimensional semantic embedding space generated by the deep learning semantic representation model (such as the Transformer encoder) under the cosine similarity, Euclidean distance and other metric standards, which meet the preset threshold (for example, when the cosine similarity value range is [0,1], the candidate samples with cosine similarity greater than 0.85 are positive samples, and when the Euclidean distance value range is [0,1], the candidate samples with Euclidean distance less than 0.15 are positive samples).

[0100] It should be noted that in the training process of the recall model, the second query data sample is the input sample for recall model training, and its function is to guide the recall model to learn semantic association patterns. Unlike the first query data sample (used to fine-tune the recall model), the second query data sample is specifically used to construct training sample pairs (such as sample pairs consisting of second query data samples and positive samples, and sample pairs consisting of second query data samples and negative samples) to optimize the parameters of the recall model. The second query data sample can also be single-modal (such as image, text) or multi-modal (such as a combination of image and text), and its modality type can be consistent with the positive sample and the negative sample, or it can be inconsistent. The embodiment of the present application does not limit this. In addition, for the sake of ease of description, the following description is based on a single second query data sample. During the training process of the recall model, the number of second query data samples can be multiple, and each second query data sample corresponds to at least one positive sample and at least one negative sample.

[0101] Positive samples are samples that have a clear semantic association with the second query data sample. During the training process, positive samples are used to indicate to the recall model "which content is relevant", so that the correct semantic matching pattern can be learned. For example, if the second query data sample is the text "red dress", then the positive sample can be an image of a red dress, text describing a red dress, or related product information; if the second query data sample is the image "a cat on a sofa", then the positive sample can be a semantically similar image (such as a cat from different angles) or a text description (such as "cat resting").

[0102] Positive samples can be generated through manual annotation, rule matching, or an existing knowledge base. The embodiments of the present application do not limit the specific method of obtaining positive samples.

[0103] Negative samples are samples that have no semantic connection with the second query data sample. During training, negative samples are used to instruct the recall model on "what content is irrelevant," thereby enhancing the model's ability to discern semantic differences. For example, if the second query data sample is the text "red dress," negative samples might be images or text of unrelated categories such as cars or electronics. If the second query data sample is an image of "beach scenery," negative samples might be images of indoor scenes or text describing technology products.

[0104] There are various strategies for obtaining negative samples, which can be random sampling (random selection from irrelevant categories), difficult example mining (selecting samples that are semantically similar to the second query data sample but are actually irrelevant), or negative samples generated based on adversarial generation, etc. The embodiments of the present application do not limit the specific method of obtaining negative samples.

[0105] In step 202, the similarity between the second query data sample and the positive sample is obtained through the recall model to be trained as a first recall score sample, and the similarity between the second query data sample and the negative sample is obtained as a second recall score sample.

[0106] In some embodiments, the second query data sample is encoded through the recall model to be trained to obtain the encoding features of the second query data sample, and the positive sample is encoded to obtain the encoding features of the positive sample, and the negative sample is encoded to obtain the encoding features of the negative sample; the similarity between the encoding features of the second query data sample and the encoding features of the positive sample is obtained as the first recall score sample, and the similarity between the encoding features of the second query data sample and the encoding features of the negative sample is obtained as the second recall score sample.

[0107] In some embodiments, taking the modality of the second query data sample as an example, encoding the second query data sample to obtain the encoding features of the second query data sample can be achieved in the following ways: performing image segmentation processing on the second query data sample to obtain multiple pixel blocks; sorting the multiple pixel blocks into a sequence of pixel blocks to be encoded; and encoding the sequence of pixel blocks to be encoded to obtain the encoding features of the second query data sample.

[0108] For example, assuming that the size of the second query data sample is 224×224, it is divided according to the set pixel block size (for example, 16×16 pixel block size), and the result after division is (224 / 16). 2 = 196 pixel blocks (Patches), multiple pixel blocks are arranged (for example, in order from left to right and from top to bottom) as a pixel block sequence to be encoded.

[0109] For example, encoding the pixel block sequence to be encoded to obtain the encoding features of the second query data sample can be achieved in the following ways: embedding encoding the pixel block sequence to obtain embedded features; performing attention encoding on the embedded features to obtain attention encoding features; and mapping the attention encoding features to obtain the encoding features of the second query data sample.

[0110] For example, the pixel block sequence to be encoded can be embedded in the embedding layer to obtain embedded features. For example, the pixel block sequence to be encoded is convolved in the embedding layer through a convolution operation, and the pixel block sequence to be encoded after the convolution is position encoded (Position Embedding) to obtain embedded features. Before the embedded coding is performed, each pixel block to be encoded in the pixel block sequence to be encoded can be flattened into a one-dimensional vector. For example, for a 16×16×3 pixel block to be encoded, the length of the flattened vector is 768. The position coding process can be generated by a fixed algorithm. For example, it can be implemented using a combination of sine and cosine functions. For each position (for example, the pixel position of each pixel point of each pixel block to be encoded) pos and the dimension i of the feature, the position coding of the even dimension uses the sine function, while the position coding of the odd dimension uses the cosine function. The embodiment of the present application does not limit the specific implementation method of the position coding.

[0111] For example, before attention encoding, the embedded features can also be normalized (such as layer normalization (Layer Norm)) to obtain normalized features. For the normalized features for attention encoding, a linear transformation is first performed to generate three matrices: query vector (Query, Q), key vector (Key, K) and value vector (Value, V). For each attention head (Head Attention), the dot product of Q and K is calculated to obtain the attention score. Next, the attention score is normalized, for example, a normalization function (such as softmax function) is applied to convert the attention score into a probability distribution, and V is weighted using the attention probability distribution to generate a new feature representation. Finally, the outputs of all attention heads are connected and linearly transformed to obtain a rich representation that includes the correlation between different positions in the first normalized feature, namely the attention encoding feature.

[0112] For example, the attention encoding features can be mapped through a feed-forward neural network (FFN) layer to obtain the encoding features of the second query data sample, wherein the feed-forward neural network layer can adopt a multilayer perceptron (MLP) structure, for example.

[0113] In other embodiments, the second query data sample can be encoded (convolution processing) through a convolutional neural network (CNN) to obtain the encoding features of the second query data sample. The embodiments of the present application do not limit the specific implementation method of encoding the second query data sample of the image modality.

[0114] In some embodiments, taking the second query data sample as a text modality as an example, the second query data sample is encoded through the recall model to be trained to obtain the encoding features of the second query data sample, which can be achieved in the following ways: segmenting the second query data sample to obtain multiple word units; embedding encoding processing is performed on the multiple word units to obtain embedded features; and attention encoding processing is performed on the embedded features to obtain the encoding features of the second query data sample.

[0115] For example, punctuation marks such as spaces, periods, and commas can be used as word segmentation identifiers to perform word segmentation on the second query data sample, that is, the second query data sample is divided into multiple input units, also called tokens. Word segmentation tools can also be used to perform word segmentation on the second query data sample, such as Jieba, NLTK, SpaCy, etc. These tools can achieve accurate Chinese and English word segmentation through algorithms and language models.

[0116] For example, assuming that the input second query data sample is "summer new style dress", after word segmentation processing, multiple tokens are obtained, for example, represented as [summer / new style / dress / ]. Next, the multiple tokens are embedded and encoded. Assuming that each token is mapped to a 2048-dimensional vector, the second query data sample is converted into a 14*2048 embedding feature, and then the 14*2048 embedding feature is attention encoded to obtain the encoded feature of the second query data sample. Here, the token refers to the basic processing unit of the second query data sample. A token can be a text, word, subword, character or other meaningful element, depending on the granularity and requirements of the text processing, and is not limited in the embodiments of the present application.

[0117] For example, the embedding layer can perform embedding encoding processing on multiple words to obtain embedding features. For example, the embedding layer can perform embedding encoding processing using embedding methods such as word embeddings, character embeddings, and subword embeddings to obtain embedding features.

[0118] For example, the attention weight of the embedded feature is calculated through an attention mechanism (such as a self-attention mechanism). The attention weight represents the importance of the embedded feature to the current task. The attention weight of the embedded feature is multiplied by the embedded feature to obtain a weighted embedded feature, that is, the encoding feature of the second query data sample.

[0119] In other embodiments, the second query data sample can also be encoded through a network model such as Recurrent Neural Networks (RNN) to obtain the encoding features of the second query data sample. The implementation of this application does not limit the specific implementation method of encoding the second query data sample in text mode.

[0120] For example, encoding the second query data sample through RNN can be achieved in the following way: before inputting the second query data sample into the RNN, preprocessing can be performed, including tokenization, part-of-speech tagging, removal of stop words and other steps. Next, each word in the second query data sample is converted into a numerical vector and combined to obtain a numerical sequence, for example, through a bag-of-words model, pre-trained word embedding (such as Word2Vec), etc. Finally, the numerical sequence is input into the RNN, and the RNN will learn how to capture the sequence dependency in the second query data sample and generate a fixed-size vector representation. This vector representation is the encoding feature of the second query data sample.

[0121] In some embodiments, taking the audio modality as an example, the second query data sample is encoded through the recall model to be trained to obtain the encoding features of the second query data sample, which can be achieved in the following ways: preprocessing the second query data sample to obtain a preprocessed audio sample; performing convolution processing on the preprocessed audio sample to obtain the encoding features of the second query data sample.

[0122] For example, the second query data sample is converted into a discrete digital signal by sampling, and the discrete digital signal is normalized to obtain a preprocessed audio sample. For example, the amplitude of the analog signal (the second query data sample) is recorded at a fixed time interval (sampling rate), and the continuous amplitude is mapped to a discrete value (such as a 16-bit integer) to obtain a discrete digital signal, and the discrete digital signal is subjected to mean-variance normalization (Z-Score Normalization) processing to obtain the preprocessed audio sample.

[0123] For example, feature maps of the preprocessed audio samples are obtained through convolutional layers. These feature maps represent multiple levels of representation of the audio samples and various features of the audio, such as frequency and timing. The feature maps can also be converted into more compact features, such as global average pooling, to obtain the encoded features corresponding to the second query data sample.

[0124] In other embodiments, obtaining the encoding features of the second query data sample in the audio modality can be achieved through a pre-trained audio feature extraction subnetwork (or audio feature encoder), such as a TitaNet model, a WaveNet model, etc.

[0125] Taking the TitaNet model as an example, the encoding features of the second query data sample can be obtained in the following way: first, the second query data sample is preprocessed, for example, the audio waveform is converted into Mel-Frequency Cepstral Coefficients (MFCCs) or other types of spectrograms. The preprocessed second query data sample is divided into a series of overlapping segments, which will serve as the basic input units of the audio feature encoder. Next, the self-attention between input units is calculated through the Transformer encoder to capture the long-term dependencies in the second query data sample. Finally, by stacking multiple layers of Transformer encoders, the encoding features with rich semantics are finally output.

[0126] In some embodiments, taking the case where the modality of the second query data sample is multimodal as an example, the second query data sample is encoded through the recall model to be trained to obtain the encoding features of the second query data sample. This can be achieved in the following way: the encoding features of the second query data sample of each modality are spliced ​​to obtain spliced ​​features; and the spliced ​​features are used as the encoding features of the second query data sample.

[0127] For example, the encoded features of the second query data sample of each modality can be mapped to a fixed-size feature space through a multilayer perceptron (MLP), and the mapped encoded features of the second query data sample of each modality are concatenated to obtain concatenated features. MLP is a feedforward neural network composed of multiple fully connected layers, each of which uses an activation function. MLP can automatically learn nonlinear relationships between features and transform them into a form suitable for subsequent tasks.

[0128] Taking the modalities of the second query data sample as image modality and text modality as an example, the encoded features of the image modality are represented as a tensor (100, 512), and the encoded features of the text modality are represented as a tensor (100, 768). Then the spliced ​​features are represented as a tensor (100, 512+768)=(100, 1280).

[0129] Here, the implementation method of encoding the positive sample to obtain the encoding features of the positive sample and encoding the negative sample to obtain the encoding features of the negative sample can be found in the above description of encoding the second query data sample, which will not be repeated here.

[0130] In some embodiments, a similarity measurement method can be selected to obtain the similarity between the encoding features of the second query data sample and the encoding features of the positive sample, as well as the similarity between the encoding features of the second query data sample and the encoding features of the negative sample. The similarity measurement methods used include Euclidean distance, Manhattan distance, cosine similarity, etc.

[0131] In step 203 , a fourth loss value is determined based on the first recall score sample and the second recall score sample.

[0132] In some embodiments, a fourth loss value is determined based on the first recall score sample and the second recall score sample using a preset loss function (eg, a contrastive loss function).

[0133] For example, the fourth loss value can be determined by formula (1):

[0134]

[0135] in, represents the similarity between the encoding features of the qth second query data sample and the encoding features of the positive sample (the first recall score sample), It represents the similarity between the encoded features of the qth second query data sample and the encoded features of the i-th sample (positive sample or negative sample) (first recall score sample or second recall score sample), and τ is a hyperparameter used to control the smoothness of the distribution.

[0136] In step 204 , based on the fourth loss value, the parameters of the recall model to be trained are updated to obtain a pre-trained recall model.

[0137] In some embodiments, the gradient information of the fourth loss value for each parameter of the recall model is obtained through the back-propagation algorithm, and the parameters of the recall model are updated using the obtained gradient information according to the gradient descent optimization algorithm (such as batch gradient descent, stochastic gradient descent, etc.). The above process is repeated until a certain number of iterations is reached or the recall model converges, thereby obtaining a trained recall model as the pre-trained recall model in step 101.

[0138] Through steps 201 to 204, the recall model pre-training based on contrastive learning is realized, the semantic association difference between positive and negative samples and the query sample (the second query data sample) is used to construct a supervision signal, the recall score sample is generated by similarity calculation, and the model parameters are optimized by back propagation based on the contrastive loss (such as Info-NCE), so that the recall model can accurately distinguish between relevant and irrelevant samples in the feature encoding stage, thereby improving the semantic understanding ability and feature discrimination ability of the pre-trained recall model for multimodal data.

[0139] Continue to see Figure 4A , in step 102, a plurality of recalled data samples are determined from a plurality of candidate data samples based on the recall scores.

[0140] In some embodiments, multiple candidate data samples are sorted in descending order according to the recall scores, and the top N candidate data samples in the sorting results are selected as multiple recalled data samples.

[0141] For example, if the recall model calculates the recall scores of 1,000 candidate products (candidate data samples) and the query "summer dress" (the first query data sample), after sorting them in descending order by score, the first 50 products are selected as the recall data samples.

[0142] In other embodiments, candidate data samples with the same modality as the first query data sample are screened from multiple candidate data samples to serve as target data samples, and each target data sample is sorted in descending order according to the recall score, and the top N target data samples of the sorting results are selected as multiple recall data samples.

[0143] For example, if the first query data sample is a text description of "red high heels" (text modality), and the candidate data samples include product pictures (image modality) and product titles (text modality), only the product titles are retained as target data samples; after sorting the titles of these text modalities (target data samples) according to the recall scores, the top 20 titles (such as "red stiletto high heels", "genuine leather red high heels") are selected as recall data samples.

[0144] In other embodiments, the candidate data samples for each modality are sorted in descending order according to the recall score, and the top N candidate data samples in the sorting results corresponding to each modality are selected as multiple recalled data samples.

[0145] For example, if the first query data sample is a combination of images and text (multimodal) for "travel backpack", and the candidate data samples include images and text descriptions (three modalities), then the candidate data samples of each modality are sorted according to the recall score:

[0146] Image modality: Select the first 10 images related to travel backpacks (e.g., “outdoor waterproof backpack” images, “large-capacity travel backpack” images);

[0147] Text description mode: select the top 10 highly relevant descriptions (e.g., “expandable capacity backpack”, “backpack suitable for 7-day travel”);

[0148] Finally, these 20 recall data samples are used as multiple recall data samples.

[0149] In step 103 , a ranking score of each recalled data sample relative to the first query data sample is determined using a pre-trained ranking model.

[0150] Here, the ranking score is a quantitative indicator calculated by the ranking model, which is used to evaluate the degree of matching between the recalled data sample and the first query data sample. Its core function is to further generate semantic matching (ranking score) for the candidate set (multiple candidate data samples) screened out in the recall stage.

[0151] For example, if the first query data sample is a text description of "summer commuter dress" (unimodal), the recall model has screened out 50 recall data samples (such as pictures and titles of dresses of different styles) from the product database, then the sorting model will calculate the sorting score for each recall data sample. The training of the sorting model can be found in the description of steps 301 to 306 below.

[0152] In other embodiments, see Figure 4CBefore determining the ranking score of each recalled data sample relative to the first query data sample through the pre-trained ranking model, the following steps 301 to 306 may also be performed, which are described in detail below.

[0153] In step 301 , a third query data sample and a positive sample corresponding to the third query data sample are obtained, wherein the positive sample is a sample that semantically matches the third query data sample.

[0154] Here, "semantic matching" refers to the relevance of the third query data sample (such as text, image or multimodal combination) with the candidate sample (positive sample) at the meaning, intention or concept level, rather than matching based solely on surface features (such as keyword co-occurrence, pixel similarity).

[0155] Semantic matching can be judged or learned through the following methods:

[0156] 1. Quantitative Standards for Semantic Relevance

[0157] High relevance threshold: A semantic match is considered when the relevance between the third query data and the sample exceeds a threshold, as determined by a preset similarity metric (e.g., cosine similarity, semantic embedding distance, etc.). For example, in ranking model training, a positive sample must have a semantic relevance greater than a threshold (e.g., cosine similarity > 0.9) with the third query data, while a negative sample must have a semantic relevance less than the threshold.

[0158] Driven by business logic and domain knowledge: Matching criteria must be combined with business scenarios (such as e-commerce recommendations and information retrieval) and determined through manual annotation, business rules, or knowledge bases (such as knowledge graphs). For example, a positive sample for the query "weekend outdoor activities" must meet business semantics such as "leisure, local, and short-term," rather than simply containing the keywords "weekend" and "outdoor."

[0159] 2. Semantic consistency between cross-modality and multimodality

[0160] Modality independence: Supports single-modal (e.g., text, image) or multi-modal (e.g., image and text combination) third-party query data samples and candidate samples. The core is to ensure the consistency of semantic meaning across different modalities. For example, semantic matching between the image-text query "red sneakers" and the text sample "red running shoes" requires the ranking model to understand the correspondence between the color and item type in the image and the text description.

[0161] Cross-modal feature fusion: Mapping inputs of different modalities to a unified semantic space through an encoding model (such as Transformer) and calculating the matching degree based on the semantic embedding vector rather than directly comparing the original modal features.

[0162] 3. Task-oriented semantic understanding

[0163] Hint-guided matching objectives: Preset hints (e.g., "determine whether a candidate sample matches the query") clarify the semantic matching task boundaries. For example, hints can specify that matching should focus on "purpose consistency" (e.g., the query "business leather shoes" matches the sample "formal leather shoes") or "entity category consistency" (e.g., the query "smartphone" matches the sample "mobile phone model" but not "tablet").

[0164] Learning objectives of the ranking model: Semantic matching judgment is optimized through ranking model training. The model needs to learn to distinguish "semantically related" from "semantically unrelated" samples. For example, the ranking loss (such as cross-entropy loss) of positive samples (semantic matching) and negative samples (semantic mismatch) is used to guide parameter updates, so that the model can capture deep semantic associations (such as implicit intent and context).

[0165] 4. Difficult Example Mining and Semantic Boundary Distinguishing

[0166] Adversarial semantic discrimination: Through difficult example mining (such as selecting samples with similar semantics but actually unrelated, such as when the third query data sample is "a certain brand of mobile phone", the negative sample is "a certain brand of tablet"), the model's ability to judge semantic boundaries is strengthened, requiring the ranking model to not only identify "explicit matches" but also distinguish "semantic ambiguity" or "category confusion".

[0167] Dynamic threshold and training stage optimization: In different stages of sorting model training (such as randomly sampling negative samples in the first stage and focusing on difficult examples with high recall scores in the second stage), the accuracy of the model for complex semantic matching is gradually improved by adjusting the semantic matching screening strategy (such as increasing the similarity threshold).

[0168] Here, the third query data sample is an input sample in the sorting model training process, which is used to construct training sample pairs (such as sample pairs consisting of the third query data sample and positive samples, and sample pairs consisting of the third query data sample and negative samples) to optimize the parameters of the sorting model. Unlike the first query data sample (used to fine-tune the recall model) and the second query data sample (used for recall model training), the third query data sample is specifically used for training the sorting model, and its function is to guide the sorting model to learn "how to correctly sort relevant and irrelevant samples". The third query data sample can also be single-modal (such as image, text) or multi-modal (such as a combination of image and text), and its modality type can be consistent with the positive sample and negative sample, or it can be inconsistent. The embodiments of the present application do not limit this.

[0169] It should be noted that, for the sake of ease of description, the description here is based only on a single third query data sample. In the training of the ranking model, the number of third query data samples can be multiple, and each third query data sample corresponds to at least one positive sample and at least one negative sample.

[0170] In the ranking model training, the positive sample is a sample that matches the semantics of the third query data sample and has a high correlation (such as greater than a preset similarity threshold). The positive sample is used to indicate to the ranking model "which recalled data samples are truly semantically matched with the third query data sample". The positive sample can be determined by manual annotation, business rules or an existing knowledge base. For example, if the third query data sample is "new smartphone", the positive sample can be a highly rated mobile phone model, a mobile phone that the user has actually purchased, or an officially recommended popular model; if the third query data sample is "weekend outdoor activities", the positive sample can be park tours, mountain climbing routes and other related content. The positive sample is given a higher labeling score than the negative sample (such as the positive sample's labeling score is 1.0 and the negative sample's labeling score is 0.1) and is used to calculate the ranking loss (the fifth loss value).

[0171] In step 302 , a pre-trained recall model is used to determine a negative sample corresponding to the third query data sample from a plurality of candidate negative samples, wherein the negative sample is a sample that does not semantically match the third query data sample.

[0172] Here, in ranking model training, negative samples are samples that do not semantically match or have low relevance (e.g., less than a preset similarity threshold) to the third query data sample. Negative samples are used to help the ranking model learn how to distinguish irrelevant content. Negative samples are selected in the following ways:

[0173] Random sampling: randomly select from irrelevant categories (for example, when querying "smartphone", negative samples are books, furniture, etc.);

[0174] Hard Negative Mining: Select samples that are semantically similar to the third query data sample but are actually irrelevant (for example, when querying "a certain brand of mobile phone", the negative sample is "a certain brand of tablet computer");

[0175] Recall model assistance: Use the recall model to filter out samples that are mistakenly judged as relevant (such as samples with high recall scores but are actually irrelevant).

[0176] Negative samples are assigned a lower labeling score than positive samples (e.g., a positive sample has a labeling score of 1.0 and a negative sample has a labeling score of 0.0).

[0177] In some embodiments, updating the parameters of the ranking model to be trained is achieved through multiple training stages. The negative sample corresponding to the third query data sample is determined from multiple candidate negative samples through a pre-trained recall model (corresponding to the recall model obtained by training in steps 201 to 204). This can be achieved in the following way: through the recall model, a third recall score sample of each candidate negative sample relative to the third query data sample is obtained; in the first training stage of the ranking model to be trained, based on the third recall score sample, a first number of candidate negative samples are obtained from multiple candidate negative samples to combine into a first candidate negative sample set; candidate negative samples are selected from the first candidate negative sample set as negative samples; in the second training stage of the ranking model to be trained, based on the third recall score sample, a second number of candidate negative samples are obtained from multiple candidate negative samples to combine into a second candidate negative sample set; wherein the second number is smaller than the first number; and candidate negative samples are selected from the second candidate negative sample set as negative samples corresponding to the third query data sample.

[0178] For example, assume that the third query data sample is "business casual leather shoes" (text mode), and the candidate negative sample library contains 1,000 products (such as sneakers, sandals, suits, backpacks, etc.). During the training process of the ranking model:

[0179] First training phase:

[0180] The recall model calculates the third recall score sample of each candidate negative sample and “business casual leather shoes”, for example, “black leather shoes” (third recall score sample 0.85), “white sneakers” (third recall score sample 0.30), and “genuine leather backpack” (third recall score sample 0.10). Based on the third recall score sample, the top 100 high-scoring samples (such as the third recall score sample > 0.2) are selected to form the first candidate negative sample set, and one sample is randomly selected from the first candidate negative sample set as the negative sample of the third query data sample being “business casual leather shoes”.

[0181] Second training phase:

[0182] As the performance of the ranking model improves, the adjustment strategy is to focus on more difficult negative samples. The top 50 high-scoring samples (third recall score samples > 0.6) are selected to form the second candidate negative sample set. These samples are mostly "difficult examples" with highly similar semantics but actually irrelevant (such as "sports casual shoes" and "formal leather shoes"). One sample is randomly selected from the second candidate negative sample set as the negative sample of the third query data sample "business casual leather shoes".

[0183] In step 303, the preset prompt word, positive sample, and third query data sample are encoded by the to-be-trained ranking model to obtain a first encoded feature, and the prompt word, negative sample, and third query data sample are encoded to obtain a second encoded feature, where the prompt word is used to indicate the task content of the ranking task that the to-be-trained ranking model needs to execute.

[0184] In some embodiments, encoding the preset prompt word, positive sample, and third query data sample by the to-be-trained ranking model to obtain a first encoded feature can be achieved by the following method: obtaining the encoded feature of the positive sample, obtaining the encoded feature of the third query data sample, and obtaining the encoded feature of the preset prompt word; concatenating the encoded feature of the positive sample, the encoded feature of the third query data sample, and the encoded feature of the preset prompt word to obtain a first concatenated feature; using the first concatenated feature as the first encoded feature.

[0185] Exemplarily, the preset prompt word can be expressed as "query: <query information> candidate: <candidate information>. Please indicate whether the above candidate data sample matches the query sample (the third query data sample), and only output 'yes' or 'no', where 'yes' is represented by the numerical value '1' and 'no' is represented by the numerical value '0'", "query: <query information> candidate: <candidate information>. Please indicate whether the above candidate data sample matches the query sample (the third query data sample), and output a ranking score between [0, 1] to quantify the matching degree, and the higher the ranking score, the higher the matching degree between the two", etc.

[0186] Here, for the implementation methods of obtaining the encoded feature of the positive sample and obtaining the encoded feature of the positive sample, reference can be made to the description in step 202 above of encoding the second query data sample to obtain the encoded feature of the second query data sample and encoding the positive sample to obtain the encoded feature of the positive sample, which will not be elaborated here.

[0187] In some embodiments, the implementation method of obtaining the encoded feature of the preset prompt word can be achieved by the following method: performing word segmentation on the preset prompt word to obtain multiple word tokens; performing embedding encoding processing on the multiple word tokens to obtain an embedding feature; performing attention encoding processing on the embedding feature to obtain the encoded feature of the prompt word.

[0188] In some other embodiments, when the preset prompt word, positive sample, and third query data sample are all in text modality, the preset prompt word, positive sample, and third query data sample are concatenated to obtain a first concatenated text, and the first concatenated text is encoded to obtain a first encoded feature.

[0189] For example, the first concatenated text can be expressed as "query: <business casual leather shoes> candidate data sample: <casual sports shoes>. Please ask whether the above candidate data sample matches the query sample (the third query data sample). The output ranking score between [0.1] is used to quantify the matching degree. The higher the ranking score, the higher the matching degree between the two."

[0190] For example, encoding the first concatenated text to obtain a first encoding feature can be achieved in the following ways: segmenting the first concatenated text to obtain multiple word units; performing embedding encoding processing on the multiple word units to obtain embedding features; performing attention encoding processing on the embedding features to obtain the first encoding feature of the first concatenated text.

[0191] In other embodiments, after obtaining the first splicing feature, attention encoding is performed on the first splicing feature to obtain a first encoded feature.

[0192] For example, the first concatenated feature is linearly mapped through the initialized query weight matrix, key weight matrix and value weight matrix to obtain a query vector, a key vector and a value vector, wherein the query weight matrix, the key weight matrix and the value weight matrix are updated during the training of the sorting model; an attention score matrix is ​​obtained based on the query vector and the key vector; and the value vector is weighted based on the attention score matrix to obtain a first encoded feature.

[0193] For example, the first concatenated feature is linearly mapped by the initialized query weight matrix, key weight matrix and value weight matrix to obtain the query vector, key vector and value vector, which can be achieved by the following method: First, the query weight matrix (W Q ), key weight matrix (W K ) and value weight matrix (W V ) perform linear transformation on the first concatenated feature respectively, that is, perform matrix multiplication of the query weight matrix, key weight matrix and value weight matrix with the first concatenated feature respectively to generate the query vector (Q), key vector (K) and value vector (V).

[0194] For example, obtaining an attention score matrix based on the query vector and the key vector can be achieved as follows: for each attention head (Head Attenti on), the dot product (Dot Product) of Q and K is calculated to obtain the attention score matrix. For example, for each attention head, a dot product operation is performed on each query vector with all the key vectors in the head, and a score equal to the number of key vectors is generated for each query vector. These scores reflect the correlation or matching degree between the query vector and each key vector. All these dot product scores are combined to form a matrix as the attention score matrix. In order to make the value of the score matrix within a reasonable range and facilitate subsequent processing, the attention score matrix can be normalized, such as using a softmax function to convert it into a probability distribution.

[0195] For example, weighting the value vector based on the attention score matrix to obtain the first encoding feature can be achieved in the following way: normalizing the attention score matrix, for example, applying a normalization function (such as a softmax function) to convert the attention score matrix into a probability distribution, using the attention probability distribution to weight V to generate a new feature representation, and finally, connecting the outputs of all attention heads to obtain a rich representation that includes the correlation between different positions in the first spliced ​​feature, that is, the first encoding feature.

[0196] Here, the prompt word, the negative sample and the third query data sample are encoded to obtain the second encoding feature, which is similar to the process of obtaining the first encoding feature above and will not be repeated here.

[0197] In step 304, the first encoding feature is decoded by the ranking model to be trained to obtain a first ranking score sample, and the second encoding feature is decoded to obtain a second ranking score sample.

[0198] In some embodiments, linear mapping is performed on the first code through a feedforward neural network layer to obtain linear features; and nonlinear mapping is performed based on the linear features to obtain a first sorting score sample.

[0199] For example, the feedforward neural network layer can adopt a multi-layer perceptron (MLP) structure. First, the first encoded feature is linearly transformed through a linear layer to obtain a linear feature. The linear layer can be, for example, a fully connected layer (Fully Connected Layer) with a weight matrix W1 and a bias vector b1. Next, the linear feature is nonlinearly transformed through an activation function (such as softmax) to obtain a first ranking score sample (such as the first ranking score between [0,1]).

[0200] Here, the process of decoding the second coding feature to obtain the second ranking score sample is similar to the process of obtaining the first ranking score sample, and will not be repeated here.

[0201] In step 305 , a fifth loss value is determined based on the first ranked score sample, the labeled scores corresponding to the positive samples, the second ranked score sample, and the labeled scores corresponding to the negative samples.

[0202] In some embodiments, a fifth loss value is determined based on the first ranked score sample, the labeled scores corresponding to the positive sample, the second ranked score sample, and the labeled scores corresponding to the negative sample through a pre-set loss function (such as a cross entropy loss function, a mean square error loss function, etc.).

[0203] For example, the fifth loss value can be determined by formula (2):

[0204]

[0205] in, represents the first ranking score sample of the positive sample corresponding to the i-th third query data sample, Represents the second ranking score sample of the negative sample corresponding to the i-th third query data sample, y i Indicates the label score corresponding to the positive sample, (1-y i ) represents the labeling score corresponding to the negative sample, and σ(·) represents the activation function (such as the sigmoid function).

[0206] In step 306, based on the fifth loss value, the parameters of the sorting model to be trained are updated to obtain a pre-trained sorting model.

[0207] In some embodiments, the gradient information of the fifth loss value for each parameter of the sorting model is obtained through the back-propagation algorithm, and the parameters of the sorting model are updated using the obtained gradient information according to the gradient descent optimization algorithm (such as batch gradient descent, stochastic gradient descent, etc.). The above process is repeated until a certain number of iterations is reached or the sorting model converges, thereby obtaining a trained sorting model as the pre-trained sorting model in step 103.

[0208] Through steps 301 to 306, the sorting model pre-training based on task prompts and dynamic hard example mining is realized. Specifically, the sorting task objectives are explicitly defined using prompt words (such as "output matching scores"), and high-quality training pairs are constructed in combination with the progressively hard negative samples screened by the recall model (step 302). The sorting scores are generated through multimodal encoding fusion and decoding (step 304), and the contrast loss is calculated using the labeled scores as supervision signals (step 305). This enables the sorting model to accurately learn fine-grained relevance discrimination patterns, thereby significantly improving the matching accuracy and robustness of the pre-trained sorting model for high-confusion samples, and providing reliable relevance feedback signals for the recall model fine-tuning stage.

[0209] Continuing to refer to FIG. A , in step 104 , the recall score and the ranking score are fused to obtain a fusion score for each recalled data sample.

[0210] In some embodiments, the following processing is performed for each recalled data sample: according to a preset third weight coefficient and a preset fourth weight coefficient, the recall score and the ranking score corresponding to the recalled data sample are weighted and summed to obtain a fusion score corresponding to the recalled data sample.

[0211] For example, the fusion score corresponding to each recall data sample can be expressed by formula (3):

[0212] S multi =a×S recall +(1-a)×S rerank (3)

[0213] Among them, S multi represents the fusion score, a represents the third weight coefficient, (1-a) represents the fourth weight coefficient, S recall represents the recall score, S rerank Represents the ranking score.

[0214] In other embodiments, the following processing is performed for each recalled data sample: determining the interval in which the recall score of the recalled data sample lies among multiple preset recall score intervals; determining the weight coefficients corresponding to the recall score and the ranking score respectively according to the calculation method preset for the interval; and performing weighted summation on the weight coefficients corresponding to the recall score and the ranking score respectively according to the weight coefficients corresponding to the recall score and the ranking score respectively, to obtain the fusion score of the recalled data sample.

[0215] For example, according to the calculation method preset for the interval, determining the weight coefficients corresponding to the recall score and the ranking score respectively can be achieved in the following way: in response to the recall score of the recalled data sample being greater than the first recall score threshold, taking the sum of the first preset value and the third weight coefficient as the fifth weight coefficient, and taking the difference between the second preset value and the fifth weight coefficient as the sixth weight coefficient.

[0216] For example, according to the weight coefficients corresponding to the recall score and the ranking score, respectively, the weighted sum of the weight coefficients corresponding to the recall score and the ranking score is performed to obtain the fusion score of the recalled data sample. This can be achieved in the following way: according to the fifth weight coefficient and the sixth weight coefficient, the recall score and the ranking score corresponding to the recalled data sample are weighted summed to obtain the fusion score corresponding to the recalled data sample.

[0217] For example, if the recall score of the recalled data sample is greater than the first recall score threshold (such as the first recall score threshold = 0.8), it indicates that the recalled data sample is highly correlated with the first query data sample, and the weight of the recall score is increased. For example, the recall score is 0.9 (> 0.8), the preset parameters are: the first preset value = 0.1, the third weight coefficient = 0.4, the second preset value = 1.0, then the fifth weight coefficient = 0.1 + 0.4 = 0.5, the sixth weight coefficient = 1.0 - 0.5 = 0.5, and the fusion score = 0.5 × 0.9 (recall) + 0.5 × 0.85 (ranking) = 0.875.

[0218] For example, according to the calculation method preset for the interval, determining the weight coefficients corresponding to the recall score and the ranking score respectively can be achieved in the following way: in response to the recall score of the recalled data sample being less than the second recall score threshold, the second recall score threshold being less than the first recall score threshold, the difference between the third weight coefficient and the first preset value is used as the seventh weight coefficient, and the difference between the second preset value and the seventh weight coefficient is used as the eighth weight coefficient.

[0219] For example, according to the weight coefficients corresponding to the recall score and the ranking score, respectively, the weighted sum of the weight coefficients corresponding to the recall score and the ranking score is performed to obtain the fusion score of the recalled data sample. This can be achieved in the following way: according to the seventh weight coefficient and the eighth weight coefficient, the recall score and the ranking score corresponding to the recalled data sample are weighted summed to obtain the fusion score corresponding to the recalled data sample; wherein, the second recall score threshold is less than the first recall score threshold.

[0220] For example, if the recall score is less than the second recall score threshold (such as the second recall score threshold = 0.3), for example, the recall score: 0.2 (<0.3), the preset parameters: the first preset value = 0.1, the third weight coefficient = 0.4, the second preset value = 1.0, then the seventh weight coefficient = 0.4-0.1 = 0.3, the eighth weight coefficient = 1.0-0.3 = 0.7, the fusion score = 0.3×0.2 (recall) + 0.7×0.6 (sorting) = 0.48.

[0221] For example, according to the calculation method preset for the interval, determining the weight coefficients corresponding to the recall score and the ranking score respectively can be achieved in the following way: in response to the recall score of the recalled data sample being greater than or equal to the second recall score threshold and less than or equal to the first recall score threshold, the preset third weight coefficient and the preset fourth weight coefficient are used as the weight coefficients of the recall score and the ranking score corresponding to the recalled data sample, respectively.

[0222] For example, if the recall score is between two thresholds (e.g., 0.3 ≤ recall score ≤ 0.8), a fixed weight is used to balance recall and ranking. For example, if the recall score is 0.5 (in the interval [0.3, 0.8]), the default parameters are: the third weight coefficient = 0.4, the fourth weight coefficient = 0.6, then the fusion score = 0.4 × 0.5 (recall) + 0.6 × 0.7 (ranking) = 0.62.

[0223] By adaptively adjusting the weight distribution of recall and ranking scores according to the threshold range of the recall score - increasing the recall weight for high-confidence recall samples (score > 0.8), reducing the recall weight for low-confidence samples (score < 0.3) to rely on the ranking model to correct the bias, and using fixed weight balance for medium-confidence samples, the accuracy of the fusion score in representing the true correlation of the samples is improved, avoiding the dual problems of high-correlation samples being mistakenly downgraded due to fixed weighting and noise samples being overly dependent on the recall score, and ultimately achieving the core effect of enhancing the reliability and discriminability of the fusion score as a supervisory signal for fine-tuning the recall model.

[0224] In step 105 , a first loss value is determined based on the recall score and the fusion score.

[0225] In some embodiments, see Figure 4D , Figure 4A The step 105 shown can be implemented by following the steps 1051 to 1053, which are described in detail below.

[0226] In step 1051 , a first score distribution corresponding to a first query data sample is generated based on the recall score corresponding to each recalled data sample.

[0227] In some embodiments, the recall score of each recalled data sample corresponding to the first query data sample is normalized to obtain a first score distribution corresponding to the first query data sample.

[0228] For example, the Softmax function is used to normalize the recall score of each recalled data sample to obtain a first score distribution.

[0229] For example, suppose there are 3 recall data samples, and the recall scores are: sample A (0.8), sample B (0.6), sample C (0.4). The normalized scores after Softmax normalization are: sample A (0.402), sample B (0.329), sample C (0.269), then the first score distribution is expressed as: [sample A (0.402), sample B (0.329), sample C (0.269)].

[0230] In step 1052 , based on the fusion score corresponding to each recall data sample, a second score distribution corresponding to the first query data sample is generated.

[0231] In some embodiments, the fusion score of each recalled data sample corresponding to the first query data sample is normalized to obtain a second score distribution corresponding to the first query data sample.

[0232] For example, the Softmax function is used to normalize the fusion score of each recall data sample to obtain the second score distribution.

[0233] In step 1053 , a first loss value is determined based on the difference between the first score distribution and the second score distribution.

[0234] In some embodiments, the first loss value is determined based on the first score distribution and the second score distribution using a pre-set loss function (such as KL divergence).

[0235] For example, the first loss value can be determined by formula (4):

[0236]

[0237] Among them, D multi Denotes the second score distribution, D single Denotes the first score distribution, D multi (i) represents the normalized fusion score of the second score distribution, D single (i) represents the i-th normalized recall score in the first score distribution.

[0238] Through steps 1051 to 1053, a knowledge distillation mechanism of the recall model based on distribution alignment is realized. Specifically, the original recall score and the fusion score are normalized into probability distributions respectively, and the difference between the two is quantified using KL divergence to construct a loss function, forcing the recall model to learn the multi-model collaborative discriminant knowledge contained in the fusion score, thereby significantly improving the alignment accuracy of the independently generated scores of the fine-tuned recall model and the true correlation, and ultimately achieving the beneficial effect of migrating the fine-grained discriminant ability of the sorting model to the recall model at a lightweight computational cost.

[0239] In some other embodiments, determining the first loss value based on the recall score and the fusion score can also be achieved in the following manner:

[0240] First, a preset number of sample pairs are sampled from a plurality of recalled data samples, wherein each sample pair includes two recalled data samples.

[0241] For example, assume that 5 (preset number) sample pairs are randomly sampled from 100 recall data samples. The sample pairs and their corresponding fusion scores and recall scores are as follows: sample pair 1 (sample A: fusion 0.85 / recall 0.92, sample B: fusion 0.72 / recall 0.78), sample pair 2 (sample C: fusion 0.61 / recall 0.65, sample D: fusion 0.93 / recall 0.60), sample pair 3 (sample E: fusion 0.78 / recall 0.81, sample F: fusion 0.78 / recall 0.75), sample pair 4 (sample G: fusion 0.55 / recall 0.59, sample H: fusion 0.68 / recall 0.63), sample pair 5 (sample I: fusion 0.91 / recall 0.88, sample J: fusion 0.89 / recall 0.90).

[0242] Next, based on the fusion score of each recalled data sample in the sample pair, the first sorting relationship of each sample pair is determined, wherein the first sorting relationship includes a first positive order relationship and a first reverse order relationship. The first positive order relationship indicates that in the sample pair, the fusion score of the preceding recalled data sample is greater than the fusion score of the following recalled data sample. The first reverse order relationship indicates that in the sample pair, the fusion score of the preceding recalled data sample is less than the fusion score of the following recalled data sample.

[0243] Continuing with the above example, based on the fusion score, sample pairs 1 and 5 are in the first positive order relationship (the fusion score of the first sample is greater than that of the second sample), sample pairs 2 and 4 are in the first reverse order relationship, and sample pair 3 is ignored because of the same fusion score.

[0244] Next, based on the recall score of each recalled data sample in the sample pair, a second sorting relationship for each sample pair is determined, wherein the second sorting relationship includes a second positive order relationship and a second reverse order relationship, the second positive order relationship indicates that in the sample pair, the recall score of the preceding recalled data sample is greater than the recall score of the following recalled data sample, and the second reverse order relationship indicates that in the sample pair, the recall score of the preceding recalled data sample is less than the recall score of the following recalled data sample.

[0245] Continuing with the above example, based on the recall score, sample pairs 1, 2, and 3 are in the second positive order relationship, and sample pairs 4 and 5 are in the second reverse order relationship.

[0246] Finally, a first loss value is determined based on the difference between the first sorting relationship and the second sorting relationship.

[0247] For example, the first loss value can be determined by formula (5):

[0248]

[0249] Where M is the number of valid sample pairs (excluding sample pairs with equal fusion scores or recall scores), represents the indicator function. If the first sorting relationship is inconsistent with the second sorting relationship (for example, the sorting relationship corresponding to sample pair 42 is the first reverse order relationship and the second positive order relationship, then the first sorting relationship and the second sorting relationship of sample pair 2 are inconsistent), it is 1; otherwise, it is 0. |fusion score difference| represents the absolute value of the fusion score difference between the two recall data samples in the sample pair.

[0250] By sampling sample pairs to construct local ranking relationships, the fusion score and recall score are used to generate independent ranking relationships respectively, and the loss function is constructed with the fusion score ranking as the supervision signal and the absolute value of the fusion score difference as the weight, forcing the recall model to learn the relative correlation between fine-grained samples determined by the ranking model, thereby significantly improving the ranking consistency of the fine-tuned recall model in complex sample comparison scenarios, and ultimately achieving the beneficial effect of enhancing the recall model's robustness to high confusion candidate sets with low computational overhead.

[0251] In other embodiments, see Figure 4E Before updating the parameters of the recall model based on the first loss value to obtain the fine-tuned recall model, the following steps 401 to 403 may be performed, which are described in detail below.

[0252] In step 401 , a second loss value is determined based on the difference between the recall score and the fusion score corresponding to each recall data sample.

[0253] In some embodiments, the second loss value is determined based on the difference between the recall score and the fusion score corresponding to each recall data sample through a preset loss function (such as a cross entropy loss function, a mean square error loss function, etc.).

[0254] In step 402, the first loss value and the second loss value are weighted and summed according to the preset first weight coefficient and the second weight coefficient to obtain a third loss value.

[0255] In some embodiments, the first loss value and the second loss value are weighted and summed according to a preset first weight coefficient (such as 0.6) and a second weight (such as 0.4) to obtain a third loss value.

[0256] In step 403, the first loss value is updated based on the third loss value, and the updated first loss value is used to update the parameters of the recall model.

[0257] In some embodiments, the gradient information of each parameter of the recall model (the recall model trained from steps 201 to 204) with respect to the first loss value is obtained through a back-propagation algorithm, and the parameters of the recall model are updated using the obtained gradient information according to a gradient descent optimization algorithm (such as batch gradient descent, stochastic gradient descent, etc.). The above process is repeated until a certain number of iterations is reached or the recall model converges, thereby obtaining a fine-tuned recall model (or a recall model obtained after distillation learning).

[0258] In other embodiments, see Figure 4F , Figure 4A The illustrated step 105 can also be implemented through steps 1054 to 1057, which are described in detail below.

[0259] In step 1054, the fusion scores corresponding to each recalled data sample are sorted in descending order to obtain a fusion score sequence, and each fusion score in the fusion score sequence is replaced with the corresponding recall score to obtain a recall score sequence.

[0260] For example, if there are three recalled data samples A, B, and C, and their fusion scores are 0.85, 0.92, and 0.78 respectively, the fusion score sequence [0.92, 0.85, 0.78] is obtained after arranging them in descending order. The corresponding sample order is B, A, and C. Replace the fusion score at each position with the recall score of the corresponding recalled data sample (assuming that the recall scores of B, A, and C are 0.88, 0.90, and 0.75 respectively), then the recall score sequence is [0.88, 0.90, 0.75].

[0261] In step 1055 , a preset number of fusion scores are sequentially obtained from the fusion score sequence to form a first score sequence, and a preset number of recall scores are sequentially obtained from the recall score sequence to form a second score sequence.

[0262] Continuing with the example in step 1054, if the preset number is 2, the first two scores are taken from the fusion score sequence [0.92, 0.85, 0.78] to form a first score sequence [0.92, 0.85], and the first two scores are taken from the recall score sequence [0.88, 0.90, 0.75] to form a second score sequence [0.88, 0.90].

[0263] In step 1056 , the difference between the fusion score and the recall score of each identical position is obtained from the first score sequence and the second score sequence.

[0264] Continuing with the example in step 1055 , the difference values ​​of the same position of the first score sequence [0.92, 0.85] and the second score sequence [0.88, 0.90] are |0.92-0.88|=0.04 and |0.85-0.90|=0.05 respectively, resulting in a difference value sequence [0.04, 0.05].

[0265] In step 1057, the difference values ​​are weighted and summed according to the preset score weights to obtain a first loss value.

[0266] Continuing with the example in step 1056 , if the preset score weights are 0.6 and 0.4 (corresponding to two positions in the difference value sequence), the first loss value is 0.04×0.6+0.05×0.4=0.024+0.02=0.044.

[0267] It should be noted that the score weight corresponding to the front position is greater than the score weight corresponding to the back position.

[0268] Through steps 1054 to 1057, the recall score sequence is reconstructed based on the fusion score ranking, focusing on the head position to calculate the differentiated score deviation, and the recall model's score fidelity for high-value samples is enhanced through position-sensitive weighted loss (high in front and low in the back), thereby significantly improving the score prediction accuracy of the fine-tuned recall model in key head results, and ultimately achieving the effect of minimizing the error in the core ranking area to drive the improvement of the overall retrieval effect.

[0269] Continue to see Figure 4A In step 106, based on the first loss value, the parameters of the recall model are updated to obtain a fine-tuned recall model.

[0270] In some embodiments, the gradient information of each parameter of the recall model (the recall model trained from steps 201 to 204) with respect to the first loss value is obtained through a back-propagation algorithm, and the parameters of the recall model are updated using the obtained gradient information according to a gradient descent optimization algorithm (such as batch gradient descent, stochastic gradient descent, etc.). The above process is repeated until a certain number of iterations is reached or the recall model converges, thereby obtaining a fine-tuned recall model (or a recall model obtained after distillation learning).

[0271] Through steps 101 to 106, the ranking score generated by the ranking model is fused with the recall score to obtain a fusion score, and the first loss value is calculated based on the recall score and the fusion score. This achieves a more accurate judgment of the correlation between the first query data sample and the recall data sample (represented by the fusion score) obtained in the two stages of the recall model and the ranking model as a supervision signal, and reversely optimizes the parameters of the recall model, so that the fine-tuned recall model has the comprehensive capabilities of the two stages of recall and ranking. Therefore, the fine-tuned recall model can be used as the knowledge distillation result of the recall model and the ranking model before fine-tuning. On the basis of maintaining the simplicity of the original recall model structure and the low latency advantage, the collaborative optimization path of the recall and ranking stages is opened up, and the end-to-end retrieval accuracy is improved.

[0272] The following will combine the exemplary application and implementation of the server provided in the embodiment of the present application, taking the server as the execution subject, to illustrate the data processing method provided in the embodiment of the present application. Figure 5 , Figure 5 This is a flow chart of the data processing method provided in the embodiment of the present application, which will be combined with Figure 5 The steps shown are explained.

[0273] In step 501, query data is obtained.

[0274] In some embodiments, in response to a user's data retrieval operation, the server obtains a data retrieval request and parses relevant query data from the received data retrieval request. The query data includes but is not limited to query text, query image, or query text + query image, etc., wherein the retrieval request may also include voice data, and the voice data is converted into query text through automatic speech recognition (ASR), thereby performing data retrieval based on voice input.

[0275] In step 502, screening scores of multiple candidate data relative to the query data are determined through a distillation model, wherein the distillation model is a fine-tuned recall model obtained by the model training method provided in the embodiment of the present application.

[0276] In some embodiments, the similarity between the query data and each candidate data is obtained through a distillation model (the fine-tuned recall model trained from steps 101 to 106), and the similarity is used as a screening score.

[0277] In step 503 , target data for responding to the query data is determined from the plurality of candidate data based on the screening scores.

[0278] In some embodiments, candidate data ranked in the top K (eg, 3) screening scores are used as target data.

[0279] In the model training method in the related art, the recall model is trained by calculating the similarity score between the query data and the candidate data. Although the structure is simple and the reasoning is fast, there is a problem of inaccurate calculation of the similarity score (recall score), which makes it difficult to meet the needs of high-precision retrieval. In addition, although the two-stage cascade structure of "recall + rerank" (that is, first preliminarily screening the candidate set through the recall model, and then using the ranking model for refined scoring) can improve the retrieval effect, the need to train and maintain two independent models (recall model and ranking model) increases the difficulty of deployment and optimization. In addition, the two-stage serial calculation increases the overall reasoning time, making it difficult to adapt to application scenarios with high real-time requirements.

[0280] In view of this, an embodiment of the present application provides a model training method (model training stage) and a data processing method (model reasoning stage), which are described in detail below.

[0281] The model training method and data processing method provided in the embodiments of this application can be used in multimodal retrieval scenarios, such as retrieving similar videos, article contents, etc. Figure 6 , Figure 6 This is a schematic diagram of the application flow of the data processing method provided by the embodiment of the present application. The query data (query) can be text, image, text + image, etc. The candidate pool (candidates) (consisting of multiple candidate data) can include candidates of all modalities, such as text, image, text + image, etc. When a query is entered, the task instruction is used to indicate the retrieval task to be performed (such as using a prompt word as an instruction, and the prompt word clearly states the task content), such as retrieving similar text, similar text + image, and retrieving academic materials related to the question.

[0282] The present invention proposes a model training method based on a multimodal retrieval framework of recall + reranking, which distills the two-stage recall + reranking into a single retrieval model (recall model). Figure 7 , Figure 7This is a schematic diagram of the training principle of the model training method provided in the embodiment of the present application. The recall model (Recall Model) is used to train samples (such as Figure 7 The images and texts shown in the figure (where the samples include query data and candidate data) are used to extract features, and preliminary retrieval results (represented by recall scores) are obtained through similarity calculation. The top-k retrieval results are then fed into the ranking model (Rerank Model). After a second scoring (i.e., ranking scores are generated for the recalled data), the final retrieval results are obtained (i.e., the final retrieval results are determined based on the ranking scores). In order to integrate the two-stage model into a single model structure, the embodiment of the present application uses a knowledge distillation method to migrate the capabilities of the two-stage model to a single model (corresponding to the fine-tuned recall model obtained in step 106 above), thereby improving the effect of the single model and reducing the time consumed in inference.

[0283] The following is a detailed description of the implementation details of the model training method provided in the embodiments of the present application.

[0284] 1. Recall Model (corresponding to the process of training the recall model from steps 201 to 204 above)

[0285] The recall model uses the MLLM model as a pre-trained model, and then fine-tunes the model through the retrieval task to enable it to have the ability to extract multimodal embeddings. The MLLM model structure is as follows Figure 7 As shown, it includes three parts: image encoder (VisionEncoder), projection layer (Projector), and large language model (LLM). Vision Encoder (such as ViT, etc.) is used to extract visual features, Projector is used to align the visual modality to the text modality, and LLM processes the features of the visual model and text modality to finally obtain a multimodal Embedding output (for example, in step 203 above, when the second query data sample is multimodal data, the encoding features of the second query data sample).

[0286] It should be noted that in machine learning and natural language processing (NLP), embedding (feature vector) is a method of converting high-dimensional data (such as words, sentences, or images) into a low-dimensional vector representation. This representation retains the semantic information of the data and makes it suitable for input to machine learning models.

[0287] (1) Input side (the input data here can be query data or candidate data, and both query data and candidate data can be single-modal or multi-modal, which is not limited in this embodiment of the application):

[0288] For the Computer Vision (CV) part (i.e., input data for image modality), the image is input into the Vision Encoder (such as ViT) for processing. If the input also includes data in the text modality, the features output by the Vision Encoder are aligned through the alignment module (i.e., Projector).

[0289] For example, see Figure 8A , Figure 8A This is a schematic diagram of the first structure of the recall model provided in the embodiment of the present application. The image is input to the ViT model, and the image is divided into multiple patches (corresponding to the pixel blocks above, corresponding to Figure 8A The image blocks in the image are input into the ViT model for processing. For each patch, the ViT model extracts its corresponding Embedding (corresponding Figure 8A The first embedded feature of the image in Figure 8A The projection layer shown in ), maps the image Embedding to the same dimension as the text TokenEmbedding, as CV Token Embedding (corresponding to Figure 8A ).

[0290] For the natural language processing (NLP) part (i.e., input data for text modality), first, the input text is tokenized, for example, the text is mapped to TokenEmbedding through Tokenizer (corresponding to the embedding encoding processing of multiple words above to obtain embedding features), the TokenEmbedding of the text is spliced ​​together with the previously extracted CVTokenEmbedding, and the spliced ​​TokenEmbedding is input into the large language model (LLM).

[0291] (2) Structural side of the recall model:

[0292] See also Figure 8B , Figure 8B This is the second structural diagram of the recall model provided by the embodiment of the present application, which changes the unidirectional attention mechanism of the generative model to a bidirectional attention mechanism to better adapt to the feature extraction task. Specifically, the original Attention mask of the generative model is a lower triangle (the lower triangle is 1, and the others are 0) (corresponding to Figure 8B The unidirectional attention mechanism shown in ) makes the current token only see the previous token information, but cannot see the following token information. Change it to the all-1 form (corresponding to Figure 8B(The bidirectional attention mechanism shown in []) is used so that each token can see the information of all tokens, which is more in line with the task form of feature extraction.

[0293] (3) Output side:

[0294] Take the average (Mean Pooling) of the Embeddings output by the large language model (corresponding to the pooling layer shown in []) to obtain the final Embedding. Figure 8B

[0295] (4) Training stage:

[0296] During the process of training the recall model, the parameters of the Vision Encoder and Projector modules remain unchanged, and only the LLM module is trained using the LoRA method. Using contrastive learning as the training objective, the loss function can use the Info-NCE function (corresponding to the loss function shown in formula (1) above).

[0297] (5) Inference stage: Use the recall model to extract the features of the query and all candidate data respectively, and then calculate the similarity between the features of the query and all candidate data to obtain the similarity score (corresponding to the recall score S(5) Inference stage: Use the recall model to extract the features of the query and all candidate data respectively, and then calculate the similarity between the features of the query and all candidate data to obtain the similarity score (corresponding to the recall score S recall )

[0298] II. Ranking model (corresponding to the process of training the ranking model in steps 301 to 306 above)

[0299] Take the top-k candidate results obtained by the recall model as candidate samples and send them into the ranking model for further fine ranking. The ranking model still uses the MLLM model, and the training method adopts the autoregressive method, that is, predicting the next token.

[0300] (1) Input side:

[0301] Concatenate the query with one of the candidate samples and add a prompt word to ask whether they match, so that the large language model outputs yes or no. For example, it is expressed as "query: <query information> candidate: <candidate information>. Is the above candidate data sample matching the query sample (the third query data sample)? Only output yes or no, where 'yes' is represented by the numerical value '1' and 'no' is represented by the numerical value '0'", "query: <query information> candidate: <candidate information>. Is the above candidate data sample matching the query sample (the third query data sample)? Output a ranking score between [0.1] to quantify the matching degree. The higher the ranking score, the higher the matching degree between the two" and so on.

[0302] (2) Model side:

[0303] You can use the MLLM model without changing the structure and train the MLLM model using the LoRA method.

[0304] (3) Output side:

[0305] Use the Next-Token Prediction method to train the model and determine whether its output is "yes" or "no".

[0306] (4) Training phase:

[0307] During training, the parameters of the Vision Encoder and Projector modules remain unchanged; only the LLM module is trained using the LoRA method. Positive and negative samples are collected for each query, with negative samples randomly selected from the top-k candidates. The ratio of positive to negative samples is 1:1. To further improve the discriminative ability of the ranking model, the difficulty of negative samples is gradually increased. For example, in the initial training stage, k is set to 100, meaning one negative sample is randomly selected from the top 100 candidates. In later stages, k is reduced to, for example, 20, meaning one negative sample is randomly selected from the top 20 candidates. This progressive application of difficult negative sample mining gradually enhances the ranking model's ability to distinguish between positive and negative samples.

[0308] (5) Reasoning stage:

[0309] The model combines the query and each candidate result in turn and sends them to the ranking model. The ranking model outputs the probability value corresponding to "yes" or "no" as a score (i.e., the ranking score S rerank ), and obtain the ranking score of each candidate result. In order to further improve the robustness of the score, the scores of the two stages are fused: the score of the recall model in the first stage is S rerank , the score of the second stage ranking model is S rerank , then the final score (fusion score) is S multi =a×S recall +(1-a)×S rerank Where a is the fusion coefficient. The fusion score obtained by fusing the two-stage scores has higher accuracy and can make the retrieval results more reliable.

[0310] 3. Single Model Optimization Based on Knowledge Distillation

[0311] This section describes how to distill the two-stage recall-rerank model into a single model, giving it capabilities close to those of the two-stage model while significantly reducing system complexity and inference time.

[0312] The above two-stage model structure has high system complexity because it needs to maintain the recall and ranking models. At the same time, during inference, it is necessary to first use the recall model to obtain candidate results, and then send each candidate result to the ranking model for scoring. K candidate results require the ranking model to infer K times, and the inference time is relatively high.

[0313] In order to solve the above problems, an embodiment of the present application provides a model training method to migrate the retrieval capability of the two-stage model to a single recall model.

[0314] The core idea of ​​the model training method provided in the embodiment of the present application is:

[0315] The quality score (S) generated by the two-stage model multi ) as a supervisory signal;

[0316] Through the comparative learning framework, a single model can learn more refined sorting capabilities;

[0317] KL divergence loss is used to ensure the effectiveness of knowledge transfer.

[0318] (1) Data preparation:

[0319] First, obtain the query (corresponding to the first query data sample above) and the score S of each candidate sample multi (two-stage fusion score), and then S multi The candidate samples are ranked from high to low (excluding positive samples, which are pre-set) and serve as difficult negative samples for training the recall model. It's important to note that not all candidate samples (corresponding to the candidate data samples mentioned above) are used together with the query to obtain a two-stage score. This is because the ranking model, as a point-wise model, only takes a query and a candidate sample as input for each inference. Assuming there are N queries and a candidate pool of size M, each query requires the ranking model to infer M times, for a total of N*M times. N and M are often large, such as N = 100,000 and M = 1,000,000. In comparison, the recall model only needs to extract N+M features and then calculate similarity. Therefore, it is not feasible to run each query and all candidates through the ranking model. Instead, only the top-100 recall model results (corresponding to the recall data samples mentioned above) are taken. The query and the top-100 recall model results are then run through the ranking model separately to obtain a fusion score for each recall data sample.

[0320] (2) Model training phase:

[0321] The original recall model (corresponding to the pre-trained recall model mentioned above) serves as the basic architecture of the student model. The parameters of the Vision Encoder and Projector modules remain unchanged. Only the LLM module is trained using LoRA, continuing training based on the original recall model.

[0322] For each query, construct a training batch containing positive samples and difficult negative samples. For example, for each query, take its positive samples and the top-50 negative samples. Unlike the random negative samples used in recall model training, difficult negative samples are used here to improve the distillation model's ability to distinguish difficult samples.

[0323] The fusion score of each sample is softmax normalized to obtain the target probability distribution (corresponding to the second score distribution above).

[0324] The KL divergence loss function is used to minimize the difference between the student model output (corresponding to the first score distribution above) and the target probability distribution. Similar to contrastive learning methods, the student model calculates the cosine similarity between the query and each positive and negative sample, then performs softmax normalization, and then performs KL divergence with the target probability distribution. The loss function can be seen in the description of formula (4) above.

[0325] (3) Reasoning stage (corresponding to the processing of steps 501 to 503 above):

[0326] The reasoning of the fine-tuned model is the same as that of the recall model. A single model is used to extract features of the query and all candidate pools respectively, and then similarity is calculated between the query and all candidate pool features to obtain a similarity score.

[0327] Through the above method, the student model (the fine-tuned recall model) can achieve the following beneficial effects:

[0328] (1) Accurately distinguish positive samples from difficult negative samples;

[0329] (2) maintain similar sorting capabilities as the two-stage model;

[0330] (3) Complete high-quality retrieval in a single forward propagation.

[0331] To sum up, the embodiment of the present application realizes multimodal retrieval through a unified feature extraction model based on a retrieval framework that integrates recall and reranking, and performs refined scoring of the recall results in combination with a special ranking model. By adopting an intelligent score fusion strategy and integrating the scoring results of the two stages, the retrieval accuracy is significantly improved while maintaining high computational efficiency. Through the knowledge distillation method, the retrieval capability of the two-stage cascade model is migrated to a single model, so that a single model (the fine-tuned recall model) has the comprehensive capabilities of recall and reranking, improves retrieval accuracy, and reduces the computational overhead brought by model cascading, reduces inference latency, and is suitable for high concurrency scenarios.

[0332] In short, the model training method provided in the embodiments of the present application greatly simplifies the model structure while maintaining high-precision retrieval, reduces the model deployment and maintenance costs, and specifically solves the problems of insufficient accuracy of single-stage recall models and low efficiency of two-stage models in related technologies. It optimizes computational efficiency while ensuring retrieval results, and is suitable for complex multimodal retrieval tasks.

[0333] The following continues to describe the exemplary structure of the model training device 133 provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 2A As shown, the software modules stored in the model training device 133 of the memory 130-1 may include:

[0334] The data processing module 1331 is configured to determine the recall scores of multiple candidate data samples in the database relative to the first query data sample using a pre-trained recall model.

[0335] In some embodiments, the data processing module 1331 is further configured to determine a plurality of recalled data samples from the plurality of candidate data samples based on the recall scores.

[0336] In some embodiments, the data processing module 1331 is further configured to determine a ranking score of each of the recalled data samples relative to the first query data sample using a pre-trained ranking model.

[0337] In some embodiments, the data processing module 1331 is further configured to fuse the recall score and the ranking score to obtain a fusion score for each of the recalled data samples.

[0338] In some embodiments, the data processing module 1331 is further configured to determine a first loss value based on the recall score and the fusion score.

[0339] The training module 1332 is used to update the parameters of the recall model based on the first loss value to obtain a fine-tuned recall model.

[0340] In some embodiments, the data processing module 1331 is also used to generate a first score distribution corresponding to the first query data sample based on the recall score corresponding to each of the recalled data samples; generate a second score distribution corresponding to the first query data sample based on the fusion score corresponding to each of the recalled data samples; and determine the first loss value based on the difference between the first score distribution and the second score distribution.

[0341] In some embodiments, the data processing module 1331 is further used to sample a preset number of sample pairs from the multiple recalled data samples, wherein each of the sample pairs includes two recalled data samples; based on the fusion score of each of the recalled data samples in the sample pair, determine a first sorting relationship for each of the sample pairs, wherein the first sorting relationship includes a first positive order relationship and a first reverse order relationship, the first positive order relationship indicating that in the sample pair, the fusion score of the preceding recalled data sample is greater than the fusion score of the succeeding recalled data sample, and the first reverse order relationship indicating that in the sample pair, the fusion score of the preceding recalled data sample is less than the fusion score of the succeeding recalled data sample. The first loss value is determined based on the difference between the first sorting relationship and the second sorting relationship.

[0342] In some embodiments, the data processing module 1331 is also used to determine a second loss value based on the difference between the recall score and the fusion score corresponding to each of the recall data samples; perform weighted summation on the first loss value and the second loss value according to a preset first weight coefficient and a second weight coefficient to obtain a third loss value; based on the third loss value, update the first loss value, and the updated first loss value is used to update the parameters of the recall model.

[0343] In some embodiments, the data processing module 1331 is further used to sort the fusion scores corresponding to each of the recalled data samples in order from large to small to obtain a fusion score sequence, and replace each of the fusion scores in the fusion score sequence with the corresponding recall score to obtain a recall score sequence; sequentially obtain a preset number of the fusion scores from the fusion score sequence to combine into a first score sequence, and sequentially obtain the preset number of the recall scores from the recall score sequence to combine into a second score sequence; obtain the difference value between the fusion score and the recall score at each same position in the first score sequence and the second score sequence; and perform weighted summation on the difference values ​​according to a preset score weight to obtain the first loss value.

[0344] In some embodiments, the data processing module 1331 is also used to obtain positive samples and negative samples corresponding to the second query data sample, wherein the positive sample is a sample having a semantic association with the second query data sample, and the negative sample is a sample having no semantic association with the second query data sample, and the second query data sample is different from the first query data sample; obtain the similarity between the second query data sample and the positive sample through the recall model to be trained as a first recall score sample, and obtain the similarity between the second query data sample and the negative sample as a second recall score sample; determine a fourth loss value based on the first recall score sample and the second recall score sample; and update the parameters of the recall model to be trained based on the fourth loss value to obtain a pre-trained recall model.

[0345] In some embodiments, the data processing module 1331 is further used to obtain a third query data sample, and a positive sample corresponding to the third query data sample, wherein the positive sample is a sample that matches the semantics of the third query data sample; determine the negative sample corresponding to the third query data sample from multiple candidate negative samples through the pre-trained recall model, wherein the negative sample is a sample that does not match the semantics of the third query data sample; encode the preset prompt word, the positive sample and the third query data sample through the sorting model to be trained to obtain a first encoding feature, and encode the prompt word, the negative sample and the third query data sample to obtain a first encoding feature. The third query data sample is encoded to obtain a second encoding feature, wherein the prompt word is used to indicate the task content of the sorting task that the sorting model to be trained needs to perform; the first encoding feature is decoded by the sorting model to be trained to obtain a first sorting score sample, and the second encoding feature is decoded to obtain a second sorting score sample; based on the first sorting score sample, the labeling score corresponding to the positive sample, the second sorting score sample and the labeling score corresponding to the negative sample, a fifth loss value is determined; based on the fifth loss value, the parameters of the sorting model to be trained are updated to obtain a pre-trained sorting model.

[0346] In some embodiments, updating the parameters of the ranking model to be trained is achieved through multiple training stages, and the data processing module 1331 is also used to obtain a third recall score sample of each candidate negative sample relative to the third query data sample through the recall model; in the first training stage of the ranking model to be trained, based on the third recall score sample, a first number of candidate negative samples are obtained from the multiple candidate negative samples to combine into a first candidate negative sample set; the candidate negative samples are selected from the first candidate negative sample set as the negative samples; in the second training stage of the ranking model to be trained, based on the third recall score sample, a second number of candidate negative samples are obtained from the multiple candidate negative samples to combine into a second candidate negative sample set; wherein, the second number is less than the first number; the candidate negative samples are selected from the second candidate negative sample set as the negative samples.

[0347] In some embodiments, the data processing module 1331 is also used to perform the following processing for each of the recalled data samples: according to a preset third weight coefficient and a preset fourth weight coefficient, the recall score and the ranking score corresponding to the recalled data sample are weightedly summed to obtain the fusion score corresponding to the recalled data sample.

[0348] In some embodiments, the data processing module 1331 is also used to perform the following processing for each of the recalled data samples: determining the interval in which the recall score of the recalled data sample is located in multiple preset recall score intervals; determining the weight coefficients corresponding to the recall score and the ranking score respectively according to the calculation method preset for the interval; and performing weighted summation on the weight coefficients corresponding to the recall score and the ranking score respectively according to the weight coefficients corresponding to the recall score and the ranking score respectively, to obtain the fusion score of the recalled data sample.

[0349] In some embodiments, the data processing module 1331 is also used to, in response to the recall score of the recalled data sample being greater than the first recall score threshold, use the sum of the first preset value and the third weight coefficient as the fifth weight coefficient, and use the difference between the second preset value and the fifth weight coefficient as the sixth weight coefficient; in response to the recall score of the recalled data sample being less than the second recall score threshold, and the second recall score threshold being less than the first recall score threshold, use the difference between the third weight coefficient and the first preset value as the seventh weight coefficient, and use the difference between the second preset value and the seventh weight coefficient as the eighth weight coefficient; in response to the recall score of the recalled data sample being greater than or equal to the second recall score threshold, and less than or equal to the first recall score threshold, use the preset third weight coefficient and the preset fourth weight coefficient as the weight coefficients of the recall score and the ranking score corresponding to the recalled data sample, respectively.

[0350] In some embodiments, the data processing module 1331 is also used to perform at least one of the following processing: sorting the multiple candidate data samples in descending order according to the recall score, and selecting the first N candidate data samples in the sorting result as the multiple recalled data samples; screening candidate data samples with the same modality as the first query data sample from the multiple candidate data samples as target data samples, and sorting each target data sample in descending order according to the recall score, and selecting the first N target data samples in the sorting result as the multiple recalled data samples; sorting the candidate data samples of each modality in descending order according to the recall score, and selecting the first N candidate data samples in the sorting result corresponding to each modality as the multiple recalled data samples.

[0351] The following continues to describe the exemplary structure of the data processing device 134 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2B As shown, the software modules stored in the data processing device 134 of the memory 130-2 may include:

[0352] The data acquisition module 1341 is used to acquire query data.

[0353] The data processing module 1342 is used to determine the screening scores of multiple candidate data relative to the query data through a distillation model, wherein the distillation model is a fine-tuned recall model obtained by the model training method provided in the embodiment of the present application.

[0354] In some embodiments, the data processing module 1342 is further configured to determine target data for responding to the query data from the plurality of candidate data based on the screening scores.

[0355] The present invention provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the model training method or data processing method described in the present invention.

[0356] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the model training method or data processing method provided in the embodiment of the present application, for example, Figure 4A The model training method shown, or Figure 5 The data processing method is shown.

[0357] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0358] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0359] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0360] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0361] To sum up, through the embodiments of the present application, the ranking score generated by the ranking model is fused with the recall score to obtain a fusion score, and the first loss value is calculated based on the recall score and the fusion score, thereby achieving a more accurate judgment of the correlation between the first query data sample and the recall data sample (represented by the fusion score) obtained in the two stages of the recall model and the ranking model as a supervision signal, and reversely optimizing the parameters of the recall model, so that the fine-tuned recall model has the comprehensive capabilities of the two stages of recall and ranking, so that the fine-tuned recall model can be used as the knowledge distillation result of the recall model and the ranking model before fine-tuning. On the basis of maintaining the simplicity of the original recall model structure and the low latency advantage, the collaborative optimization path of the recall and ranking stages is opened up, and the end-to-end retrieval accuracy is improved.

[0362] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Determining, by a pre-trained recall model, recall scores of a plurality of candidate data samples in the database relative to the first query data sample; determining a plurality of recalled data samples from the plurality of candidate data samples based on the recall scores; Determining a ranking score of each of the recalled data samples relative to the first query data sample using a pre-trained ranking model; fusing the recall score and the ranking score to obtain a fusion score for each of the recalled data samples; determining a first loss value based on the recall score and the fusion score; Based on the first loss value, the parameters of the recall model are updated to obtain a fine-tuned recall model.

2. The method according to claim 1, characterized in that The determining a first loss value based on the recall score and the fusion score includes: generating a first score distribution corresponding to the first query data sample based on the recall score corresponding to each of the recalled data samples; generating a second score distribution corresponding to the first query data sample based on the fusion score corresponding to each of the recalled data samples; The first loss value is determined based on a difference between the first score distribution and the second score distribution.

3. The method according to claim 1, characterized in that The determining a first loss value based on the recall score and the fusion score includes: Sampling a preset number of sample pairs from the plurality of recalled data samples, wherein each of the sample pairs includes two of the recalled data samples; Determining a first sorting relationship for each of the sample pairs based on the fusion score of each of the recalled data samples in the sample pairs, wherein the first sorting relationship includes a first positive order relationship and a first reverse order relationship, the first positive order relationship indicating that in the sample pair, the fusion score of the preceding recalled data sample is greater than the fusion score of the succeeding recalled data sample, and the first reverse order relationship indicating that in the sample pair, the fusion score of the preceding recalled data sample is less than the fusion score of the succeeding recalled data sample; Determining a second sorting relationship for each of the sample pairs based on the recall score of each of the recalled data samples in the sample pairs, wherein the second sorting relationship includes a second positive order relationship and a second reverse order relationship, the second positive order relationship indicating that in the sample pair, the recall score of the preceding recalled data sample is greater than the recall score of the succeeding recalled data sample, and the second reverse order relationship indicating that in the sample pair, the recall score of the preceding recalled data sample is less than the recall score of the succeeding recalled data sample; The first loss value is determined based on a difference between the first ranking relationship and the second ranking relationship.

4. The method according to claim 3, characterized in that Before updating the parameters of the recall model based on the first loss value to obtain a fine-tuned recall model, the method further includes: determining a second loss value based on a difference between the recall score and the fusion score corresponding to each of the recalled data samples; Performing a weighted summation of the first loss value and the second loss value according to a preset first weight coefficient and a second weight coefficient to obtain a third loss value; Based on the third loss value, the first loss value is updated, and the updated first loss value is used to update the parameters of the recall model.

5. The method according to claim 1, wherein The determining a first loss value based on the recall score and the fusion score includes: Sorting the fusion scores corresponding to each of the recalled data samples in descending order to obtain a fusion score sequence, and replacing each of the fusion scores in the fusion score sequence with the corresponding recall score to obtain a recall score sequence; Sequentially acquiring a preset number of the fusion scores from the fusion score sequence to combine into a first score sequence, and sequentially acquiring the preset number of the recall scores from the recall score sequence to combine into a second score sequence; Obtaining, from the first score sequence and the second score sequence, a difference value between the fusion score and the recall score at each identical position; The difference values ​​are weighted and summed according to a preset score weight to obtain the first loss value.

6. The method according to claim 1, characterized in that Before determining the recall scores of the plurality of candidate data samples in the database relative to the first query data sample using the pre-trained recall model, the method further includes: Obtaining positive samples and negative samples corresponding to a second query data sample, wherein the positive sample is a sample having a semantic association with the second query data sample, the negative sample is a sample having no semantic association with the second query data sample, and the second query data sample is different from the first query data sample; Obtaining, by the recall model to be trained, a similarity between the second query data sample and the positive sample as a first recall score sample, and obtaining a similarity between the second query data sample and the negative sample as a second recall score sample; determining a fourth loss value based on the first recall score sample and the second recall score sample; Based on the fourth loss value, the parameters of the recall model to be trained are updated to obtain a pre-trained recall model.

7. The method according to any one of claims 1 to 6, characterized in that Before determining the ranking score of each of the recall data samples relative to the first query data sample using the pre-trained ranking model, the method further includes: Obtaining a third query data sample and a positive sample corresponding to the third query data sample, wherein the positive sample is a sample that semantically matches the third query data sample; Determining, by using the pre-trained recall model, a negative sample corresponding to the third query data sample from a plurality of candidate negative samples, wherein the negative sample is a sample that does not semantically match the third query data sample; encoding the preset prompt word, the positive sample, and the third query data sample using the ranking model to be trained to obtain a first encoding feature, and encoding the prompt word, the negative sample, and the third query data sample to obtain a second encoding feature, wherein the prompt word is used to indicate the task content of the ranking task to be performed by the ranking model to be trained; Decoding the first encoding feature using the ranking model to be trained to obtain a first ranking score sample, and decoding the second encoding feature to obtain a second ranking score sample; Determining a fifth loss value based on the first ranking score sample, the labeling score corresponding to the positive sample, the second ranking score sample, and the labeling score corresponding to the negative sample; Based on the fifth loss value, the parameters of the sorting model to be trained are updated to obtain a pre-trained sorting model.

8. The method according to claim 7, characterized in that Updating the parameters of the ranking model to be trained is achieved through multiple training stages; Determining a negative sample corresponding to the third query data sample from a plurality of candidate negative samples using the pre-trained recall model includes: Obtaining, by the recall model, a third recall score sample of each candidate negative sample relative to the third query data sample; In a first training phase of the ranking model to be trained, based on the third recall score sample, obtaining a first number of the candidate negative samples from the plurality of candidate negative samples to combine them into a first candidate negative sample set; Selecting the candidate negative sample from the first candidate negative sample set as the negative sample; In a second training phase of the ranking model to be trained, based on the third recall score sample, obtaining a second number of the candidate negative samples from the plurality of candidate negative samples to combine them into a second candidate negative sample set; wherein the second number is smaller than the first number; The candidate negative sample is selected from the second candidate negative sample set as the negative sample.

9. The method according to any one of claims 1 to 6, characterized in that The fusing of the recall score and the ranking score to obtain a fusion score for each of the recalled data samples includes: The following processing is performed for each of the recalled data samples: According to a preset third weight coefficient and a preset fourth weight coefficient, a weighted sum is performed on the recall score and the ranking score corresponding to the recalled data sample to obtain the fusion score corresponding to the recalled data sample.

10. The method according to any one of claims 1 to 6, characterized in that The fusing of the recall score and the ranking score to obtain a fusion score for each of the recalled data samples includes: The following processing is performed for each of the recalled data samples: Determining the interval in which the recall score of the recalled data sample lies among a plurality of preset recall score intervals; Determining weight coefficients corresponding to the recall score and the ranking score respectively according to a preset calculation method for the interval; According to the weight coefficients corresponding to the recall score and the ranking score respectively, a weighted sum is performed on the weight coefficients corresponding to the recall score and the ranking score respectively to obtain a fusion score of the recalled data sample.

11. The method according to claim 10, characterized in that Determining the weight coefficients corresponding to the recall score and the ranking score respectively according to a calculation method preset for the interval includes: In response to the recall score of the recalled data sample being greater than a first recall score threshold, using a sum of a first preset value and a third weight coefficient as a fifth weight coefficient, and using a difference between a second preset value and the fifth weight coefficient as a sixth weight coefficient; In response to the recall score of the recalled data sample being less than a second recall score threshold, and the second recall score threshold being less than the first recall score threshold, using a difference between the third weight coefficient and the first preset value as a seventh weight coefficient, and using a difference between the second preset value and the seventh weight coefficient as an eighth weight coefficient; In response to the recall score of the recalled data sample being greater than or equal to the second recall score threshold and less than or equal to the first recall score threshold, the preset third weight coefficient and the preset fourth weight coefficient are respectively used as the weight coefficients of the recall score and the ranking score corresponding to the recalled data sample.

12. The method according to any one of claims 1 to 6, characterized in that The determining a plurality of recalled data samples from the plurality of candidate data samples based on the recall scores comprises: Perform at least one of the following actions: Sorting the multiple candidate data samples in descending order according to the recall scores, and selecting the top N candidate data samples in the sorting result as the multiple recalled data samples; Selecting candidate data samples of the same modality as the first query data sample from the multiple candidate data samples to serve as target data samples, and sorting each of the target data samples in descending order according to the recall score, and selecting the top N target data samples in the sorting result as the multiple recalled data samples; The candidate data samples of each modality are sorted in descending order according to the recall scores, and the top N candidate data samples in the sorting results corresponding to each modality are selected as the multiple recalled data samples.

13. A data processing method, characterized in that: The method comprises: Get query data; Determining screening scores of a plurality of candidate data relative to the query data by using a distillation model, wherein the distillation model is a fine-tuned recall model obtained by the method according to any one of claims 1 to 12; Target data for responding to the query data is determined from the plurality of candidate data based on the screening scores.

14. A model training device, characterized in that: The device comprises: a data processing module, configured to determine, by using a pre-trained recall model, recall scores of a plurality of candidate data samples in the database relative to the first query data sample; The data processing module is further configured to determine a plurality of recalled data samples from the plurality of candidate data samples based on the recall scores; The data processing module is further configured to determine a ranking score of each of the recalled data samples relative to the first query data sample using a pre-trained ranking model; The data processing module is further configured to fuse the recall score and the ranking score to obtain a fusion score for each of the recalled data samples; The data processing module is further configured to determine a first loss value based on the recall score and the fusion score; A training module is used to update the parameters of the recall model based on the first loss value to obtain a fine-tuned recall model.

15. A data processing device, characterized in that: The device comprises: Data acquisition module, used to obtain query data; a data processing module, configured to determine, by using a distillation model, screening scores of a plurality of candidate data relative to the query data, wherein the distillation model is a fine-tuned recall model obtained by the method according to any one of claims 1 to 12; The data processing module is further configured to determine target data for responding to the query data from the plurality of candidate data based on the screening scores.

16. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions or computer programs; The processor is used to implement the model training method described in any one of claims 1 to 12, or the data processing method described in claim 13 when executing the computer-executable instructions or computer programs stored in the memory.

17. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer-executable instructions or computer program are executed by a processor, the model training method according to any one of claims 1 to 12 is implemented, or the data processing method according to claim 13 is implemented.

18. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer-executable instructions or computer program are executed by a processor, the model training method described in any one of claims 1 to 12 is implemented, or the data processing method described in claim 13 is implemented.

Images