Model training method, device, electronic device and computer-readable storage medium

Through the gradual training and knowledge distillation of student models by multiple teacher models, the problems of overfitting and resource consumption in neural network model training are solved, and high-performance model training is achieved.

CN118627571BActive Publication Date: 2024-11-22TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410933763.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-22
Estimated Expiration
2044-07-12

AI Technical Summary

Technical Problem

Existing neural network models are prone to overfitting during training, and complex models have high requirements for hardware resources, and simplified models have shortcomings in performance and computing storage.

Method used

By obtaining multiple teacher models with different performances, gradually train the student models to be trained, use the teacher models with gradually enhanced performance to distillate knowledge, gradually improve the performance of the student models, and reduce the consumption of computing resources and storage space.

Benefits of technology

It realizes that while keeping the model parameters small, the performance of student models is improved, the computing resources and storage space requirements for model training are reduced, and overfitting is avoided.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118627571B_ABST
    Figure CN118627571B_ABST
Patent Text Reader

Abstract

The present application provides a model training method, device, electronic device and computer-readable storage medium; the method comprises: obtaining a student model to be trained and N teacher models, wherein the model performances of the teacher models are different and N is a positive integer greater than 1; training the student model to be trained by a first teacher model to obtain a student model after the first training; training the student model after the n-1th training by an nth teacher model to obtain a student model after the nth training; and determining the student model after the Nth training as a trained student model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to a model training method, device, electronic device and computer-readable storage medium. Background Art

[0002] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0003] Model training is one of the core technologies in the field of artificial intelligence. It refers to the use of large amounts of data through algorithm iterative optimization, so that computer models can simulate the human learning process, thereby realizing automated processing of specific tasks. In related technologies, complex neural network models with many parameters usually have higher accuracy. However, this also means that the model is larger in size and has relatively high requirements for hardware equipment, which often puts pressure on storage and computing resources. Although simplified models with fewer parameters have lower requirements for storage and computing resources, are smaller in size, and are easier to deploy and accelerate the reasoning process, such models often sacrifice certain performance and are prone to overfitting when faced with complex data distributions. Summary of the invention

[0004] The embodiments of the present application provide a model training method, device, electronic device and computer-readable storage medium, which can improve the performance of model training.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application embodiment provides a model training method, including:

[0007] Obtain a student model to be trained and N teacher models, wherein the model performances of the teacher models are different, and N is a positive integer greater than 1;

[0008] Training the student model to be trained by a first teacher model to obtain a student model after the first training, wherein the first teacher model is a teacher model with the lowest model performance among the N teacher models;

[0009] The student model after the n-1th training is trained by the nth teacher model to obtain the student model after the n-1th training, wherein n is a positive integer that increases successively, 1<n≤N, the student model after the n-1th training is obtained by training the n-1th teacher model, the nth teacher model and the n-1th teacher model are any teacher models among the N teacher models, and the model performance of the nth teacher model is higher than the model performance of the n-1th teacher model;

[0010] The student model after the Nth training is determined as the trained student model.

[0011] The present application provides a model training device, including:

[0012] An acquisition module, used to acquire a student model to be trained and N teacher models, wherein the model performances of the teacher models are different, and N is a positive integer greater than 1;

[0013] A distillation module, used for training the student model to be trained by using a first teacher model to obtain a student model after the first training, wherein the first teacher model is a teacher model with the lowest model performance among the N teacher models;

[0014] The distillation module is further used to train the student model after the n-1th training through the nth teacher model to obtain the student model after the n-1th training, wherein n is a positive integer that increases successively, 1<n≤N, the student model after the n-1th training is obtained by training the n-1th teacher model, the nth teacher model and the n-1th teacher model are any teacher model among the N teacher models, and the model performance of the nth teacher model is higher than the model performance of the n-1th teacher model;

[0015] The determination module is used to determine the student model after the Nth training as the trained student model.

[0016] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0017] A memory for storing computer executable instructions;

[0018] The processor is used to implement the model training method provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0019] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, can implement the model training method provided in the embodiment of the present application.

[0020] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, can implement the model training method provided in the embodiment of the present application.

[0021] The embodiments of the present application have the following beneficial effects:

[0022] The student model to be trained is trained by N teacher models with gradually enhanced model performance, so that the student model to be trained gradually absorbs the knowledge from the teacher model with gradually enhanced performance during the training process, and performs gradual incremental learning based on the existing knowledge, thereby ensuring that although the trained student model has a smaller parameter scale than the teacher model, it still maintains high performance, thereby greatly reducing the consumption of computing resources and storage space for model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a structural diagram of the system architecture of the model training method provided in the embodiment of the present application;

[0024] Figure 2 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0025] Figure 3A It is a first flow chart of the model training method provided in the embodiment of the present application;

[0026] Figure 3B It is a second flow chart of the model training method provided in the embodiment of the present application;

[0027] Figure 3C It is a third flow chart of the model training method provided in the embodiment of the present application;

[0028] Figure 3D It is a fourth flow chart of the model training method provided in the embodiment of the present application;

[0029] Figure 3E It is a fifth flow chart of the model training method provided in the embodiment of the present application;

[0030] Figure 3F It is a sixth flow chart of the model training method provided in the embodiment of the present application;

[0031] Figure 3G It is a seventh flow chart of the model training method provided in the embodiment of the present application;

[0032] Figure 4 is a model application flow chart provided in an embodiment of the present application;

[0033] Figure 5 is a schematic diagram of the dual encoder structure provided in an embodiment of the present application;

[0034] Figure 6 is a schematic diagram of the cross encoder structure provided in an embodiment of the present application;

[0035] Figure 7 It is a schematic diagram of the progressive knowledge distillation process provided by the embodiment of the present application;

[0036] Figure 8 It is a flowchart of a text retrieval application provided in an embodiment of the present application.

[0037] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0039] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0040] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0041] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0042] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0043] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0044] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0045] 1) Knowledge distillation: Knowledge distillation is a neural network model compression technology that aims to transfer the knowledge of a large and complex teacher model to a smaller student model so that the student model can achieve similar performance to the teacher model at a lower computational cost. The basic idea of ​​knowledge distillation is to train a small student model to imitate a large, already trained teacher model.

[0046] 2) Hard Loss Function: Hard loss function refers to calculating the loss using the true label, which measures the difference between the model output and the true label. The true label is also called hard label, which is a clear and definite label or classification marked on the training sample.

[0047] 3) Soft Loss Function: In knowledge distillation, the soft loss function refers to the loss function that uses the prediction of the teacher model as the soft label to train the student model. It measures the difference between the output of the teacher model and the output of the student model. In the field of knowledge distillation, the soft label is generated by the teacher model and represents the possible classification probability distribution of the teacher model for the input data. That is, the soft label is not a single, definite label, but a continuous probability distribution that includes the correlation between categories.

[0048] 4) Contrastive Learning: Contrastive learning refers to a learning method that identifies and distinguishes relevant positive sample texts from other irrelevant negative sample texts, where the positive sample pairs are similar or related data, while the negative sample pairs are dissimilar or unrelated data.

[0049] 5) Dense Latent Semantic Space: It is a technology used in natural language processing and other information retrieval applications. It improves the efficiency of information organization by capturing latent and abstract semantic relationships in data.

[0050] 6) Dense Vector Retrieval: It is an information retrieval technology based on dense vector representation. In natural language processing and other fields, dense vector retrieval mainly focuses on how to effectively store and retrieve large-scale dense vector data to support complex query and analysis tasks.

[0051] 7) Hidden layer vector dimension: The hidden layer vector dimension refers to the dimension of the vector output by each neuron in the hidden layer of the neural network. The size of the hidden layer vector dimension determines the complexity of the features that the neural network model can learn when processing input data. The hidden layer is one or more layers between the input layer and the output layer of the neural network. It does not directly interact with the input data or output results, but is only responsible for extracting and transforming features in the data.

[0052] 8) Generalization Ability: One of the criteria used to measure model performance. The generalization ability of a model refers to the ability of the model to maintain good performance when faced with new or unseen data. For example, a model used to identify cats has good generalization ability if it can correctly identify cats in pictures of cats that have never been seen outside the training set.

[0053] 9) Explainability: One of the criteria used to measure model performance, the explainability of a model refers to the extent to which the model's decision-making process and results can be understood and explained by humans. For example, a decision tree is a highly explainable machine learning model. Each decision node in the decision tree represents a test of a feature, and each branch represents a decision result. Humans can intuitively understand how the model makes decisions based on the value of the feature.

[0054] 10) Adaptability: One of the criteria used to measure model performance. Model adaptability refers to the ability of the model to maintain its performance in different environments, with different data, or on different tasks. For example, during the training phase of the model, diverse training data can help the model learn richer features, thereby improving its adaptability.

[0055] The embodiments of the present application provide a model training method, device, electronic device, computer-readable storage medium and computer program product, which can improve the performance of model training. The model training method provided in the embodiments of the present application can be applied to various fields, for example, it can be applied to the field of text processing, such as text vector retrieval and virtual human question-answering system, etc., it can also be applied to the field of image processing, such as image recognition and image classification, and it can also be applied to the field of audio processing, such as audio recognition, etc., which are not limited here.

[0056] The model training method provided in the embodiment of the present application can be implemented by the terminal alone; it can also be implemented jointly by the terminal and the server, for example, the terminal alone undertakes the model training method described below, or the terminal sends a training request for the student model to the server, and the server executes the model training method according to the received training request for the student model.

[0057] The electronic device provided in the embodiments of the present application may be various types of terminal devices or servers, wherein the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms; the terminal may be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0058] Taking the server as an example, it can be a server cluster deployed in the cloud, opening artificial intelligence cloud services (AI as a Service, AIaaS) to users. The AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to an AI theme mall. All users can access one or more artificial intelligence services provided by the AIaaS platform through an application programming interface.

[0059] For example, one of the artificial intelligence cloud services may be a model training service, that is, a server in the cloud is encapsulated with a model training program provided by an embodiment of the present application. The server deployed in the cloud calls the encapsulated model training program, obtains a student model to be trained and N teacher models, and trains the student model to be trained through each of the N teacher models, and determines the student model after the Nth training as a trained student model, where N is a positive integer greater than 1.

[0060] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of the model training system 10 provided in an embodiment of the present application, wherein the terminal 200 is connected to the server 100 via a network 300, wherein the network 300 may be a wide area network or a local area network, or a combination of the two.

[0061] Terminal 200 may be used to obtain a training request for a model to be trained.

[0062] In some embodiments, a model training plug-in may be implanted in the client running in the terminal to implement the model training method locally on the client. For example, the terminal 200 calls the model training plug-in to implement the model training method, obtains the student model to be trained and N teacher models, trains the student model to be trained through each of the N teacher models, and determines the student model after the Nth training as the trained student model, where N is a positive integer greater than 1.

[0063] It is worth noting that the trained student model can be stored locally in the terminal 200 and called when needed.

[0064] In some embodiments, after the terminal 200 obtains a training request for the student model, it calls the model training interface of the server 100 (which can be provided in the form of a cloud service, i.e., a model training service). The server 100 implements the model training method through a model training plug-in, obtains the student model to be trained and N teacher models, and trains the student model to be trained through each of the N teacher models, and determines the student model after the Nth training as the trained student model, where N is a positive integer greater than 1.

[0065] It is worth noting that the trained student model can be stored locally on the server 100 and called when needed.

[0066] In some embodiments, the terminal or server can implement the model training method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a live broadcast application; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0067] In some embodiments, server 100 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms. Among them, the cloud service can be a model training service for terminals to call.

[0068] In some embodiments, multiple servers may form a blockchain, and server 100 is a node on the blockchain. Information connections may exist between each node in the blockchain, and information may be transmitted between nodes through the above information connections. Among them, data related to the model training method provided in the embodiment of the present application (such as model training logic, trained student model) may be stored on the blockchain.

[0069] The structure of the electronic device for model training provided in the embodiment of the present application is described below. Figure 2 , Figure 2 is a structural diagram of an electronic device 500 for model training provided in an embodiment of the present application, taking the electronic device 500 as a server as an example, Figure 2 The electronic device 500 for model training shown includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 2 Various buses are labeled as bus system 540 .

[0070] The processor 510 may be an integrated circuit chip having signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor may be a microprocessor or any conventional processor, etc.

[0071] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiment of the present application is intended to include any suitable type of memory. The memory 550 may optionally include one or more storage devices that are physically far away from the processor 510.

[0072] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0073] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0074] A network communication module 552, used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0075] In some embodiments, the model training device provided in the embodiments of the present application can be implemented in software, for example, it can be the model training service in the server described above, or it can be the model training plug-in in the terminal described above. Of course, it is not limited to this, and the model training device provided in the embodiments of the present application can be provided as various software embodiments, including various forms including application programs, software, software modules, scripts or codes.

[0076] Figure 2 A model training device 555 stored in a memory 550 is shown, which may be software in the form of a program and a plug-in, such as a model training plug-in, and includes a series of modules, including an acquisition module 5551, a distillation module 5552, a determination module 5553, an encoding module 5554, and a pre-training module 5555. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0077] In other embodiments, the model training device provided in the embodiments of the present application can be implemented in hardware. As an example, the model training device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the model training method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0078] As mentioned above, the model training method provided in the embodiments of the present application can be implemented by various types of electronic devices. Figure 3A , Figure 3A This is a first flow chart of the model training method provided in the embodiment of the present application, combined with Figure 3A The steps shown are explained.

[0079] In the following steps, the student model can be a neural network model for text encoding, or a neural network model for image encoding, or a neural network model for audio encoding, which is not limited here; correspondingly, the teacher model can be a neural network model for text encoding, or a neural network model for image encoding, or a neural network model for audio encoding, which is not limited here.

[0080] In step 101, a student model to be trained and N teacher models are obtained.

[0081] As an example, the model performance of the teacher model is different, N is a positive integer greater than 1, the model processing field of the student model to be trained and the N teacher models are the same, and the model processing fields include: text processing field, image processing field and audio processing field, that is, the student model and the teacher model to be trained can be neural network models for the text processing field, or neural network models for the image processing field, or neural network models for the audio processing field, which are not limited here, wherein the model performance of the teacher model is higher than the model performance of the student model to be trained.

[0082] For example, when N is equal to 2, a student model to be trained and two teacher models are obtained, the first teacher model can be a dual encoder for the field of text processing, the second teacher model can be a cross encoder for the field of text processing, and the student model to be trained can be a dual encoder for the field of text processing, wherein the model performance of the first teacher model is not the same as the performance of the second teacher model, and the model performance of the first teacher model and the second teacher model are both higher than the student model to be trained.

[0083] In some embodiments, model performance includes at least one of the accuracy, precision, inference speed, generalization ability, interpretability and adaptability of the model. The difference in model performance can be reflected by different types of models and different parameter scales of the models. For two models with the same model parameter scale, different types of models correspond to different model performances. The model performances of the two models are compared by at least one of the accuracy, precision, inference speed, generalization ability, interpretability and adaptability of the models. For example, if the accuracy of the cross encoder is better than that of the dual encoder, then the model performance of the cross encoder is better than that of the dual encoder; for two models of the same model type, the performance of the model with a larger parameter scale is higher than that of the model with a relatively smaller parameter scale, wherein the determining factors of the parameter scale of the model include the number of layers of the model and the hidden layer vector dimension, that is, the parameter scale of the model is proportional to the number of layers of the model and the hidden layer vector dimension. The more layers of the model, the higher the hidden layer vector dimension, and the larger the parameter scale of the model.

[0084] In step 102, the student model to be trained is trained by using the first teacher model to obtain the student model after the first training.

[0085] As an example, the student model to be trained is trained by the first teacher model to update the parameters of the student model to be trained, and then the student model after the first training is obtained, wherein the first teacher model is the teacher model with the lowest model performance among the N teacher models, and the model performance of the student model to be trained is lower than that of the first teacher model, wherein the difference in model performance can be reflected by different model types and different model parameter scales.

[0086] For example, the model performance of the cross encoder with the same parameter scale is better than that of the dual encoder, and the model performance of the cross encoder with a relatively large parameter scale is better than that of the cross encoder with a relatively small parameter scale.

[0087] As an example, the above-mentioned "the first teacher model is the teacher model with the lowest model performance among N teacher models, and the model performance of the student model to be trained is lower than that of the first teacher model" can be achieved in the following ways: when the model parameter scales of the first teacher model and the student model to be trained are the same, the accuracy, precision, reasoning speed, generalization ability, interpretability and adaptability of the first teacher model are higher than the accuracy, precision, reasoning speed, generalization ability, interpretability and adaptability of the student model to be trained; when the model types of the first teacher model and the student model to be trained are the same, the parameter scale of the first teacher model is larger than the parameter scale of the student model to be trained.

[0088] For example, when the parameter scale of the student model to be trained and the first teacher model is the same, the student model to be trained can be a 6-layer dual encoder, and the first teacher model can be a 6-layer cross encoder; when the model type of the student model to be trained and the first teacher model are the same (the student model to be trained and the first teacher model are both dual encoders, or the student model to be trained and the first teacher model are both cross encoders), the parameter scale of the teacher model (any one of the N teacher models including the first teacher model) can be 2 to 10 times the parameter scale of the student model to be trained, that is, the parameter scale of the teacher model is greater than or equal to 2 times the parameter scale of the student model to be trained, and the parameter scale of the teacher model is less than or equal to 10 times the parameter scale of the student model to be trained, the student model to be trained is a 6-layer encoder, the first teacher model can be a 12-layer encoder, the second teacher model can be a 24-layer encoder, and the Nth teacher model can be a 60-layer encoder.

[0089] The embodiment of the present application trains the student model through the teacher model, transferring the knowledge in the teacher model to the student model, so that the student model can learn the complex feature representation obtained by training a large amount of data in the teacher model, thereby improving the prediction performance and generalization ability of the student model on unknown data.

[0090] In step 103, the student model after the n-1th training is trained by the nth teacher model to obtain the student model after the nth training.

[0091] Among them, the student model after the n-1th training is obtained by training the n-1th teacher model, the nth teacher model and the n-1th teacher model are any teacher models among the N teacher models, and the model performance of the nth teacher model is higher than the model performance of the n-1th teacher model, wherein n is a positive integer that increases successively, 1<n≤N.

[0092] For example, see Figure 7 , Figure 7 is a schematic diagram of the progressive knowledge distillation process provided by the embodiment of the present application. When N is equal to 3, that is, when there are three teacher models, the first teacher model (i.e. Figure 7 The teacher model in 1) treats the trained student model (i.e. Figure 7 The first knowledge distillation is performed to obtain the student model after the first training. Figure 7 The teacher model 2 in the first training is used to perform a second knowledge distillation on the student model after the first training to obtain the student model after the second training. Figure 7The teacher model in 3) performs a third knowledge distillation on the student model after the second training to obtain the student model after the third training, among which the model performance of the third teacher is higher than that of the second teacher model, the model performance of the second teacher model is higher than that of the first teacher model, and the model performance of the first teacher model is higher than that of the student model to be trained.

[0093] In some embodiments, when the parameter scales of the nth teacher model and the n-1th teacher model are the same, the accuracy, precision, reasoning speed, generalization ability, interpretability and adaptability of the nth teacher model are higher than those of the n-1th teacher model; when the model types of the nth teacher model and the n-1th teacher model are the same, the parameter scale of the nth teacher model is larger than the parameter scale of the n-1th teacher model, and the parameter scale of the nth teacher model may be 2 to 3 times the parameter scale of the n-1th teacher model, that is, the parameter scale of the nth teacher model is greater than or equal to 2 times the parameter scale of the n-1th teacher model, and the parameter scale of the nth teacher model is less than or equal to 3 times the parameter scale of the n-1th teacher model.

[0094] As an example, when the parameter scale of the nth teacher model and the n-1th teacher model is the same, the nth teacher model can be a 6-layer cross encoder, and the n-1th teacher model can be a 6-layer dual encoder; when the model type of the nth teacher model and the n-1th teacher model is the same, the n-1th teacher model is a 6-layer dual encoder, and the nth teacher model can be any one of a 12-layer dual encoder to an 18-layer dual encoder, thereby ensuring that the model performance of adjacent teacher models does not differ significantly, avoiding the problem of too large a span of model performance and the inability to achieve progressive knowledge distillation, so that the student model gradually absorbs knowledge from different teacher models, thereby ensuring that the student model can perform more effective incremental learning based on existing knowledge.

[0095] For example, the first teacher model can be a dual encoder composed of twelve cascaded text encoding layers, the second teacher model can be a cross encoder composed of twelve cascaded text encoding layers, and the third teacher model can be a cross encoder composed of twenty-four cascaded text encoding layers. The model performance of the cross encoder is higher than the model performance of the dual encoder. When the parameter scale of the second teacher model and the first teacher model is the same, the model performance of the second teacher model is higher than the model performance of the first teacher model; when the third teacher model and the second teacher model are both cross encoders, the number of model layers of the third teacher model is greater than the number of model layers of the second teacher model, that is, the parameter scale of the third teacher model is greater than the parameter scale of the second teacher model, and the model performance of the third teacher model is higher than the model performance of the second teacher model.

[0096] The embodiment of the present application uses a teacher model with multiple model performances enhanced in sequence to perform multi-stage progressive knowledge distillation on the student model, so that the student model gradually absorbs knowledge from different teacher models, and then performs incremental learning based on the existing knowledge, which effectively promotes the improvement of the model performance of the student model.

[0097] In some embodiments, see Figure 3B , Figure 3B is a second flow chart of the model training method provided in an embodiment of the present application, Figure 3A The illustrated step 103 can be implemented by following the steps 201 to 204 , which are described in detail below.

[0098] In step 201, the first training sample is encoded by the nth teacher model to obtain the encoding result of the nth teacher model.

[0099] As an example, the first training sample includes a first anchor sample, a first positive sample and a first negative sample. The first anchor sample, the first positive sample and the first negative sample are respectively preprocessed and encoded by the nth teacher model to obtain the feature vector of the first anchor sample, the feature vector of the first positive sample and the feature vector of the first negative sample, and the feature vector of the first anchor sample, the feature vector of the first positive sample and the feature vector of the first negative sample are used as the encoding result of the nth teacher model, wherein the first negative sample can be a negative sample data or a negative sample list, and the first training sample can use a manually annotated data set, such as T2Ranking, mMARCO data set, which is not limited here.

[0100] For example, the first training sample is ,in, represents the first anchor sample, represents the first positive sample, is the first negative sample, which may include multiple sub-negative samples, expressed as ,in, is the number of sub-negative samples contained in the first negative sample; or, the first training sample is ,in, represents the first anchor sample, represents the first positive sample, is the first negative sample, which is a negative sample data.

[0101] For example, in the field of text processing, the anchor sample is the text "What are synonyms?", the positive sample is the text "A synonym is a word that has the same meaning as another word", and the negative sample is a text list including sub-negative sample 1, sub-negative sample 2, and sub-negative sample 3, where sub-negative sample 1 is the text "Remove synonyms, top Removing synonyms (another way of saying remove) is to eliminate, withdraw, and take away"; sub-negative sample 2 is the text "Synonyms, crossword answers, and other related words for concrete We hope that the following list of synonyms for the word concrete will help you complete today's crossword puzzle. We have arranged the synonyms in length order to make them easier to find"; sub-negative sample 3 is the text "Synonyms for broccoli: broccoli, cabbage, cauliflower, mustard, kale, watercress, cruciferous plants".

[0102] In some embodiments, when both the n-1th teacher model and the nth teacher model are dual encoders, see Figure 3C , Figure 3C is a third flow chart of the model training method provided in the embodiment of the present application, Figure 3B The illustrated step 201 may be implemented by following steps 2011A to 2013A, which are described in detail below.

[0103] In step 2011A, data preprocessing is performed on the first anchor sample, the first positive sample, and the first negative sample, respectively, to obtain an input sequence of the first anchor sample, an input sequence of the first positive sample, and an input sequence of the first negative sample.

[0104] As an example, data preprocessing is performed on the first anchor sample, the first positive sample, and the first negative sample, respectively, and the first anchor sample, the first positive sample, and the first negative sample are respectively converted into an input sequence of the first anchor sample, an input sequence of the first positive sample, and an input sequence of the first negative sample that can be understood and directly processed by the model.

[0105] As an example, in the field of text processing, performing data preprocessing on the first anchor sample, the first positive sample, and the first negative sample includes performing word segmentation processing and indexing processing on the first anchor sample, the first positive sample, and the first negative sample.

[0106] For example, taking the data preprocessing of the first anchor sample as an example, the first anchor sample is the text "first anchor sample". First, a special symbol "cls" is added to the first anchor sample to represent the beginning of the sentence, and "cls first anchor sample" is obtained. Then, the first anchor sample is segmented to obtain the segmentation sequence of the first anchor sample "first", "anchor", and "sample". The index value corresponding to each segmentation is queried as follows: , get the input sequence of the first anchor point sample .

[0107] As an example, the first training sample is ,in, represents the first anchor sample, represents the first positive sample, The first negative sample is the first negative sample (the first negative sample here is a negative sample data), and the first anchor sample, the first positive sample and the first negative sample are preprocessed to obtain the input sequence of the first anchor sample. , the input sequence of the first positive sample and multiple input sequences of the first negative sample ,in, The number of word segments of the first anchor point sample obtained by word segmentation processing is The number of word segments of the first positive sample obtained by performing word segmentation processing on the first positive sample is The number of word segments of the first negative sample obtained by performing word segmentation processing on the first negative sample.

[0108] In an embodiment of the present application, by performing data preprocessing on the first anchor sample, the first positive sample and the first negative sample respectively, the first anchor sample, the first positive sample and the first negative sample are converted into data that can be directly processed by the model, thereby improving the computational efficiency and stability of the model, which is an indispensable part of building a high-quality neural network model.

[0109] In step 2012A, the input sequence of the first anchor sample, the input sequence of the first positive sample, and the input sequence of the first negative sample are encoded by the nth teacher model to obtain the feature vector of the first anchor sample, the feature vector of the first positive sample, and the feature vector of the first negative sample.

[0110] As an example, the nth teacher model includes M cascaded encoding layers, and the first training sample is ,in, represents the first anchor sample, represents the first positive sample, For the first negative sample, the first anchor sample is encoded by the M cascaded encoding layers included in the nth teacher model The input sequence Perform encoding processing to obtain the feature vector of the first anchor sample ; The first positive sample is encoded by the M cascaded encoding layers included in the nth teacher model The input sequence Perform encoding processing to obtain the feature vector of the first positive sample ; The first negative sample is encoded by the M cascaded encoding layers included in the nth teacher model The input sequence of each sub-negative sample is encoded to obtain the feature vector set of the first negative sample ,in, is the number of sub-negative samples in the first negative sample.

[0111] The teacher model as a dual encoder in the embodiment of the present application has the ability to individually encode the input training samples. Before knowledge distillation, the first positive sample and the first negative sample can be pre-encoded. During the knowledge distillation process, only the first anchor point sample needs to be encoded. By pre-encoding the first positive sample and the first negative sample, the encoding efficiency of the teacher model is significantly improved.

[0112] In some embodiments, encoding processing of the input sequence of the first anchor sample is implemented through M cascaded encoding layers included in the nth teacher model. The encoding processing of the input sequence of the first anchor sample can be implemented in the following manner: encoding processing of the input sequence of the first anchor sample is performed through the first encoding layer of the M cascaded encoding layers to obtain the output features of the first encoding layer; encoding processing of the output features of the m-1th encoding layer is performed through the mth encoding layer to obtain the output features of the mth encoding layer, and based on the output features of the Mth encoding layer, the feature vector of the first anchor sample is determined, wherein m is a successively increasing positive integer, 1<m≤M, and M is a positive integer greater than 1.

[0113] The specific steps for encoding the first positive and the first negative samples are consistent with those for encoding the first anchor sample, and are both implemented through the M cascaded encoding layers included in the nth teacher model. Here, the encoding processing steps of the first anchor sample are used as an example to illustrate.

[0114] As an example, the input sequence of the first anchor sample is input into the first coding layer of the nth teacher model to obtain the output features of the first coding layer; the output features of the first coding layer are input into the second coding layer of the nth teacher model to obtain the output features of the second coding layer, and so on. For the last coding layer (Mth coding layer) of the nth teacher model, the output features of the M-1th coding layer are input into the last coding layer of the nth teacher model to obtain the output features of the last coding layer. Based on the output features of the last coding layer, the feature vector of the first anchor sample is determined, wherein the output features of the last coding layer can be directly used as the feature vector of the first anchor sample, or the feature vector of the first anchor sample can be obtained by performing pooling processing on the output features of the last coding layer. The pooling processing includes category pooling processing and pooling processing, which are not limited here.

[0115] For example, the output features of the last coding layer are subjected to category pooling (cls pooling), and among the multiple output features of the last coding layer, the output vector corresponding to the special symbol "cls" is used as the feature vector of the first anchor sample.

[0116] As an example of a coding layer, each coding layer in the embodiment of the present application includes a multi-head self-attention layer, a feedforward neural network layer, a residual connection layer and a normalization layer, wherein the feedforward neural network layer may include two fully connected layers and an activation function. "The first coding layer in the M cascaded coding layers encodes the input sequence of the first anchor sample to obtain the output features of the first coding layer" can be implemented in the following ways: based on the multi-head self-attention layer of the first coding layer, the input sequence of the first anchor point sample is processed with attention to obtain the output features of the multi-head self-attention layer; based on the feedforward neural network layer, the output features of the multi-head self-attention layer are linearly transformed to obtain the output features of the feedforward neural network layer; based on the residual connection layer, the output features of the feedforward neural network layer are processed to obtain the output features of the residual connection layer; based on the normalization layer, the output features of the residual connection layer are normalized to obtain the output features of the first coding layer.

[0117] The encoding layer in the embodiment of the present application is composed of a feedforward neural network layer and a multi-head self-attention layer, wherein the feedforward neural network layer is dedicated to mining the local features of the training samples, while the multi-head self-attention layer focuses on detecting the interdependencies between the input sample sequences, thereby enabling the teacher model including M cascaded encoding layers to simultaneously focus on the subtle correlations between the local features of information at different positions of the input sequence. Especially in the field of text processing, the teacher model can effectively capture the long-range dependencies in sentences.

[0118] In step 2013A, the feature vector of the first anchor sample, the feature vector of the first positive sample, and the feature vector of the first negative sample are used as the encoding result of the nth teacher model.

[0119] As an example, the process of encoding the input sequence of the first positive sample and the process of encoding the input sequence of the first negative sample are consistent with the process of encoding the input sequence of the first anchor sample in step 2012A above, and will not be repeated here. After completing the encoding process of the input sequence of the first positive sample and the encoding process of the input sequence of the first negative sample, the feature vector of the first positive sample and the feature vector of the first negative sample are obtained, and then, the feature vector of the first anchor sample, the feature vector of the first positive sample and the feature vector of the first negative sample are used as the encoding results of the nth teacher model.

[0120] In some embodiments, when the n-1th teacher model is a dual encoder and the nth teacher model is a cross encoder, see Figure 3D , Figure 3D is a fourth flow chart of the model training method provided in an embodiment of the present application, Figure 3B The illustrated step 201 may be implemented by following steps 2011B to 2014B, which are described in detail below.

[0121] In step 2011B, the first anchor sample and the first positive sample are concatenated to obtain a first concatenated sample, and the first anchor sample and the first negative sample are concatenated to obtain a second concatenated sample.

[0122] As an example, in the field of text processing, the first anchor sample and the first positive sample are spliced ​​by a special delimiter to obtain the splicing result of the first anchor sample and the first positive sample, and a special symbol is added before the splicing result after the first anchor sample and the first positive sample are spliced ​​to obtain the first spliced ​​sample; the first anchor sample and the first negative sample are spliced ​​by a special delimiter to obtain the splicing result of the first anchor sample and the first negative sample, and a special symbol is added before the sample after the splicing of the first anchor sample and the first negative sample to obtain the second spliced ​​sample.

[0123] For example, in the field of text processing, the first anchor point sample is the text "first anchor point sample", the first positive sample is the text "first positive sample", and the first negative sample is the text "first negative sample". The first spliced ​​sample obtained by splicing the first anchor sample and the first positive sample with the special separator "sep" and the special symbol "cls" is "cls first anchor point sample sep first positive sample", and the second spliced ​​sample obtained by splicing the first anchor sample and the first negative sample is "cls first anchor point sample sep first negative sample".

[0124] In step 2012B, data preprocessing is performed on the first spliced ​​sample and the second spliced ​​sample respectively to obtain an input sequence of the first spliced ​​sample and an input sequence of the second spliced ​​sample.

[0125] As an example, the specific process can refer to step 2011A. The process of performing data preprocessing on the first spliced ​​sample (or the second spliced ​​sample) is consistent with the process of performing data preprocessing on the first anchor sample in step 2011A, and will not be repeated here.

[0126] In step 2013B, the input sequence of the first spliced ​​sample and the input sequence of the second spliced ​​sample are respectively encoded by the nth teacher model to obtain a feature vector of the first spliced ​​sample and a feature vector of the second spliced ​​sample.

[0127] As an example, encoding processing of the input sequence of the first spliced ​​sample and the input sequence of the second spliced ​​sample is implemented through M cascaded encoding layers included in the nth teacher model. The specific process can be seen in step 2012A, which will not be repeated here.

[0128] In step 2014B, the feature vector of the first concatenated sample and the feature vector of the second concatenated sample are used as encoding results of the nth teacher model.

[0129] In an embodiment of the present application, when the teacher model is a cross-encoder, the teacher model concatenates the first anchor sample and the first positive sample (or the first negative sample) into one sample to capture the subtle interaction between the first anchor sample and the first positive sample. For example, in the field of text processing, the cross-encoder can capture the fine-grained semantic interaction and semantic association between the first anchor sample and the first positive sample.

[0130] In some embodiments, when both the n-1th teacher model and the nth teacher model are cross encoders, Figure 3B The illustrated step 201 can be implemented by the following steps: splicing the first anchor sample and the first positive sample to obtain a first spliced ​​sample, and splicing the first anchor sample and the first negative sample to obtain a second spliced ​​sample; performing data preprocessing on the first spliced ​​sample and the second spliced ​​sample respectively to obtain an input sequence of the first spliced ​​sample and an input sequence of the second spliced ​​sample; encoding the input sequence of the first spliced ​​sample and the input sequence of the second spliced ​​sample respectively through the nth teacher model to obtain a feature vector of the first spliced ​​sample and a feature vector of the second spliced ​​sample; and using the feature vector of the first spliced ​​sample and the feature vector of the second spliced ​​sample as the encoding result of the nth teacher model.

[0131] As an example, when the n-1th teacher model and the nth teacher model are both cross encoders, the encoding processing steps of the first training sample by the nth teacher model can refer to steps 2011B to 2014B, which will not be repeated here.

[0132] In step 202, the first training sample is encoded using the student model after the n-1th training to obtain an encoding result of the student model after the n-1th training.

[0133] As an example, the model types of the student model include a dual encoder and a cross encoder. When the student model after the n-1th training is a dual encoder, refer to the process of encoding the first training sample by the teacher model in steps 2011A to 2013A, which is not repeated here. The encoding result of the student model after the n-1th training includes the feature vector of the first anchor sample of the student model after the n-1th training, the feature vector of the first positive sample of the student model after the n-1th training, and the feature vector of the first negative sample of the student model after the n-1th training; when the student model after the n-1th training is a cross encoder, refer to the process of encoding the first training sample by the teacher model in steps 2011B to 2014B, which is not repeated here. The encoding result of the student model after the n-1th training includes the feature vector of the first spliced ​​sample of the student model after the n-1th training and the feature vector of the second spliced ​​sample of the student model after the n-1th training.

[0134] In step 203, a total loss function is constructed based on the encoding result of the student model after the n-1th training and the encoding result of the nth teacher model.

[0135] As an example, the total loss function includes a hard loss function and a soft loss function, wherein the hard loss function is used to calculate the loss value between the output result of the student model after the n-1th training and the true label, and the soft loss function is used to calculate the loss value between the output result of the student model after the n-1th training and the output result of the nth teacher model.

[0136] In some embodiments, see Figure 3E , Figure 3E is a fifth flow chart of the model training method provided in the embodiment of the present application, Figure 3B The illustrated step 203 may be implemented by following steps 2031 to 2033, which are described in detail below.

[0137] In step 2031, a hard loss function is constructed based on the encoding result of the student model after the n-1th training.

[0138] As an example, based on the encoding results of the student model after the n-1th training, the similarity scores between the first anchor point sample and the first positive sample, and the similarity scores between the first anchor point sample and the first negative sample are calculated, and then based on the similarity scores between the first anchor point sample and the first positive sample, and the similarity scores between the first anchor point sample and the first negative sample, a hard loss function is constructed.

[0139] Taking the similarity score between the first anchor point sample and the first positive sample as an example, when the student model after the n-1th training is a dual encoder, the first similarity is obtained by performing similarity processing on the feature vector of the first anchor sample of the student model after the n-1th training and the feature vector of the first positive sample of the student model after the n-1th training, and then the first similarity is used as the similarity score between the first anchor point sample and the first positive sample; when the student model after the n-1th training is a cross encoder, the similarity score between the first anchor point sample and the first positive sample is determined based on the linear layer of the student model after the n-1th training.

[0140] In some embodiments, when the student model after the n-1th training is a dual encoder, the encoding result of the student model after the n-1th training includes a feature vector of a first anchor sample of the student model after the n-1th training, a feature vector of a first positive sample of the student model after the n-1th training, and a feature vector of a first negative sample of the student model after the n-1th training, Figure 3EThe shown step 2031 can be implemented in the following manner: perform similarity processing on the feature vector of the first anchor sample of the student model after the n-1th training and the feature vector of the first positive sample of the student model after the n-1th training to obtain a first similarity; perform similarity processing on the feature vector of the first anchor sample of the student model after the n-1th training and the feature vector of the first negative sample of the student model after the n-1th training to obtain a second similarity; determine the hard loss function based on the first similarity and the second similarity.

[0141] As an example, when the student model after the n-1th training is a dual encoder, the first similarity is the similarity score between the first anchor point sample and the first positive sample, and the second similarity is the similarity score between the first anchor point sample and the first negative sample. The calculation methods of the first similarity and the second similarity may be the same or different.

[0142] Taking the first similarity as an example, the first similarity can be obtained by calculating the cosine similarity, Euclidean distance, Manhattan distance, etc. of the feature vector of the first anchor sample of the student model after the n-1th training and the feature vector of the first positive sample of the student model after the n-1th training, which is not limited here.

[0143] Taking the cosine similarity of the feature vector of the first anchor sample of the student model after the n-1th training and the feature vector of the first positive sample of the student model after the n-1th training as an example, the first similarity is calculated by formula (1.1): :

[0144] (1.1)

[0145] in, To scale the temperature coefficient, is the feature vector of the first anchor sample of the student model after the n-1th training, is the feature vector of the first positive sample of the student model after the n-1th training.

[0146] For example, for a triplet training sample , represents the first anchor sample, represents the first positive sample, is the first negative sample, which may include multiple sub-negative samples, expressed as ,in, is the number of sub-negative samples in the first negative sample. By using formula (1.1), the feature vector of the first anchor sample output by the student model after the n-1th training is calculated. and the feature vector of the first positive sample The cosine similarity between them gives the first similarity ; Using formula (1.1), calculate the feature vector of the first anchor sample output by the student model after the n-1th training respectively and the first negative sample The cosine similarity between the vector features of each sub-negative sample in the second similarity is obtained. .

[0147] As an example, based on the first similarity and the second similarity, a hard loss function is determined, wherein the hard loss function may be a 0-1 loss function, a cross entropy loss function, a logarithmic loss function, a square loss function, etc., which are not limited here. Formula (1.2) is a hard loss function provided in an embodiment of the present application. Build method:

[0148] (1.2)

[0149] in, is the first similarity, is the feature vector of the first anchor sample output by the student model after the n-1th training in the second similarity And the feature vector set of the first negative sample Middle Vector features of sub-negative samples The cosine similarity between .

[0150] In some embodiments, when the student model after the n-1th training is a cross encoder, the encoding result of the student model after the n-1th training includes a feature vector of a first concatenated sample of the student model after the n-1th training and a feature vector of a second concatenated sample of the student model after the n-1th training, Figure 3E The shown step 2031 can be implemented in the following manner: based on the linear layer of the student model after the n-1th training, the feature vector of the first spliced ​​sample of the student model after the n-1th training is subjected to dimensionality reduction processing to obtain the similarity score between the first anchor point sample and the first positive sample; based on the linear layer of the student model after the n-1th training, the feature vector of the second spliced ​​sample of the student model after the n-1th training is subjected to dimensionality reduction processing to obtain the similarity score between the first anchor point sample and the first negative sample; based on the similarity score between the first anchor point sample and the first positive sample and the similarity score between the first anchor point sample and the first negative sample, a hard loss function is constructed.

[0151] As an example, the feature vector of the first spliced ​​sample of the student model after the n-1th training and the feature vector of the second spliced ​​sample of the student model after the n-1th training are respectively input into the linear layer of the student model after the n-1th training, and the feature vector of the first spliced ​​sample and the feature vector of the second spliced ​​sample are subjected to dimensionality reduction to obtain the one-dimensional similarity score between the first anchor point sample and the first positive sample and the one-dimensional similarity score between the first anchor point sample and the first negative sample.

[0152] For example, for a triplet training sample , represents the first anchor sample, represents the first positive sample, is the first negative sample. The similarity score between the first anchor sample and the first positive sample is obtained by the linear layer of the student model after the n-1th training. , the similarity score between the first anchor sample and the first negative sample ,in, is the number of sub-negative samples in the first negative sample.

[0153] As an example, after determining the similarity score between the first anchor point sample and the first positive sample and the similarity score between the first anchor point sample and the first negative sample, a hard loss function is determined, wherein the hard loss function may be a 0-1 loss function, a cross entropy loss function, a logarithmic loss function, a square loss function, etc., which are not limited here. Formula (1.3) is a loss function provided in an embodiment of the present application. How to build:

[0154] (1.3)

[0155] in, is the similarity score between the first anchor sample and the first positive sample, is the similarity score set between the first anchor sample and the first negative sample. Similarity score.

[0156] The hard loss function in the embodiment of the present application can effectively measure the difference between the output result of the student model and the true label, thereby helping the student model to improve the accuracy of the output. At the same time, the computational complexity of the hard loss function is low, and it is more efficient in guiding the optimization of model parameters.

[0157] In step 2032, a soft loss function is constructed based on the encoding result of the nth teacher model and the encoding result of the student model after the n-1th training.

[0158] In some embodiments, Figure 3EThe shown step 2032 can be implemented in the following ways: based on the encoding result of the nth teacher model, determine the first output probability distribution of the nth teacher model; based on the encoding result of the student model after the n-1th training, determine the second output probability distribution of the student model after the n-1th training; based on the first output probability distribution and the second output probability distribution, determine the soft loss function.

[0159] As an example, based on the encoding result of the nth teacher model, the similarity score between the first anchor point sample and the first positive sample, and the similarity score between the first anchor point sample and the first negative sample are calculated, and then the first output probability distribution of the nth teacher model is determined, wherein the first output probability distribution includes a polynomial distribution, a Gaussian distribution, etc., or the encoding result of the nth teacher model is converted into the first output probability distribution of the nth teacher model by a soft exponential function (Softmax function), which is not limited here.

[0160] As an example, when the nth teacher model is a dual encoder, the encoding result of the nth teacher model is converted into the first output probability distribution of the nth teacher model through the Softmax function, and the formula (1.4) for determining the first output probability distribution of the nth teacher model is:

[0161] (1.4)

[0162] in, Represents the set of the first positive sample and the first negative sample , is the first anchor sample output by the nth teacher model and The similarity score between each sample in .

[0163] As an example, when the nth teacher model is a cross encoder, the encoding result of the nth teacher model is converted into the first output probability distribution of the nth teacher model through the Softmax function, and the formula (1.5) for determining the first output probability distribution of the nth teacher model is:

[0164] (1.5)

[0165] in, Represents the set of the first positive sample and the first negative sample , is the first anchor sample output by the nth teacher model and The similarity score between each sample in .

[0166] As an example, based on the encoding result of the student model after the n-1th training, the similarity score between the first anchor point sample of the student model after the n-1th training and the first positive sample of the student model after the n-1th training, as well as the similarity score between the first anchor point sample of the student model after the n-1th training and the first negative sample of the student model after the n-1th training are calculated, and then the second output probability distribution of the student model after the n-1th training is determined, wherein the second output probability distribution includes a polynomial distribution and a Gaussian distribution, etc., or the encoding result of the student model after the n-1th training is converted into the second output probability distribution of the student model after the n-1th training by the Softmax function, which is not limited here.

[0167] As an example, when the student model after the n-1th training is a dual encoder, the encoding result of the student model after the n-1th training is converted into the first output probability distribution of the student model after the n-1th training through the Softmax function, and the formula (1.6) for determining the second output probability distribution of the student model after the n-1th training is:

[0168] (1.6)

[0169] in, Represents the set of the first positive sample and the first negative sample , is the first anchor sample output by the student model after the n-1th training and The similarity score between each sample in .

[0170] As an example, when the student model after the n-1th training is a cross encoder, the encoding result of the student model after the n-1th training is converted into the first output probability distribution of the student model after the n-1th training through the Softmax function, and the formula (1.7) for determining the second output probability distribution of the student model after the n-1th training is:

[0171] (1.7)

[0172] in, Represents the set of the first positive sample and the first negative sample , is the first anchor sample output by the student model after the n-1th training and The similarity score between each sample in .

[0173] As an example, in determining the first output probability distribution and the second output probability distribution After that, the soft loss function is determined, where the soft loss function can be a cross entropy loss function, a logarithmic loss function, a KL divergence loss function, etc., which are not limited here. Formula (1.8) is the soft loss function provided in the embodiment of the present application The construction method for KL divergence loss function is:

[0174] (1.8)

[0175] In some embodiments, the first training sample includes a first anchor sample, a first positive sample and a first negative sample, and "determining the first output probability distribution of the nth teacher model based on the encoding result of the nth teacher model" can be achieved in the following way: based on the encoding result of the nth teacher model, determine the third similarity between the first anchor sample and the first positive sample, and determine the fourth similarity between the first anchor sample and the first negative sample; map the third similarity and the fourth similarity to obtain the fifth similarity; determine the ratio of the third similarity to the fifth similarity as the first probability value, and determine the ratio of the fourth similarity to the fifth similarity as the second probability value; determine the first probability value and the second probability value as the first output probability distribution of the nth teacher model.

[0176] As an example, for a triplet training sample , represents the first anchor sample, Represents the set of the first positive sample and the first negative sample , represents the first positive sample, is the first negative sample, and the third similarity between the first anchor sample and the first positive sample is calculated by formula (1.1) , the fourth similarity between the first anchor sample and the first negative sample is calculated by formula (1.1) , for the third similarity Similarity with the fourth Perform mapping processing, integrate the third similarity and the fourth similarity to obtain the fifth similarity , the ratio of the third similarity to the fifth similarity is determined as the first probability value , the ratio of the fourth similarity to the fifth similarity is determined as the second probability , integrate the distribution of the first probability value and the second probability value in the vector space, and then obtain the first output probability distribution of the nth teacher model.

[0177] The soft loss function focuses more on the relative relationship between the first anchor sample, the first positive sample, and the first negative sample. Therefore, the soft loss function can enable the student model to learn richer feature representations between training samples while reducing the student model's dependence on true labels.

[0178] In step 2033, the hard loss function and the soft loss function are integrated to obtain a total loss function.

[0179] As an example, the total loss function is constructed by weighting the hard loss function and the soft loss function according to formula (1.9):

[0180] (1.9)

[0181] in, and are the weight factors that control the importance of hard loss function and soft loss function respectively.

[0182] In the embodiment of the present application, the soft loss function tends to learn the relative relationship between samples, while the hard loss function directly focuses on the gap between the output results of the student model and the true label. By constructing the total loss function, the advantages of the soft loss function and the hard loss function can be combined.

[0183] In step 204, based on the total loss function, the parameters of the student model after the (n-1)th training are updated to obtain the student model after the (n)th training.

[0184] As an example, by minimizing the total loss function, the model parameters of the student model after the n-1th training are updated to obtain the student model after the n-1th training, wherein the method for updating the model parameters of the student model after the n-1th training can use gradient descent or other optimization algorithms, which are not limited here.

[0185] After multiple iterations of knowledge distillation of multiple teacher models, the trained student model obtained in the embodiment of the present application will be a model with relatively few parameters and extremely high efficiency. When the trained student model is deployed to the target device, the demand for computing resources and storage space of the target device is significantly reduced.

[0186] In step 104, the student model after the Nth training is determined as the trained student model.

[0187] As an example, in the field of text processing, the trained student model can be applied to the field of text retrieval, see Figure 8 , Figure 8 : is a flowchart of a text retrieval application provided in an embodiment of the present application, in which text retrieval is performed through a trained student model, wherein text retrieval can be divided into two stages: indexing and retrieval. Specifically: in the indexing stage, multiple documents in the corpus are encoded into document vectors of fixed dimensions through a trained student model (6-layer dual encoder) ; In the retrieval stage, for an input query text (query), the query text is encoded into a query vector representation through the trained student model (6-layer dual encoder) , calculate the query vector and the document vector set in the corpus one by one The cosine similarity score of each document vector in , the similarity score ranges from 0 to 1, and the cosine similarity score calculation formula (1.10) is as follows:

[0188] (1.10)

[0189] in, is the query vector, For the document vectors.

[0190] The higher the cosine similarity score, the closer the two query texts are to each other. The greater the similarity between the query text and the first document text, the smaller the score is, indicating that the query text and the first document text are closer. The smaller the similarity of the document texts.

[0191] When performing text retrieval tasks through the trained student model, by converting the query text and document text into high-dimensional text vectors, similarity search can be quickly performed, thereby more accurately measuring the semantic similarity between the query text and the document text, significantly improving the relevance of text vector search results, the quality of text vector search results, and the search efficiency of text vector search tasks.

[0192] In some embodiments, Figure 3A Before the shown step 101, the following steps can also be performed: obtain a second training sample, a third training sample, N initial teacher models and an initial student model; based on the second training sample, train the nth initial teacher model to obtain the nth teacher model; based on the third training sample, pre-train the initial student model to obtain a student model to be trained, wherein n is a successively increasing positive integer, 1<n≤N.

[0193] As an example, the second training sample includes a second anchor sample, a second positive sample and a second negative sample, and the third training sample includes a third anchor sample, a third positive sample and a third negative sample, wherein the second training sample and the third training sample can use manually annotated data sets, such as T2Ranking, mMARCO data sets, and the first training sample, the second training sample and the third training sample can be the same training sample or different training samples, which is not limited here, and the N initial teacher models and the initial student model can be models with model parameters as default values.

[0194] In some embodiments, when the nth initial teacher model is a dual encoder, see Figure 3F , Figure 3F is a sixth flow chart of the model training method provided in the embodiment of the present application, Figure 3A Before step 101 shown, "training the nth initial teacher model based on the second training sample" can also be achieved by following steps 301A to 304A, which are described in detail below.

[0195] In step 301A, a second training sample and an nth initial teacher model are obtained.

[0196] As an example, the second training sample includes a second anchor sample, a second positive sample, and a second negative sample.

[0197] For example, the second training sample can be a training data set , among which, for the training data, represents the second anchor sample, represents the second positive sample, represents the second negative sample.

[0198] In step 302A, data preprocessing is performed on the second anchor sample, the second positive sample, and the second negative sample, respectively, to obtain an input sequence of the second anchor sample, an input sequence of the second positive sample, and an input sequence of the second negative sample.

[0199] As an example, the data preprocessing performed on the second anchor sample, the second positive sample and the second negative sample is the same as the data preprocessing performed on the first anchor sample, the first positive sample and the first negative sample in step 2011A above, and will not be repeated here.

[0200] In step 303A, the input sequence of the second anchor sample, the input sequence of the second positive sample, and the input sequence of the second negative sample are respectively encoded through the nth initial teacher model to obtain the feature vector of the second anchor sample, the feature vector of the second positive sample, and the feature vector of the second negative sample.

[0201] As an example, refer to step 2012A here, which will not be repeated here.

[0202] In step 304A, the nth initial teacher model is trained based on the feature vector of the second anchor sample, the feature vector of the second positive sample, and the feature vector of the second negative sample to obtain the nth teacher model.

[0203] As an example, based on the feature vector of the second anchor sample, the feature vector of the second positive sample, and the feature vector of the second negative sample, a loss function of the nth initial teacher model pre-training is constructed, and the model parameters of the nth initial teacher model are updated by minimizing the loss function of the nth initial teacher model pre-training, and the nth initial teacher model after the parameter update is determined as the nth teacher model. Specifically, the loss function of the nth initial teacher model pre-training is The calculation formula (1.11) is:

[0204] (1.11)

[0205] in, is the first The similarity score between the second anchor sample and the second positive sample of the training data, In the second training sample The second anchor sample of the training data and the second negative sample The similarity score between the sub-negative samples, is the number of samples of the second training sample.

[0206] In some embodiments, when the nth initial teacher model is a cross encoder, see Figure 3G , Figure 3G is a seventh flow chart of the model training method provided in the embodiment of the present application, Figure 3A Before step 101 shown, "training the nth initial teacher model based on the second training sample" can also be achieved by following steps 301B to 305B, which are described in detail below.

[0207] In step 301B, a second training sample and an nth initial teacher model are obtained.

[0208] As an example, the second training sample includes a second anchor sample, a second positive sample, and a second negative sample.

[0209] In step 302B, the second anchor sample and the second positive sample are spliced ​​to obtain a third spliced ​​sample, and the second anchor sample and the second negative sample are spliced ​​to obtain a fourth spliced ​​sample.

[0210] As an example, refer to step 2011B here, which will not be repeated here.

[0211] In step 303B, data preprocessing is performed on the third spliced ​​sample and the fourth spliced ​​sample respectively to obtain an input sequence of the third spliced ​​sample and an input sequence of the fourth spliced ​​sample.

[0212] As an example, refer to step 2012B here, which will not be repeated here.

[0213] In step 304B, the input sequence of the third spliced ​​sample and the input sequence of the fourth spliced ​​sample are respectively encoded by the nth initial teacher model to obtain a feature vector of the third spliced ​​sample and a feature vector of the fourth spliced ​​sample.

[0214] As an example, refer to step 2013B here, which will not be repeated here.

[0215] In step 305B, based on the feature vector of the third concatenated sample and the feature vector of the fourth concatenated sample, the nth initial teacher model is trained to obtain the nth teacher model.

[0216] As an example, the feature vector of the third spliced ​​sample and the feature vector of the fourth spliced ​​sample are respectively input into the linear layer of the nth initial teacher model, and the feature vector of the third spliced ​​sample and the feature vector of the fourth spliced ​​sample are subjected to dimensionality reduction processing to obtain a one-dimensional similarity score between the second anchor point sample and the second positive sample, and a one-dimensional similarity score between the second anchor point sample and the second negative sample. Based on the similarity score between the second anchor point sample and the second positive sample, and the similarity score between the second anchor point sample and the second negative sample, a loss function for pre-training of the nth initial teacher model is constructed, and the model parameters of the nth initial teacher model are updated by minimizing the loss function of the pre-training of the nth initial teacher model, and the nth initial teacher model after the parameter update is determined as the nth teacher model. Specifically, the loss function for pre-training of the nth initial teacher model is The calculation formula (1.12) is:

[0217] (1.12)

[0218] in, is the first The similarity score between the second anchor sample and the second positive sample of the training data, In the second training sample The second anchor sample of the training data and the second negative sample The similarity score between the sub-negative samples, is the number of samples of the second training sample.

[0219] The embodiment of the present application minimizes the loss function of the pre-training of the nth initial teacher model and updates the model parameters of the initial teacher model, so that the teacher model can better measure the similarities between different samples, and thus more accurately implement the training of the student model to be trained in subsequent knowledge distillation tasks.

[0220] In some embodiments, when the initial student model is a dual encoder, the specific process of "pre-training the initial student model based on the third training sample to obtain the student model to be trained" can refer to the training method of the dual encoder initial teacher model in steps 301A to 304A, which will not be repeated here.

[0221] In some embodiments, when the initial student model is a cross-encoder, the specific process of "pre-training the initial student model based on the third training sample to obtain the student model to be trained" can be referred to the training method of the cross-encoder initial teacher model in steps 301B to 305B, which will not be repeated here.

[0222] By pre-training the initial student model, the student model to be trained already has a certain model accuracy before knowledge distillation. In the subsequent knowledge distillation process, it can better understand and absorb the knowledge from the teacher model, thereby improving the training efficiency of knowledge distillation.

[0223] The model training method provided in the embodiments of the present application can be applied to various fields, for example, it can be applied to the field of text processing (such as text vector retrieval and virtual human question and answer system, etc.), it can also be applied to the field of image processing (such as image recognition and image classification, etc.), and it can also be applied to the field of audio processing (such as audio recognition, etc.), without limitation here.

[0224] In the application scenario of text vector retrieval, obtain a student model to be trained (such as a text vector retrieval model), N teacher models (such as two teacher models, both of which are text vector retrieval models), and training samples for knowledge distillation (such as the anchor sample "How is the weather today", the positive sample "Today is a sunny day", the negative sample "Start dressing according to the weather today", "Finish today's things today", "Don't get angry every day, it's bad for your health"), input the training samples into the first teacher model and the student model to be trained respectively, train the student model to be trained by the first teacher model to obtain the student model after the first training, and then input the training samples into the second teacher model and the student model after the first training respectively, train the student model after the first training by the second teacher model to obtain the trained student model. After the student model is trained, it can be put into use. For example, when the query text (such as "What's the weather like today") is input into the trained student model, the trained student model will select "Today's weather is fine" from the document text collection (such as the document text collection {"Today's weather is fine", "Start dressing according to the weather today", "Finish today's work today", "Don't get angry every day, it's bad for your health"}) as the retrieval result of the trained student model to achieve accurate text retrieval function.

[0225] In the application scenario of image recognition, obtain a student model to be trained (such as an image recognition student model), N teacher models (such as two teacher models, both of which are image recognition models), and training samples for knowledge distillation (such as an anchor sample of a white cat picture "white cat.jpg", a positive sample of an orange cat picture "orange cat.jpg", and a negative sample of other animals pictures "crocodile.jpg", "tiger.jpg", and "lion.jpg"). Input the training samples into the first teacher model and the student model to be trained respectively, and train the student model to be trained by the first teacher model to obtain the student model after the first training. Then input the training samples into the second teacher model and the student model after the first training respectively, and train the student model after the first training by the second teacher model to obtain the trained student model. After the student model is trained, the trained student model can be put into application. For example, input a black cat picture to be recognized (such as "black cat.jpg") into the trained student model, and the trained student model will output the picture recognition result "cat" to achieve accurate picture recognition function.

[0226] In the application scenario of audio recognition, obtain a student model to be trained (such as an audio recognition model), N teacher models (such as two teacher models, both of which are audio recognition models), and training samples for knowledge distillation (such as the anchor sample is the audio file of the first segment of classical music A, "classical music A-part1.mp3", the positive sample is the audio file of the second segment of classical music A, "classical music A-part2.mp3", and the negative sample is the audio file of music "popular music A.mp3", "popular music B.mp3", and "popular music C.mp3"). Input the training samples into the first teacher model and the student model to be trained respectively, train the student model to be trained by the first teacher model, and obtain the student model after the first training. Then input the training samples into the second teacher model and the student model after the first training respectively, and train the student model after the first training by the second teacher model to obtain the trained student model. After the student model is trained, the trained student model can be put into application. For example, input the audio file to be recognized "classical music A-part1.mp3" into the trained student model, and the trained student model will output the audio recognition result "classical music A" to achieve accurate audio recognition function.

[0227] The following is an exemplary application of an embodiment of the present application in an application scenario of text vector retrieval.

[0228] The vector retrieval method based on deep learning has demonstrated its unique value in product applications and is widely used in search engines, intelligent question-answering systems, and chatbots. Specifically, in the search engine scenario, for search engines for web pages, documents, or other resources, the vector retrieval method based on deep learning can help users find content related to the query text more accurately. At the same time, by capturing the deep semantic relationship between the query text and the document to be queried, the relevance and quality of the search results can be improved. In the intelligent question-answering system or chatbot scenario, the vector retrieval method based on deep learning can be used to retrieve the answers most relevant to the user's questions from the knowledge base, thereby helping the robot to more accurately understand the user's needs and provide more targeted answers.

[0229] The vector retrieval technology solutions in related technologies are mainly implemented in the following ways:

[0230] 1) Pre-trained word vector representation model: Each word in the text is represented as a word vector, then the average word vector of the text is calculated as the vector representation of the entire text, and finally the cosine similarity between the vectors is calculated.

[0231] 2) Pre-trained language models: These models learn rich language knowledge through unsupervised training on large-scale corpora. These models can generate semantic vectors of texts by extracting the output of specific layers or using special aggregation methods. Then, they calculate the similarity between texts using metrics such as cosine similarity or Euclidean distance.

[0232] However, in the related art, the vector retrieval method based on the pre-trained word vector representation model, although the output word vector can capture the semantic and grammatical relationship between words, the pre-trained word vector representation model has limited context sensitivity to the text, especially when dealing with synonyms and polysemous words. This method mainly focuses on the vector representation of a single word, and the text vector obtained by averaging the word vector makes it difficult to fully capture the semantic information of the entire text; on the other hand, the method based on the pre-trained language model usually takes the masked language model as the training target, which makes the pre-trained language model better at representing a single word rather than the entire input text, thereby showing a lower representation ability in terms of overall text representation.

[0233] In order to solve the above problems, the present application embodiment proposes a model training method, see Figure 4 , Figure 4 It is a product flow chart of the model training method provided in the embodiment of the present application. The product flow of the model training method proposed in the embodiment of the present application includes: a contrastive learning training dual encoder stage, a contrastive learning training cross encoder stage, a knowledge distillation stage using a dual encoder teacher model, a knowledge distillation stage using a cross encoder teacher model, and an inference stage.

[0234] First, in the contrastive learning dual encoder training stage and the contrastive learning cross encoder training stage, two dual encoders with different parameter scales and different numbers of encoding layers (such as Transformer layers) are trained through the contrastive learning method, namely, a dual encoder with 6 Transformer layers (hereinafter referred to as the 6-layer dual encoder) and a dual encoder with 12 Transformer layers (hereinafter referred to as the 12-layer dual encoder). The dual encoders independently encode the query text and the document text to obtain the query vector and the document vector, and calculate the correlation score between the two text vectors; then, two cross encoders with different parameter scales are trained through the contrastive learning method, namely, a cross encoder with 12 Transformer layers (hereinafter referred to as the 12-layer cross encoder) and a cross encoder with 24 Transformer layers (hereinafter referred to as the 24-layer cross encoder). The cross encoder concatenates the query text and the document text as a complete sentence for encoding, which can effectively capture the fine-grained semantic interactions and subtle semantic associations between the two text sequences. Through the above process, we obtained 2 dual encoders with different numbers of layers and 2 cross encoders with different numbers of layers. In terms of model effect, the 24-layer cross encoder is better than the 12-layer cross encoder, the 12-layer cross encoder is better than the 12-layer dual encoder, and the 12-layer dual encoder is better than the 6-layer dual encoder.

[0235] Then, in the knowledge distillation stage using the dual encoder teacher model and the knowledge distillation stage using the cross encoder teacher model, when the gap between the student model and the teacher model is too large, the knowledge distillation is not very effective in improving the student model. In order to avoid the gap between the student model and the teacher model being too large, the embodiment of the present application gradually improves the effect of the student model through progressive knowledge distillation. Specifically, the 6-layer dual encoder with weaker performance and worse retrieval effect is used as the student model, and the 12-layer dual encoder, 12-layer cross encoder, and 24-layer cross encoder with stronger performance and better retrieval effect are used as teacher models in turn, and multi-stage progressive knowledge distillation is performed to gradually transfer the semantic matching knowledge from the teacher model with stronger performance to the student model with worse performance, thereby gradually and steadily improving the model capability and text retrieval effect of the 6-layer dual encoder.

[0236] Finally, in the inference stage, a 6-layer dual encoder model is used for text retrieval. The 6-layer dual encoder Transformer has fewer layers, fewer model parameters, shorter calculation time, and higher calculation efficiency. After multi-stage progressive knowledge distillation, the text retrieval effect of the 6-layer dual encoder can be significantly improved.

[0237] See also Figure 4 , Figure 4401 to 405, which are specifically described below:

[0238] In step 401, a dual encoder is trained by contrastive learning.

[0239] The dual encoder adopts a deep learning model (Transformer model) structure based on the self-attention mechanism. The Transformer model has strong modeling capabilities, good scalability, and can perform parallel computing well. The Transformer model includes multiple identical Transformer layers, each of which contains two sublayers: a multi-head self-attention mechanism and a feed-forward neural network. In addition, the Transformer layer is followed by a residual connection and layer normalization. The feed-forward neural network usually contains two fully connected layers and an activation function. Among them, the multi-head self-attention mechanism calculates the degree of association between each word in the input sequence and other words, thereby capturing the long-distance dependency in the sentence, and the multi-head self-attention mechanism allows the Transformer model to simultaneously pay attention to information at different positions in the text sequence, while the feed-forward neural network is used to extract local features of the input sequence.

[0240] In the stage of contrastive learning and dual encoder training, two dual encoders with different parameter amounts (different scales) are trained, namely a 6-layer dual encoder and a 12-layer dual encoder. The training method and training process of the 6-layer dual encoder and the 12-layer dual encoder are the same. Compared with the 6-layer dual encoder, the 12-layer dual encoder has more Transformer layers, a larger model parameter amount, and a better model effect.

[0241] The essence of dense vector retrieval is to learn the vector representation of text in a dense latent semantic space, model the semantic association and semantic interaction between query text (query) and document text (passage), and thus calculate and measure the relevance score between query text and document text.

[0242] A bi-encoder encodes each text sequence independently instead of taking them as a pair of inputs. Specifically, the bi-encoder adopts a dual-tower structure, using two independent encoders to learn the latent semantic representations of the query text and the document text, that is, the query vector and the document vector respectively; the relevance score is calculated based on these two vectors, and the relevance score between the query text and the document text is obtained by calculating the relevance function (vector dot product or cosine similarity) between the query vector and the document vector. In the bi-encoder structure, two independent encoders can be used, or the same encoder can be shared to encode the query text and the document text. In the embodiment of the present application, the encoding of the query text and the document text is achieved separately by sharing the same encoder.

[0243] The contrastive learning method is used to train the dual encoder. Contrastive learning refers to a learning method that identifies and distinguishes relevant positive sample texts from other irrelevant negative sample texts. Positive sample pairs are similar or related data pairs, while negative sample pairs are dissimilar or irrelevant data pairs. The basic idea of ​​contrastive learning is to bring similar and related samples (positive samples) closer to each other, and push dissimilar and irrelevant samples (negative samples) away from each other.

[0244] When contrastive learning is used to train the dual encoder, each piece of training data in each training dataset is represented by a triple containing three elements. A piece of training data includes a query text, a positive sample document text, and a negative sample document text list. That is, each query text (query) corresponds to a positive sample document text and multiple negative sample document texts. Among them, the training datasets can use T2Ranking and mMARCO datasets (on the order of hundreds of thousands). These datasets are manually annotated, and each query text has multiple high-quality negative sample document texts.

[0245] Here is an example of training data:

[0246] Query text: What are synonyms?

[0247] Positive document text (passage): Synonyms are words that have the same meaning (or almost the same) as another word.

[0248] Negative sample document text (passage) list:

[0249] Negative Sample 1: Remove Synonyms, Top Removal Synonyms for removal (another way of saying remove) are eliminate, withdraw, and take away.

[0250] Negative Sample 2: Synonyms, crossword answers and other related words for concrete We hope that the following list of synonyms for the word concrete will help you to finish your crossword today, we have arranged the synonyms in order of length to make them easier to find.

[0251] Negative sample 3: Synonyms of Alyssum: Alyssum (noun) belongs to Alyssum, other synonyms: broccoli, cabbage, cauliflower, mustard, kale, watercress, cruciferous plants.

[0252] See also Figure 5 , Figure 5 This is a schematic diagram of the dual encoder structure provided in the embodiment of the present application, which will be referred to below Figure 5 The training process of the dual encoder in the embodiment of the present application is specifically described.

[0253] First, a training data set is constructed and preprocessed to obtain a pre-trained training data set.

[0254] For example, given a training dataset containing many triples of training data , where, for the i-th training data, Represents the query text (query), Represents the positive sample document text (passage) corresponding to the query text, Represents the m negative sample document texts (passages) corresponding to the query text.

[0255] Preprocess the query text and document text to convert the text data into a form acceptable to the encoder model. Specifically, add a special symbol "cls" at the beginning of the query text to indicate the beginning of the sentence. Then, perform word segmentation and indexing on the query text to obtain the query text input sequence of the dual encoder. Similarly, a special symbol "cls" is added to the beginning of the document text to indicate the beginning of the sentence, and then the document text is segmented and indexed to obtain the document text input sequence of the dual encoder .

[0256] Then, the pre-trained training data set is input into the dual encoder to obtain the query text vector and document text vector output by the dual encoder.

[0257] See also Figure 5 ,Will As the input of the dual encoder model, after multiple Transformer layers of the dual encoder Transformer model are calculated, the output vector of the last Transformer layer corresponding to the starting character "cls" is used as the dense vector representation of the query text ; Similarly, As the input of the Transformer model, after multiple Transformer layers of the Transformer model are calculated, the last Transformer layer output vector corresponding to the starting character "cls" is used as the dense vector representation of the document text .

[0258] See also Figure 5 , based on the query text vector and document text vector , the cosine similarity between the two vectors is calculated using formula (2.1).

[0259] (2.1)

[0260] in, To scale the temperature coefficient, is the scoring function, through the query vector and document vector The cosine similarity between them is used to obtain the similarity score between the query text (query, q) and the document text (passage, p).

[0261] Finally, the dual encoder is trained by minimizing the contrastive learning loss function (2.2).

[0262] (2.2)

[0263] in, is the loss function of the Transformer model with dual encoders in the current stage, is the similarity score between the query text and the positive sample document text sample of the i-th training data in the training sample set, is the similarity score between the query text of the i-th training data in the training sample set and the j-th negative sample document text in the negative sample document text list, is the total number of samples in the training dataset.

[0264] During the contrastive learning training process, the model parameters of the dual encoder are updated by minimizing the loss function so that the similarity of positive samples is as high as possible and the similarity of negative samples is as low as possible.

[0265] At the current stage, two trained dual encoders with different parameter sizes can be obtained, namely a 6-layer dual encoder and a 12-layer dual encoder. Among them, the model effect of the 12-layer dual encoder is better than that of the 6-layer dual encoder.

[0266] In step 402, a cross encoder is trained by contrastive learning.

[0267] The cross encoder adopts a deep learning model (Transformer model) structure based on the self-attention mechanism. In the contrastive learning stage of training the cross encoder, two cross encoders with different parameter amounts (different scales) are trained, namely, a 12-layer cross encoder and a 24-layer cross encoder. The training methods and processes of the 12-layer cross encoder and the 24-layer cross encoder are the same. Compared with the 12-layer dual encoder, the 12-layer cross encoder can capture the semantic interaction between the query text and the document text in a more fine-grained manner. The model effect of the 12-layer cross encoder is better. Compared with the 12-layer cross encoder, the 24-layer cross encoder has more Transformer layers, a larger model parameter amount, and a better model effect.

[0268] The cross-encoder treats the query text and the document text as a text pair, concatenates the query text and the document text, and treats them as a complete sentence. Specifically, the cross-encoder concatenates the query text and the document text with a special separator "sep". In order to obtain the vector representation of the entire concatenated sentence, a special symbol "cls" is inserted at the beginning of the concatenated sentence. Then the concatenated sentence is used as the input of the cross-encoder model. After learning the vector representation of any word in the concatenated sentence, some calculation operations are performed to obtain the semantic matching representation (similarity score, which is a numerical value) between the query text and the document text. A common practice is to use the vector representation corresponding to the special symbol "cls" as the semantic matching vector between the two text sequences (a vector, which can be obtained by linear transformation. Similarity score / semantic matching representation) representation. The cross-encoder encodes the query text and the document text as a text pair, which can capture the fine-grained semantic interaction between the two text sequences, and can effectively capture the subtle semantic associations between the query text and the document text, thereby having better effects and higher performance in text retrieval tasks.

[0269] Similar to the training method of the dual encoder, the cross encoder also uses the contrastive learning method for model training. The training data set can also use T2Ranking and mMARCO. The format of the training data set is the same as that of the training data set when training the dual encoder.

[0270] See also Figure 6 , Figure 6 This is a schematic diagram of the cross encoder structure provided in the embodiment of the present application, and will be referred to below Figure 6 The training process of the cross encoder in the embodiment of the present application is specifically described.

[0271] First, a training data set is constructed and preprocessed to obtain a pre-trained training data set.

[0272] Specifically, given a training dataset containing many triplet training data , where, for the i-th training data, Represents the query text (query), Represents the positive sample document text (passage) corresponding to the query text, Represents the m negative sample document texts (passages) corresponding to the query text.

[0273] Preprocess the query text and document text to convert the text data into a form acceptable to the cross encoder model. Specifically, first use a special separator "sep" to concatenate the query text and document text. In order to obtain the vector representation of the entire concatenated sentence, a special symbol "cls" is inserted at the beginning of the concatenated sentence. Then, the concatenated sentence is segmented and indexed to obtain the input sequence of the cross encoder model. .

[0274] Then, the pre-trained training dataset is input into the cross encoder.

[0275] See also Figure 6 , the input sequence of the cross encoder model The cross encoder is input, and after calculations by multiple Transformer layers of the Transformer model of the cross encoder, the vector corresponding to the special symbol "cls" is used as the semantic matching vector between the two text sequences.

[0276] The semantic matching vector is fed into a linear layer whose input dimension is the vector dimension (e.g. 1024 or 2048) and whose output dimension is 1. The output of the linear layer is used as the semantic matching representation (similarity score) between the query text and the document text. .

[0277] Finally, the cross encoder is trained by minimizing the contrastive learning loss function (2.3).

[0278] (2.3)

[0279] in, is the loss function of the Transformer model of the cross encoder in the current stage, is the similarity score between the query text and the positive sample document text sample of the i-th training data in the training sample set, is the similarity score between the query text of the i-th training data in the training sample set and the j-th negative sample document text in the negative sample document text list, is the total number of samples in the training dataset.

[0280] During the contrastive learning training process, the model parameters of the cross encoder are updated by minimizing the loss function so that the similarity of positive samples is as high as possible and the similarity of negative samples is as low as possible. The cross encoder encodes the query text and document text as a text pair, which can capture the fine-grained semantic interactions and subtle semantic associations between the two text sequences, thereby achieving better results in text retrieval tasks.

[0281] At the current stage, two trained cross encoders with different parameter sizes can be obtained, namely a 12-layer cross encoder and a 24-layer cross encoder. Among them, the model effect of the 24-layer cross encoder is better than that of the 12-layer cross encoder.

[0282] In step 403, knowledge distillation is performed through a dual encoder teacher model.

[0283] By contrasting the learning and training dual encoder stage and the learning and training cross encoder stage, we have obtained 2 dual encoders with different numbers of layers (6-layer dual encoder and 12-layer dual encoder) and 2 cross encoders with different numbers of layers (12-layer cross encoder and 24-layer cross encoder). Compared with the 6-layer dual encoder, the 12-layer dual encoder has more transformer layers, larger model parameters and model capacity, and better model effect. The 12-layer cross encoder and the 12-layer dual encoder have the same number of transformer layers, but the 12-layer cross encoder can model the semantic interaction between the query text and the document text in a more fine-grained manner, with strong modeling ability and better model effect. Compared with the 12-layer cross encoder, the 24-layer cross encoder has more transformer layers, larger model parameters and model capacity, and better model effect. In general, in terms of model effect, the 24-layer cross encoder is better than the 12-layer cross encoder, the 12-layer cross encoder is better than the 12-layer dual encoder, and the 12-layer dual encoder is better than the 6-layer dual encoder.

[0284] See also Figure 7 , Figure 7 : is a schematic diagram of the progressive knowledge distillation process provided by the embodiment of the present application, wherein the 6-layer dual encoder with weaker performance and worse retrieval effect is used as the student model, and the 12-layer dual encoder, 12-layer cross encoder, and 24-layer cross encoder with stronger performance and better retrieval effect are used as the teacher model in sequence ( Figure 7 The teacher model 1, teacher model 2 and teacher model 3 in the 6-layer dual encoder are used to perform multi-stage progressive knowledge distillation, gradually transferring the semantic matching knowledge from the teacher model with stronger performance to the student model with weaker performance, thereby improving the model capability and text retrieval effect of the 6-layer dual encoder.

[0285] In the knowledge distillation stage through the dual encoder teacher model, the 12-layer dual encoder is used as the teacher model to perform knowledge distillation on the 6-layer dual encoder student model, where the total loss function of the knowledge distillation is the weighted sum of the hard loss and the soft loss.

[0286] Next, the calculation method of hard loss and soft loss is explained through one of the triplet training data in the training dataset.

[0287] For a triplet of training data , Represents the query text (query), Represents a collection of document text , Represents the corresponding positive sample document text (passage), Represents m negative sample document texts (passages), that is, in a triple training data, for the query text , there are m+1 document texts , including 1 positive sample document text and m negative sample document texts.

[0288] The first part of the hard loss is the loss function of contrastive learning, which calculates the loss between the probability distribution output by the 6-layer dual encoder student model and the hard label. The calculation formula (2.4) of the hard loss is:

[0289] (2.4)

[0290] in, For hard loss, Similarity scores between query text and positive document text samples output by the 6-layer dual encoder student model.

[0291] The second part of the soft loss is to calculate the probability distribution of the output of the 6-layer dual encoder student model (2.5) and the probability distribution of the 12-layer dual encoder teacher model output (2.6) is obtained by the KL divergence loss function (2.7).

[0292] Probability distribution of the output of the 6-layer dual encoder student model The calculation formula (2.5) is:

[0293] (2.5)

[0294] in, It is the similarity score between the query text output by the 6-layer dual encoder student model and each document text sample in each document text collection.

[0295] Probability distribution of the output of the 12-layer dual encoder teacher model The calculation formula (2.6) is:

[0296] (2.6)

[0297] in, It is the similarity score between the query text output by the 12-layer dual encoder teacher model and each document text sample in each document text collection.

[0298] The calculation formula (2.7) of soft loss (KL divergence loss function) is:

[0299] (2.7)

[0300] in, is the probability distribution of the output of the 6-layer dual encoder student model, is the probability distribution of the output of the 12-layer dual encoder teacher model.

[0301] The total loss function for knowledge distillation using the dual encoder teacher model is the weighted sum of hard loss and soft loss. The formula (2.8) of the total loss function is:

[0302] (2.8)

[0303] in, and are weight factors controlling the importance of hard loss and soft loss respectively.

[0304] When using the dual encoder teacher model for knowledge distillation, the model parameters of the 12-layer dual encoder teacher model are frozen, and the model parameters of the 6-layer dual encoder student model are updated by minimizing the total loss function, thereby transferring the semantic matching knowledge from the teacher model with stronger performance to the student model with weaker performance, thereby improving the text retrieval effect of the 6-layer dual encoder.

[0305] In step 404, knowledge distillation is performed through the cross-encoder teacher model.

[0306] Although the 12-layer cross encoder and the 12-layer dual encoder have the same number of Transformer layers, the cross encoder can capture the semantic interaction between the query text and the document text in more detail, thus having stronger modeling capabilities and achieving better model results, see Figure 7After using the 12-layer dual encoder teacher model for knowledge distillation, in the knowledge distillation stage of the cross encoder teacher model, the 12-layer cross encoder is first used for secondary knowledge distillation, and then the 24-layer cross encoder is used for knowledge distillation. Whether using the 12-layer or 24-layer cross encoder as the teacher model, the knowledge distillation method and process of the 6-layer dual encoder student model are the same, which is described in detail below.

[0307] In the knowledge distillation stage through the cross-encoder teacher model, the total loss function of knowledge distillation is the weighted sum of hard loss and soft loss.

[0308] Next, the calculation method of hard loss and soft loss is explained through one of the triplet training data in the training dataset.

[0309] For a triplet of training data , Represents the query text (query), Represents a collection of document text , Represents the corresponding positive sample document text (passage), Represents m negative sample document texts (passages), that is, in a triple training data, for the query text , there are m+1 document texts , including 1 positive sample document text and m negative sample document texts.

[0310] The first part, hard loss, is the loss function of contrastive learning, which calculates the loss between the probability distribution output by the 6-layer dual encoder student model and the hard label. The calculation formula of hard loss can be found in formula (2.4) above.

[0311] The second part of the soft loss is to calculate the probability distribution of the output of the 6-layer dual encoder student model Probability distribution of the output of the cross-encoder teacher model The KL divergence loss function between .

[0312] Among them, the probability distribution of the output of the 6-layer dual encoder student model is The calculation formula of is shown in formula (2.5) above.

[0313] Probability distribution of the output of the cross-encoder teacher model The calculation formula (2.9) is:

[0314] (2.9)

[0315] in, is the similarity score between the query text output by the cross-encoder teacher model and each document text sample in each document text collection.

[0316] The calculation formula (2.10) of soft loss (KL divergence loss function) is:

[0317] (2.10)

[0318] in, is the probability distribution of the output of the 6-layer dual encoder student model, is the probability distribution of the output of the cross-encoder teacher model.

[0319] The total loss function for knowledge distillation using the cross-encoder teacher model is the weighted sum of hard loss and soft loss. The formula (2.11) of the total loss function is:

[0320] (2.11)

[0321] and are weight factors controlling the importance of hard loss and soft loss respectively.

[0322] When using the cross-encoder teacher model for knowledge distillation, the model parameters of the cross-encoder teacher model are frozen, and the model parameters of the 6-layer dual-encoder student model are updated by minimizing the loss function, thereby transferring the semantic matching knowledge from the teacher model with stronger performance to the student model with weaker performance, thereby improving the text retrieval effect of the 6-layer dual encoder.

[0323] In step 405, in the inference phase, relevant text is retrieved.

[0324] See also Figure 8 , Figure 8 It is a flow chart of the text retrieval application provided in the embodiment of the present application. In the inference stage, a 6-layer dual encoder model is used for text retrieval. The 6-layer dual encoder transformer has fewer layers, fewer model parameters, shorter calculation time and higher calculation efficiency.

[0325] The inference phase of the 6-layer dual encoder model is divided into two steps: indexing and retrieval.

[0326] During the indexing phase, see Figure 8 , the N documents in the corpus are encoded into fixed-dimensional document vector representations through a 6-layer dual encoder model .

[0327] Retrieval phase, see Figure 8 For an input query, the query text is encoded into a query vector representation through a 6-layer dual encoder model , then the query vector is represented as The document vector representation in the corpus , calculate the cosine similarity score one by one, the similarity score range is between 0 and 1, and the cosine similarity score calculation formula (2.12) is as follows:

[0328] (2.12)

[0329] in, is the query vector representation, is the document vector of the i-th document.

[0330] The higher the cosine similarity score, the greater the similarity between the two texts; conversely, the smaller the score, the smaller the similarity between the two texts. The document with the highest similarity score is selected and returned.

[0331] In summary, progressive knowledge distillation allows the student model to gradually absorb knowledge from different teacher models during the learning process. When there is a new teacher model for knowledge distillation, the student model can perform incremental learning based on the existing knowledge, which helps to improve the performance of the student model in text retrieval tasks. Progressive knowledge distillation can obtain an efficient student model with a relatively small parameter size, which helps to reduce the consumption of computing resources and storage space, which is especially important for text retrieval tasks on devices with limited resources.

[0332] The following is a description of an exemplary structure of a model training device 555 provided in an embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the model training device 555 of the memory 550 may include:

[0333] The acquisition module 5551 is used to obtain the student model to be trained and N teacher models, wherein the model performance of the teacher models is different, and N is a positive integer greater than 1.

[0334] The distillation module 5552 is used to train the student model to be trained through the first teacher model to obtain the student model after the first training, wherein the first teacher model is the teacher model with the lowest model performance among the N teacher models.

[0335] The distillation module 5552 is also used to train the student model after the n-1th training through the nth teacher model to obtain the student model after the n-1th training, wherein n is a positive integer that increases successively, 1<n≤N, and the student model after the n-1th training is obtained by training with the n-1th teacher model. The nth teacher model and the n-1th teacher model are any teacher models among the N teacher models, and the model performance of the nth teacher model is higher than the model performance of the n-1th teacher model.

[0336] The determination module 5553 is used to determine the student model after the Nth training as the trained student model.

[0337] In the above technical scheme, the distillation module 5552 is also used to encode the first training sample through the nth teacher model to obtain the encoding result of the nth teacher model; encode the first training sample through the student model after the n-1th training to obtain the encoding result of the student model after the n-1th training; construct a total loss function based on the encoding result of the student model after the n-1th training and the encoding result of the nth teacher model; based on the total loss function, update the parameters of the student model after the n-1th training to obtain the student model after the n-1th training.

[0338] In the above technical solution, the first training sample includes a first anchor sample, a first positive sample and a first negative sample, and the n-1th teacher model and the nth teacher model are both dual encoders; the distillation module 5552 is also used to encode the first training sample through the nth teacher model to obtain the encoding result of the nth teacher model, including: performing data preprocessing on the first anchor sample, the first positive sample and the first negative sample respectively to obtain the input sequence of the first anchor sample, the input sequence of the first positive sample and the input sequence of the first negative sample; encoding the input sequence of the first anchor sample, the input sequence of the first positive sample and the input sequence of the first negative sample respectively through the nth teacher model to obtain the feature vector of the first anchor sample, the feature vector of the first positive sample and the feature vector of the first negative sample; using the feature vector of the first anchor sample, the feature vector of the first positive sample and the feature vector of the first negative sample as the encoding result of the nth teacher model.

[0339] In the above technical solution, the encoding module 5554 is used to encode the input sequence of the first anchor sample through the first encoding layer of the M cascaded encoding layers to obtain the output features of the first encoding layer; encode the output features of the m-1th encoding layer through the mth encoding layer to obtain the output features of the mth encoding layer, and determine the feature vector of the first anchor sample based on the output features of the Mth encoding layer, wherein m is a successively increasing positive integer, 1<m≤M.

[0340] In the above technical solution, the first training sample includes a first anchor sample, a first positive sample and a first negative sample, the n-1th teacher model is a dual encoder, and the nth teacher model is a cross encoder; the encoding module 5554 is also used to splice the first anchor sample and the first positive sample to obtain a first spliced ​​sample, and to splice the first anchor sample and the first negative sample to obtain a second spliced ​​sample; data preprocessing is performed on the first spliced ​​sample and the second spliced ​​sample respectively to obtain an input sequence of the first spliced ​​sample and an input sequence of the second spliced ​​sample; the input sequence of the first spliced ​​sample and the input sequence of the second spliced ​​sample are encoded by the nth teacher model respectively to obtain a feature vector of the first spliced ​​sample and a feature vector of the second spliced ​​sample; the feature vector of the first spliced ​​sample and the feature vector of the second spliced ​​sample are used as the encoding result of the nth teacher model, wherein the first training sample includes a first anchor sample, a first positive sample and a first negative sample.

[0341] In the above technical solution, the distillation module 5552 is also used to construct a hard loss function based on the encoding result of the student model after the n-1th training; to construct a soft loss function based on the encoding result of the nth teacher model and the encoding result of the student model after the n-1th training; and to integrate the hard loss function and the soft loss function to obtain a total loss function.

[0342] In the above technical scheme, the distillation module 5552 is also used to perform similarity processing on the feature vector of the first anchor sample of the student model after the n-1th training and the feature vector of the first positive sample of the student model after the n-1th training to obtain a first similarity; perform similarity processing on the feature vector of the first anchor sample of the student model after the n-1th training and the feature vector of the first negative sample of the student model after the n-1th training to obtain a second similarity; determine a hard loss function based on the first similarity and the second similarity, wherein the encoding result of the student model after the n-1th training includes the feature vector of the first anchor sample of the student model after the n-1th training, the feature vector of the first positive sample of the student model after the n-1th training, and the feature vector of the first negative sample of the student model after the n-1th training.

[0343] In the above technical solution, the distillation module 5552 is also used to determine the first output probability distribution of the nth teacher model based on the encoding result of the nth teacher model; determine the second output probability distribution of the student model after the n-1th training based on the encoding result of the student model after the n-1th training; and determine the soft loss function based on the first output probability distribution and the second output probability distribution.

[0344] In the above technical solution, the distillation module 5552 is also used to determine the third similarity between the first anchor sample and the first positive sample based on the encoding result of the nth teacher model, and determine the fourth similarity between the first anchor sample and the first negative sample; map the third similarity and the fourth similarity to obtain a fifth similarity; determine the ratio of the third similarity to the fifth similarity as a first probability value, and determine the ratio of the fourth similarity to the fifth similarity as a second probability value; determine the first probability value and the second probability value as the first output probability distribution of the nth teacher model.

[0345] In the above technical solution, the pre-training module 5555 is used to obtain a second training sample and an nth initial teacher model, wherein the second training sample includes a second anchor sample, a second positive sample and a second negative sample; data preprocessing is performed on the second anchor sample, the second positive sample and the second negative sample respectively to obtain an input sequence of the second anchor sample, an input sequence of the second positive sample and an input sequence of the second negative sample; the input sequence of the second anchor sample, the input sequence of the second positive sample and the input sequence of the second negative sample are encoded by the nth initial teacher model respectively to obtain a feature vector of the second anchor sample, a feature vector of the second positive sample and a feature vector of the second negative sample; based on the feature vector of the second anchor sample, the feature vector of the second positive sample and the feature vector of the second negative sample, the nth initial teacher model is trained to obtain the nth teacher model.

[0346] In the above technical solution, the pre-training module 5555 is also used to obtain a second training sample and an nth initial teacher model, wherein the second training sample includes a second anchor sample, a second positive sample and a second negative sample; splicing the second anchor sample and the second positive sample to obtain a third spliced ​​sample, and splicing the second anchor sample and the second negative sample to obtain a fourth spliced ​​sample; performing data preprocessing on the third spliced ​​sample and the fourth spliced ​​sample respectively to obtain an input sequence of the third spliced ​​sample and an input sequence of the fourth spliced ​​sample; encoding the input sequence of the third spliced ​​sample and the input sequence of the fourth spliced ​​sample respectively through the nth initial teacher model to obtain a feature vector of the third spliced ​​sample and a feature vector of the fourth spliced ​​sample; based on the feature vector of the third spliced ​​sample and the feature vector of the fourth spliced ​​sample, training the nth initial teacher model to obtain the nth teacher model.

[0347] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device performs the model training method described above in the embodiment of the present application.

[0348] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the model training method provided by the present application embodiment, for example, Figures 3A to 3G The model training method shown.

[0349] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0350] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0351] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0352] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0353] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Obtain a student model to be trained and N teacher models, wherein the model performances of the teacher models are different, and N is a positive integer greater than 1; Training the student model to be trained by a first teacher model to obtain a student model after the first training, wherein the first teacher model is a teacher model with the lowest model performance among the N teacher models; The student model after the n-1th training is trained by the nth teacher model to obtain the student model after the n-1th training, wherein n is a positive integer that increases successively, 1<n≤N, the student model after the n-1th training is obtained by training the n-1th teacher model, the nth teacher model and the n-1th teacher model are any teacher models among the N teacher models, and the model performance of the nth teacher model is higher than the model performance of the n-1th teacher model; The student model after the Nth training is determined as the trained student model.

2. The method according to claim 1, characterized in that The step of training the student model after the n-1th training by the nth teacher model to obtain the student model after the nth training includes: Performing encoding processing on the first training sample by using the nth teacher model to obtain an encoding result of the nth teacher model; Performing encoding processing on the first training sample by using the student model after the n-1th training to obtain an encoding result of the student model after the n-1th training; Constructing a total loss function based on the encoding result of the student model after the n-1th training and the encoding result of the nth teacher model; Based on the total loss function, the parameters of the student model after the (n-1)th training are updated to obtain the student model after the (n)th training.

3. The method according to claim 2, characterized in that The first training sample includes a first anchor sample, a first positive sample and a first negative sample, and both the n-1th teacher model and the nth teacher model are dual encoders; The encoding process of the first training sample by the nth teacher model to obtain the encoding result of the nth teacher model includes: Performing data preprocessing on the first anchor sample, the first positive sample, and the first negative sample respectively to obtain an input sequence of the first anchor sample, an input sequence of the first positive sample, and an input sequence of the first negative sample; By using the nth teacher model, the input sequence of the first anchor sample, the input sequence of the first positive sample, and the input sequence of the first negative sample are respectively encoded to obtain a feature vector of the first anchor sample, a feature vector of the first positive sample, and a feature vector of the first negative sample; The feature vector of the first anchor sample, the feature vector of the first positive sample, and the feature vector of the first negative sample are used as encoding results of the nth teacher model.

4. The method according to claim 3, characterized in that The encoding process of the input sequence of the first anchor sample is implemented by M cascaded encoding layers included in the nth teacher model, where M is a positive integer greater than 1; Performing encoding processing on the input sequence of the first anchor sample through a first encoding layer of the M cascaded encoding layers to obtain output features of the first encoding layer; The output features of the m-1th coding layer are encoded through the mth coding layer to obtain the output features of the mth coding layer, and the feature vector of the first anchor sample is determined based on the output features of the Mth coding layer, wherein m is a successively increasing positive integer, 1<m≤M.

5. The method according to claim 2, characterized in that: The first training sample includes a first anchor sample, a first positive sample and a first negative sample, the n-1th teacher model is a dual encoder, and the nth teacher model is a cross encoder; The encoding process of the first training sample by the nth teacher model to obtain the encoding result of the nth teacher model includes: splicing the first anchor sample and the first positive sample to obtain a first spliced ​​sample, and splicing the first anchor sample and the first negative sample to obtain a second spliced ​​sample; Performing data preprocessing on the first spliced ​​sample and the second spliced ​​sample respectively to obtain an input sequence of the first spliced ​​sample and an input sequence of the second spliced ​​sample; Encoding the input sequence of the first spliced ​​sample and the input sequence of the second spliced ​​sample respectively through the nth teacher model to obtain a feature vector of the first spliced ​​sample and a feature vector of the second spliced ​​sample; The feature vector of the first spliced ​​sample and the feature vector of the second spliced ​​sample are used as encoding results of the nth teacher model.

6. The method according to claim 2, characterized in that The total loss function is constructed based on the encoding result of the student model after the n-1th training and the encoding result of the nth teacher model, including: Constructing a hard loss function based on the encoding result of the student model after the n-1th training; Constructing a soft loss function based on the encoding result of the nth teacher model and the encoding result of the student model after the n-1th training; The hard loss function and the soft loss function are integrated to obtain a total loss function.

7. The method according to claim 6, characterized in that The encoding result of the student model after the n-1th training includes a feature vector of a first anchor sample of the student model after the n-1th training, a feature vector of a first positive sample of the student model after the n-1th training, and a feature vector of a first negative sample of the student model after the n-1th training; The hard loss function is constructed based on the encoding result of the student model after the n-1th training, including: Performing similarity processing on a feature vector of a first anchor sample of the student model after the n-1th training and a feature vector of a first positive sample of the student model after the n-1th training to obtain a first similarity; Performing similarity processing on a feature vector of a first anchor sample of the student model after the n-1th training and a feature vector of a first negative sample of the student model after the n-1th training to obtain a second similarity; Based on the first similarity and the second similarity, a hard loss function is determined.

8. The method according to claim 6, characterized in that The constructing of a soft loss function based on the encoding result of the nth teacher model and the encoding result of the student model after the n-1th training includes: Determining a first output probability distribution of the nth teacher model based on the encoding result of the nth teacher model; Determine a second output probability distribution of the student model after the n-1th training based on the encoding result of the student model after the n-1th training; Based on the first output probability distribution and the second output probability distribution, a soft loss function is determined.

9. The method according to claim 8, characterized in that The first training sample includes a first anchor sample, a first positive sample and a first negative sample; The step of determining a first output probability distribution of the nth teacher model based on the encoding result of the nth teacher model comprises: Based on the encoding result of the nth teacher model, determining a third similarity between the first anchor sample and the first positive sample, and determining a fourth similarity between the first anchor sample and the first negative sample; Performing mapping processing on the third similarity and the fourth similarity to obtain a fifth similarity; determining a ratio of the third similarity to the fifth similarity as a first probability value, and determining a ratio of the fourth similarity to the fifth similarity as a second probability value; The first probability value and the second probability value are determined as a first output probability distribution of the nth teacher model.

10. The method according to claim 1, characterized in that Before obtaining the student model to be trained and N teacher models, the method further includes: Obtain a second training sample and an nth initial teacher model, wherein the second training sample includes a second anchor sample, a second positive sample, and a second negative sample; Performing data preprocessing on the second anchor sample, the second positive sample, and the second negative sample respectively to obtain an input sequence of the second anchor sample, an input sequence of the second positive sample, and an input sequence of the second negative sample; encoding the input sequence of the second anchor sample, the input sequence of the second positive sample, and the input sequence of the second negative sample respectively through the nth initial teacher model to obtain a feature vector of the second anchor sample, a feature vector of the second positive sample, and a feature vector of the second negative sample; The nth initial teacher model is trained based on the feature vector of the second anchor sample, the feature vector of the second positive sample, and the feature vector of the second negative sample to obtain the nth teacher model.

11. The method according to claim 1, characterized in that: Before obtaining the student model to be trained and N teacher models, the method further includes: Obtain a second training sample and an nth initial teacher model, wherein the second training sample includes a second anchor sample, a second positive sample, and a second negative sample; splicing the second anchor sample and the second positive sample to obtain a third spliced ​​sample, and splicing the second anchor sample and the second negative sample to obtain a fourth spliced ​​sample; Performing data preprocessing on the third spliced ​​sample and the fourth spliced ​​sample respectively to obtain an input sequence of the third spliced ​​sample and an input sequence of the fourth spliced ​​sample; By using the nth initial teacher model, the input sequence of the third spliced ​​sample and the input sequence of the fourth spliced ​​sample are respectively encoded to obtain a feature vector of the third spliced ​​sample and a feature vector of the fourth spliced ​​sample; Based on the feature vector of the third spliced ​​sample and the feature vector of the fourth spliced ​​sample, the nth initial teacher model is trained to obtain the nth teacher model.

12. A model training device, characterized in that: The device comprises: An acquisition module, used to acquire a student model to be trained and N teacher models, wherein the model performances of the teacher models are different, and N is a positive integer greater than 1; A distillation module, used for training the student model to be trained by using a first teacher model to obtain a student model after the first training, wherein the first teacher model is a teacher model with the lowest model performance among the N teacher models; The distillation module is further used to train the student model after the n-1th training through the nth teacher model to obtain the student model after the n-1th training, wherein n is a positive integer that increases successively, 1<n≤N, the student model after the n-1th training is obtained by training the n-1th teacher model, the nth teacher model and the n-1th teacher model are any teacher model among the N teacher models, and the model performance of the nth teacher model is higher than the model performance of the n-1th teacher model; The determination module is used to determine the student model after the Nth training as the trained student model.

13. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; The processor is used to implement the model training method described in any one of claims 1 to 11 when executing the computer executable instructions or computer programs stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the model training method described in any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the model training method described in any one of claims 1 to 11 is implemented.