Student model optimization method and equipment based on distillation learning, and readable storage medium

Abstract

The invention discloses a student model optimization method and device based on distillation learning and a readable storage medium. The method comprises the following steps: determining a first vector relation value of each first coding module and / or a second vector relation value between any two first coding modules according to at least one of a query vector, a key vector and a value vector of each attention layer in each first coding module of a student model; determining a third vector relation value of each second coding module and / or a fourth vector relation value between any two second coding modules according to at least one of a query vector, a key vector and a value vector of each attention layer in each second coding module of the teacher model; and determining a target loss value of the student model according to a first relation difference between the first vector relation value and the third vector relation value and / or a second relation difference between the second vector relation value and the fourth vector relation value, and optimizing the student model according to the target loss value. Therefore, the performance of the student model can be improved.

Description

Technical Field

[0001] The present invention relates to the field of distillation learning processing technology, and in particular to a student model optimization method based on distillation learning, an electronic device and a computer-readable storage medium. Background Art

[0002] Model distillation is a technique that transfers the knowledge of a large and complex teacher model to a small student model, allowing the student model to inherit the performance advantages of the teacher model while maintaining lower computational cost and faster inference speed.

[0003] During the model distillation process, in order to make the output of the student model as close as possible to the value you really want to predict, you can compare the difference between the value output by the teacher model and the value output by the student model, as well as the difference between the value output by the student model and the true value. The two are combined to obtain the loss value of the student model, and the student model is optimized with the loss value.

[0004] It can be seen that the loss value is an important parameter used to measure the quality of the student model, and the accuracy of the loss value determines the optimization ability of the model. The traditional loss value calculation method is mainly based on the predicted value of the final output of the model, but ignores the influence of the model's internal structure, resulting in the student model being unable to fully learn the deep features of the teacher model, affecting the optimization performance of the student model. Summary of the invention

[0005] The main technical problem solved by this application is to provide a student model optimization method based on distillation learning, an electronic device and a computer-readable storage medium to improve the optimization effect of the student model.

[0006] To solve the above technical problems, a technical solution adopted in the present application is: a student model optimization method based on distillation learning is provided, and the student model optimization method based on distillation learning is applied to a student model optimization device, and the student model optimization device includes a student model and a teacher model, and the student model includes at least one first encoding module, and each of the first encoding modules includes at least one first attention layer, and the teacher model includes at least one second encoding module, and each of the second encoding modules includes at least one second attention layer. The method includes: inputting the data to be processed into the student model to obtain the query vector, key vector and value vector of each first attention layer in each first encoding module; determining the first vector relationship value of each first encoding module and / or the second vector relationship value between any two first encoding modules according to at least one of the query vector, key vector and value vector of each first attention layer; inputting the data to be processed into the student model to obtain the query vector, key vector and value vector of each first attention layer in each first encoding module; determining the first vector relationship value of each first encoding module and / or the second vector relationship value between any two first encoding modules according to at least one of the query vector, key vector and value vector of each first attention layer; and The method comprises the steps of: inputting the query vector, the key vector and the value vector of each second attention layer in each second encoding module into the teacher model to obtain the query vector, the key vector and the value vector of each second attention layer; determining the third vector relationship value of each second encoding module and / or the fourth vector relationship value between any two second encoding modules according to at least one of the query vector, the key vector and the value vector of each second attention layer; determining the first relationship difference between the first vector relationship value of each first encoding module in the student model and the third vector relationship value of the corresponding second encoding module in the teacher model according to the attribute information of the encoding module, and / or determining the second relationship difference between the second vector relationship value in the student model and the fourth vector relationship value in the teacher model according to the attribute information of the encoding module; determining the target loss value of the student model according to the first relationship difference and / or the second relationship difference; optimizing the student model according to the target loss value to obtain the optimized student model.

[0007] In order to solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, including a memory and a processor, the memory storing program instructions, and the processor calling the program instructions from the memory to execute the above-mentioned student model optimization method based on distillation learning.

[0008] In order to solve the above technical problems, another technical solution adopted in the present application is: providing a computer-readable storage medium including program data stored therein, and the program data is used to implement the above-mentioned student model optimization method based on distillation learning when executed by a processor.

[0009] The above scheme determines the first vector relationship value of each first coding module and / or the second vector relationship value between any two first coding modules according to at least one of the query vector, key vector and value vector of each attention layer in each first coding module of the student model; determines the third vector relationship value of each second coding module and / or the fourth vector relationship value between any two second coding modules according to at least one of the query vector, key vector and value vector of each attention layer in each second coding module of the teacher model. In this way, not only the vector relationship of the attention layer inside the model can be fully learned, but also the vector relationship between multiple coding modules in the same model can be learned; then the first relationship difference is determined according to the first vector relationship value and the third vector relationship value and / or the second relationship difference is determined according to the second vector relationship value and the fourth vector relationship value; the target loss value of the student model is determined according to the first relationship difference and / or the second relationship difference, and the student model is optimized according to the target loss value, thereby optimizing the student model through the difference between the student model and the teacher model in the attention layer, and improving the performance of the student model. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which: Figure 1 is a flow chart of an exemplary embodiment of the student model optimization method shown in the present application; Figure 2 yes Figure 1 A flow chart of an exemplary embodiment of step S120 in the student model optimization method is shown; Figure 3 is a schematic diagram of a framework of an exemplary embodiment of the student model optimization method shown in the present application; Figure 4 is a structural schematic diagram of an exemplary embodiment of a student model optimization device shown in the present application; Figure 5 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application; Figure 6 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0011] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be appreciated that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the art without making creative work are within the scope of protection of the present application.

[0012] First of all, it should be noted that distillation learning usually transfers the knowledge of a large and complex teacher model to a smaller and more efficient student model. The teacher model is generally a trained model that provides high-quality prediction results to guide the learning of the student model. The student model is generally a model with a simpler structure and smaller parameters. It can improve its own performance by imitating the teacher model and is more efficient than the teacher model.

[0013] In order to better enable the student model to learn the deeper structure of the teacher model, this application provides a student model optimization method based on distillation learning, hereinafter referred to as the student model optimization method, which deeply considers the impact of the attention layer on distillation learning, and optimizes the student model by using the difference between the attention layer of the teacher model and the attention layer of the student model, thereby improving the optimization performance of the student model. For details, please refer to Figure 1 , Figure 1 It is a flowchart of an exemplary embodiment of the student model optimization method shown in the present application.

[0014] The execution subject of the student model optimization method can be a terminal device or a server or other processing device, wherein the terminal device can be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The execution subject of the student model optimization method can also be a student model optimization device. In some possible implementations, the student model optimization method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0015] Specifically, the student model optimization method of this embodiment includes the following steps: S110: Input the data to be processed into the student model to obtain the query vector, key vector and value vector of each first attention layer in each first encoding module.

[0016] The data to be processed may be a data to be processed selected from a data set. For example, it may be any data in the data set, or it may be the data with the best quality in the data set, which is not limited in the embodiment of the present application. It should be noted that a data to be processed may include but is not limited to image data, text data, etc.

[0017] The student model refers to a model that needs to be optimized. In the embodiment of the present application, the process of model optimization is mainly based on distillation learning of the teacher model. The teacher model refers to a model pre-trained by a data set. Based on the teacher model, a student model with a smaller number of parameters is initialized, and then the student model is continuously optimized by the teacher model.

[0018] The first encoding module is an encoding module in the student model, and the student model includes at least one first encoding module. Exemplarily, the first encoding module can be an image encoding module for extracting image features of image data; it can also be a text encoding module for extracting text features of text data; it can also be a fusion encoding module for fusing image features and / or text features to obtain fusion features.

[0019] The first encoding module includes at least one first attention layer. When processing each element in the data to be processed, the attention layer can dynamically focus on other related elements to obtain contextual information of the data to be processed. The architecture of the first encoding module can be based on the Transformer architecture, which includes an embedding layer, an attention layer, a fully connected layer, and a linear transformation layer. After the data to be processed is processed and input into the first attention layer of the student model, the data to be processed is linearly transformed to obtain the query vector, key vector, and value vector of the data to be processed.

[0020] S120: Determine the first vector relationship value of each first encoding module and / or the second vector relationship value between any two first encoding modules according to at least one of the query vector, key vector and value vector of each first attention layer.

[0021] The first vector relationship value includes the relationship between the vectors in the first encoding module. Exemplarily, the relationship value between the vectors of each first attention layer in the first encoding module can be obtained first, and then the first vector relationship value of the first encoding module can be determined based on the relationship value of each first attention layer. In each first attention layer, the relationship value between the vectors is determined by the query vector, key vector or value vector in the first attention layer. For example, the first vector relationship value can be determined by the relationship between the query vector and any two of the key vector and the value vector, or the first vector relationship value can be determined by the relationship between the query vector, the key vector and the value vector.

[0022] The second vector relationship value includes the relationship between the vectors of any two first encoding modules in the student model. In the student model, the first attention layers of any two first encoding modules correspond to each other, so the cross-relationship value of the vectors between the two first encoding modules corresponding to each first attention layer can be obtained first, and then the second vector relationship value between the two first encoding modules can be determined based on the cross-relationship value of each first attention layer.

[0023] S130: Input the data to be processed into the teacher model to obtain the query vector, key vector and value vector of each second attention layer in each second encoding module.

[0024] The teacher model can be a trained model used to optimize the student model. The model complexity of the teacher model is greater than the model complexity of the student model. In some embodiments, the teacher model can optimize only one first encoding module in the student model, or can optimize two or more first encoding modules in the student model. For example, when the student model includes three first encoding modules, the three first encoding modules can be an image encoding module, a text encoding module, and a fusion encoding module, respectively. The teacher model can only optimize the image encoding module; or only optimize the text encoding module or the fusion encoding module, which is not specifically limited here, and can also optimize two or three of them.

[0025] The second encoding module is an encoding module in the teacher model, and the teacher model includes at least one second encoding module. Exemplarily, the second encoding module can be an image encoding module for extracting image features of image data; it can also be a text encoding module for extracting text features of text data; it can also be a fusion encoding module for fusing image features and / or text features to obtain fusion features.

[0026] The second encoding module includes at least one second attention layer. The architecture of the second encoding module can also be based on the Transformer architecture, which includes an embedding layer, an attention layer, a fully connected layer, and a linear transformation layer. After the processed data is input into the second attention layer of the teacher model, the processed data is linearly transformed to obtain the query vector, key vector, and value vector of the processed data. It should be noted that the student model is a model with a smaller number of parameters. The first attention layer of the student model is different from the second attention layer of the teacher model. Therefore, during the model optimization process, the student model needs to learn the performance of the second attention layer of the teacher model and improve the first attention layer of the student model, so as to ensure that the performance is not inferior to the teacher model, but also more efficient and simple than the teacher model.

[0027] S140: Determine the third vector relationship value of each second encoding module and / or the fourth vector relationship value between any two second encoding modules according to at least one of the query vector, key vector and value vector of each second attention layer.

[0028] The third vector relationship value includes the relationship between the vectors in the second encoding module. Exemplarily, the relationship value between the vectors of each second attention layer in the second encoding module can be obtained first, and then the third vector relationship value of the second encoding module can be determined based on the relationship value of each second attention layer. In each second attention layer, the relationship value between the vectors is determined by the query vector, key vector or value vector in the second attention layer. For example, the third vector relationship value can be determined by the relationship between the query vector and any two of the key vector and the value vector, or the third vector relationship value can be determined by the relationship between the query vector, the key vector and the value vector.

[0029] The fourth vector relationship value includes the relationship between the vectors of any two second encoding modules in the teacher model. In the teacher model, the second attention layers of any two second encoding modules correspond to each other, so the cross-relationship value of the vectors between the two second encoding modules corresponding to each second attention layer can be obtained first, and then the fourth vector relationship value between the two second encoding modules can be determined based on the cross-relationship value of each second attention layer.

[0030] It should be noted that the time for inputting the data to be processed into the teacher model and the student model respectively can be the student model first, the teacher model first, or the teacher model and the student model at the same time, which is not specifically limited here.

[0031] S150: Determine the first relationship difference between the first vector relationship value of each first encoding module in the student model and the third vector relationship value of the corresponding second encoding module in the teacher model according to the attribute information of the encoding module, and / or determine the second relationship difference between the second vector relationship value in the student model and the corresponding fourth vector relationship value in the teacher model according to the attribute information of the encoding module.

[0032] The attribute information may include functional attributes of the encoding module, such as encoding modules for processing images and processing texts, etc. Before determining the first relationship difference, first determine the corresponding relationship between the first encoding module in the student model and the second encoding module in the teacher model based on the attribute information of the encoding module, for example, determine the first encoding module for processing images in the student model and the second encoding module for processing images in the teacher model as a corresponding relationship, and determine the first encoding module for processing texts in the student model and the second encoding module for processing texts in the teacher model as a corresponding relationship; then determine the first relationship difference between the two based on the first vector relationship value of the corresponding first encoding module and the third vector relationship value of the second encoding module.

[0033] The first relation difference is used to measure the amount of information loss of the first vector relation value compared to the corresponding third vector relation value, that is, the amount of information loss of the first encoding module of the student model compared to the corresponding second encoding module of the teacher model. The first relation difference is the difference between a single first encoding module in the student model and the corresponding second encoding module in the teacher model, which can learn the excellent performance of each second encoding module of the teacher model.

[0034] The second relation difference is usually used when the student model and the teacher model include multiple encoding modules at the same time, and is used to measure the amount of information loss of the second vector relationship value compared to the corresponding fourth vector relationship value. The second relation difference is the difference between the two first encoding modules in the student model and the corresponding two second encoding modules in the teacher model, thereby learning the implicit relationship between different second encoding modules in the teacher model. When the two encoding modules belong to encoding modules with different functional attributes, for example, one is an image encoding module and the other is a text encoding module, the implicit knowledge between the image encoding module and the text encoding module in the teacher model can be fully learned, thereby improving the cross-modal processing capability of the student model.

[0035] S160: Determine a target loss value of the student model according to the first relation difference and / or the second relation difference.

[0036] The target loss value is used to measure whether the student model meets the preset requirements. Exemplarily, in addition to determining the target loss value of the student model based on the first relationship difference and / or the second relationship difference, it can also be determined based on the first loss value of the loss function between the predicted result of the student model and the predicted result of the teacher model, and the second loss value of the loss function between the predicted result of the student model and the actual result. As an example, the target loss value can be determined based on the first relationship difference, the first loss value, and the second loss value. As another example, the target loss value can also be determined based on the second relationship difference, the first loss value, and the second loss value. As another example, the target loss value can also be determined based on the first relationship difference, the second relationship difference, the first loss value, and the second loss value. Among them, the parameters can be adjusted by setting weights to obtain the target loss value.

[0037] S170: Optimize the student model according to the target loss value to obtain an optimized student model.

[0038] After obtaining the target loss value, the student model optimization device adjusts the student model with the purpose of reducing the target loss value to obtain an optimized student model; then the next data to be processed is processed with the optimized student model until the student model meets the preset requirements, that is, the prediction accuracy of the student model meets the requirements.

[0039] It can be seen that the student model optimization method based on distillation learning in the embodiment of the present application determines the first vector relationship value of each first encoding module and / or the second vector relationship value between any two first encoding modules according to at least one of the query vector, key vector and value vector of each attention layer in each first encoding module of the student model; determines the third vector relationship value of each second encoding module and / or the fourth vector relationship value between any two second encoding modules according to at least one of the query vector, key vector and value vector of each attention layer in each second encoding module of the teacher model, thereby not only being able to fully learn the vector relationship of the attention layer within the model, but also being able to learn the vector relationship between multiple encoding modules in the same model; then determining the first relationship difference according to the first vector relationship value and the third vector relationship value and / or determining the second relationship difference according to the second vector relationship value and the fourth vector relationship value; determining the target loss value of the student model according to the first relationship difference and / or the second relationship difference, and optimizing the student model according to the target loss value, thereby optimizing the student model through the difference between the student model and the teacher model in the attention layer, thereby improving the performance of the student model.

[0040] In order to clearly explain the student model optimization method of the embodiment of the present application, various situations are described: As an example, the data to be processed is input into the student model to obtain the query vector, key vector and value vector of each first attention layer in each first encoding module; the first vector relationship value of each first encoding module is determined according to at least one of the query vector, key vector and value vector of each first attention layer; the data to be processed is input into the teacher model to obtain the query vector, key vector and value vector of each second attention layer in each second encoding module; the third vector relationship value of each second encoding module is determined according to at least one of the query vector, key vector and value vector of each second attention layer; the first relationship difference between the first vector relationship value of each first encoding module in the student model and the third vector relationship value of the corresponding second encoding module in the teacher model is determined according to the attribute information of the encoding module; the target loss value of the student model is determined according to the first relationship difference; the student model is optimized according to the target loss value to obtain the optimized student model.

[0041] Among them, the first vector relationship value can be determined according to the query vector of each first attention layer, according to the key vector of each first attention layer, according to the key vector value vector of each first attention layer, according to the query vector and key vector of each first attention layer, according to the query vector and value vector of each first attention layer, according to the key vector and value vector of each first attention layer, or according to the query vector, key vector and value vector of each first attention layer; the second vector relationship value can also be determined according to the query vector of each second attention layer, according to the key vector of each second attention layer, according to the key vector value vector of each second attention layer, according to the query vector and key vector of each second attention layer, according to the query vector and value vector of each second attention layer, according to the key vector and value vector of each second attention layer, or according to the query vector, key vector and value vector of each second attention layer.

[0042] As another example, the data to be processed is input into the student model to obtain the query vector, key vector and value vector of each first attention layer in each first encoding module; the second vector relationship value between any two first encoding modules is determined according to at least one of the query vector, key vector and value vector of each first attention layer; the data to be processed is input into the teacher model to obtain the query vector, key vector and value vector of each second attention layer in each second encoding module; the fourth vector relationship value between any two second encoding modules is determined according to at least one of the query vector, key vector and value vector of each second attention layer; the second relationship difference between the second vector relationship value in the student model and the corresponding fourth vector relationship value in the teacher model is determined according to the attribute information of the encoding module; the target loss value of the student model is determined according to the second relationship difference; the student model is optimized according to the target loss value to obtain the optimized student model.

[0043] As another example, the data to be processed is input into the student model to obtain the query vector, key vector and value vector of each first attention layer in each first encoding module; the first vector relationship value of each first encoding module is determined according to at least one of the query vector, key vector and value vector of each first attention layer, and the second vector relationship value between any two first encoding modules is determined according to at least one of the query vector, key vector and value vector of each first attention layer; the data to be processed is input into the teacher model to obtain the query vector, key vector and value vector of each second attention layer in each second encoding module; the third vector relationship value of each second encoding module is determined according to at least one of the query vector, key vector and value vector of each second attention layer, and the fourth vector relationship value between any two second encoding modules is determined according to at least one of the query vector, key vector and value vector of each second attention layer; Among them, the first relationship difference between the first vector relationship value of each first encoding module in the student model and the third vector relationship value of the corresponding second encoding module in the teacher model can be determined according to the attribute information of the encoding module; the target loss value of the student model can be determined according to the first relationship difference; the student model is optimized according to the target loss value to obtain the optimized student model. The second relationship difference between the second vector relationship value in the student model and the corresponding fourth vector relationship value in the teacher model can also be determined according to the attribute information of the encoding module; the target loss value of the student model can be determined according to the second relationship difference; the student model is optimized according to the target loss value to obtain the optimized student model. The first relationship difference between the first vector relationship value of each first encoding module in the student model and the third vector relationship value of the corresponding second encoding module in the teacher model can also be determined according to the attribute information of the encoding module, and the second relationship difference between the second vector relationship value in the student model and the corresponding fourth vector relationship value in the teacher model can be determined according to the attribute information of the encoding module; the target loss value of the student model can be determined according to the first relationship difference and the second relationship difference; the student model is optimized according to the target loss value to obtain the optimized student model.

[0044] Based on the above embodiments, the embodiments of the present application adopt Figure 2 The flowchart details how to obtain the first vector relationship value of the first encoding module and the second vector relationship value between any two first encoding modules. Figure 2 , Figure 2 yes Figure 1 A flowchart of an exemplary embodiment of step S120 in the student model optimization method is shown. Specifically, the process of determining the first vector relationship value of each first encoding module and / or the second vector relationship value between any two first encoding modules according to at least one of the query vector, key vector and value vector of each first attention layer in step S120 includes the following steps: S210: Determine the first target vector similarity of each first attention layer according to the vector similarities among the query vector, the key vector and the value vector of each first attention layer in the first encoding module.

[0045] After the student model optimization device inputs the data to be processed into the student model, it extracts the query vector, key vector and value vector of each first attention layer in each first encoding module; then obtains the vector similarity between the query vector, key vector and value vector, and the vector similarity characterizes the vector relationship between the query vector, key vector and value vector; determines the first target vector similarity of each first attention layer according to the vector similarity.

[0046] In some embodiments, the first target vector similarity may be determined by the vector similarity between the query vector and the key vector, or by the vector similarity between the query vector and the value vector, or by the vector similarity between the key vector and the value vector.

[0047] In some other embodiments, the first target vector similarity can be determined by the vector similarity between the query vector, the key vector and the value vector. Specifically, at each first attention layer, the sum of the vector similarities between the query vector and the query vector, the query vector and the key vector, and the query vector and the value vector is calculated to obtain the vector similarity of the query vector; at each first attention layer, the sum of the vector similarities between the key vector and the query vector, the key vector and the key vector, and the key vector and the value vector is calculated to obtain the vector similarity of the key vector; at each first attention layer, the sum of the vector similarities between the value vector and the query vector, the value vector and the key vector, and the value vector and the value vector is calculated to obtain the vector similarity of the value vector; the sum of the vector similarities between the vector similarity of the query vector, the vector similarity of the key vector, and the vector similarity of the value vector in the same first attention layer is determined as the first target vector similarity of the corresponding first attention layer.

[0048] For each second attention layer of each second encoding module in the teacher model, the target vector similarity of each second attention layer can also be calculated according to the above method.

[0049] For ease of understanding, the query vector can be represented as Q, the key vector can be represented as K, and the value vector can be represented as V. QKV is extracted from each first attention layer. Then, at each first attention layer, the sum of the vector similarities between Q and QKV is calculated to obtain the vector similarity of the query vector, which can be expressed as ; Calculate the sum of the vector similarities between K and QKV to obtain the vector similarity of the key vector, which can be expressed as ; Calculate the sum of the vector similarities between V and QKV to obtain the vector similarity of the value vector, which can be expressed as ;Will , and The sum between them is determined as the first target vector similarity corresponding to the first attention layer.

[0050] S220: Sum the similarities of the first target vectors of each first attention layer to obtain the first vector relationship value of the first encoding module.

[0051] After obtaining the first target vector similarities of each first attention layer in the first coding module, the student model optimization device sums the first target vector similarities of all first attention layers in the first coding module to obtain the first vector relationship value of the first coding module. In other embodiments, the student model optimization device may also first sum the query vectors in each first attention layer of the first coding module to obtain the target query vector; sum the key vectors in each first attention layer of the first coding module to obtain the target key vector; sum the median vectors in each first attention layer of the first coding module to obtain the target value vector; and then perform vector similarity calculation based on the target query vector, the target key vector and the target value vector to obtain the first vector relationship value of the first coding module.

[0052] Exemplarily, at each first attention layer, the sum of the vector similarities between the query vector and the query vector, the query vector and the key vector, and the query vector and the value vector is calculated, and the calculation process of obtaining the vector similarity of the query vector includes: respectively calculating the vector product between the transpose of the query vector and the query vector, the key vector and the value vector; respectively activating each vector product to obtain the vector similarity between the query vector and the query vector, the vector similarity between the query vector and the key vector, and the vector similarity between the query vector and the value vector; summing the vector similarity between the query vector and the query vector, the vector similarity between the query vector and the key vector, and the vector similarity between the query vector and the value vector to obtain the vector similarity of the query vector. In this way, the relationship between each vector can be learned through simple vector similarity.

[0053] At each first attention layer, the sum of the vector similarities between the key vector and the query vector, the key vector and the key vector, and the key vector and the value vector is calculated to obtain the vector similarity of the key vector; the sum of the vector similarities between the value vector and the query vector, the value vector and the key vector, and the value vector and the value vector is calculated to obtain the vector similarity of the value vector. The calculation process can refer to the above-mentioned query vector calculation process.

[0054] In other embodiments, the method of calculating the vector similarity between query vectors and query vectors, query vectors and key vectors, query vectors and value vectors may also include but is not limited to cosine similarity, Euclidean distance, or Hamming distance.

[0055] Exemplarily, the calculation of vector similarity between any two vectors satisfies the following formula:

[0056] in, Indicates Layer attention layer vector and vector The vector similarity between ; represents the softmax activation function; Indicates The transpose of a vector in the attention layer, belongs to [1,3], when hour, Indicates The transpose of the query vector in the layer, when hour, Indicates The transpose of the key vector of the layer, when hour, Indicates The transpose of the layer's value vector; Indicates A vector of the attention layer; belongs to [1,3], when hour, Indicates The query vector in the layer, when hour, Indicates The key vector of the layer, when hour, Indicates The value vector of the layer; Indicates Attention layer, Represents the dimension of the attention layer in the encoding module.

[0057] S230: Determine the second target vector similarity of each first attention layer based on the vector similarity between the query vector, key vector and value vector of each first attention layer of one first encoding module in any two first encoding modules and the query vector, key vector and value vector of the corresponding first attention layer of the other first encoding module.

[0058] When there are multiple first encoding modules in the student model, the relationship between any two of the first encoding modules can also be learned. For the convenience of description, any two first encoding modules in the student model can be the first student encoding module and the second student encoding module. Specifically, the student model optimization device inputs the data to be processed into the first student encoding module and the second student encoding module of the student model, and obtains the query vector, key vector and value vector of each first attention layer in the first student encoding module and the second student encoding module; then obtains the vector similarity between the query vector, key vector and value vector of each first attention layer in the first student encoding module and the query vector, key vector and value vector of the corresponding attention layer in the second student encoding module; and determines the second target vector similarity between the first student encoding module and the second student encoding module on each first attention layer according to the vector similarity.

[0059] In some embodiments, the second target vector similarity can be determined by the vector similarity between the query vector in each first attention layer in the first student coding module and the query vector in the corresponding first attention layer in the second student coding module; it can also be determined by the vector similarity between the key vector in each first attention layer in the first student coding module and the key vector in the corresponding first attention layer in the second student coding module; it can also be determined by the vector similarity between the value vector in each first attention layer in the first student coding module and the value vector of the corresponding first attention layer in the second student coding module; it can also be determined by the vector similarity between the query vector, key vector and value vector of each first attention layer in the first student coding module and the query vector, key vector and value vector of the corresponding attention layer in the second student coding module.

[0060] In some other embodiments, the process of determining the second target vector similarity may further include the following steps: at each first attention layer, obtaining the sum of vector similarities between a query vector in a first coding module and a query vector, key vector and value vector in another first coding module to obtain a first cross-vector similarity; at each first attention layer, obtaining the vector similarity between a key vector in a first coding module and a query vector, key vector and value vector in another first coding module to obtain a second cross-vector similarity; at each first attention layer, obtaining the vector similarity between a value vector in a first coding module and a query vector, key vector and value vector in another first coding module to obtain a third cross-vector similarity; determining the sum of the first cross-vector similarity, the second cross-vector similarity and the third cross-vector similarity as the second target vector similarity of the corresponding first attention layer in the corresponding two first coding modules. One of the first coding modules can be expressed as a first student coding module, and the other first coding module can be expressed as a second student coding module.

[0061] For any two second attention layers of each second encoding module in the teacher model, the target vector similarity of any two second encoding modules on each second attention layer can also be calculated according to the above method.

[0062] As an example, the query vector can be represented as Q, the key vector can be represented as K, and the value vector can be represented as V. QKV is extracted from the first attention layer of each first encoding module. Then, at each first attention layer, the sum of the vector similarities between Q in the first student encoding module and QKV in the second student encoding module is calculated to obtain the cross-vector similarity of the query vector in the first student encoding module, and the sum of the vector similarities between Q in the second student encoding module and QKV in the first student encoding module is calculated to obtain the cross-vector similarity of the query vector in the second student encoding module. The sum of the two obtains the first cross-vector similarity of the first student encoding module and the second student encoding module; the sum of the vector similarities between K in the first student encoding module and QKV in the second student encoding module is calculated to obtain the cross-vector similarity of the key vector in the first student encoding module, and Calculate the sum of the vector similarities between K in the second student coding module and QKV in the first student coding module to obtain the cross-vector similarity of the key vector in the second student coding module, and the sum of the two obtains the second cross-vector similarity of the first student coding module and the second student coding module; calculate the sum of the vector similarities between V in the first student coding module and QKV in the second student coding module to obtain the cross-vector similarity of the median vector of the first student coding module, and calculate the sum of the vector similarities between V in the second student coding module and QKV in the first student coding module to obtain the cross-vector similarity of the median vector of the second student coding module, and the sum of the two obtains the third cross-vector similarity of the first student coding module and the second student coding module; determine the sum of the first cross-vector similarity, the second cross-vector similarity and the third cross-vector similarity as the second target vector similarity.

[0063] If the student model includes an image encoding module, a text encoding module and a fusion encoding module, in some embodiments, the second target vector similarity between the image encoding module and the text encoding module can be obtained; the second target vector similarity between the image encoding module and the fusion encoding module can also be obtained; the second target vector similarity between the text encoding module and the fusion encoding module can also be obtained.

[0064] S240: Sum the similarities of the second target vectors of each first attention layer to obtain the second vector relationship value corresponding to the two first encoding modules.

[0065] After obtaining the similarity of the second target vectors of each first attention layer in any two first coding modules, the student model optimization device sums up the similarities of the second target vectors of all first attention layers to obtain the second vector relationship value between the corresponding two first coding modules. In other embodiments, the student model optimization device may also first sum up the query vectors in each first attention layer of all first coding modules to obtain a target query vector; sum up the key vectors in each first attention layer of the first coding module to obtain a target key vector; sum up the value vectors in each first attention layer of the first coding module to obtain a target value vector; and then calculate the second vector relationship value between any two coding modules based on the target query vector, the target key vector and the target value vector.

[0066] Exemplarily, at each first attention layer, the calculation of the vector similarity between each vector in the first student encoding module and the second student encoding module satisfies the following formula:

[0067] in, represents the vector similarity between two vectors, represents the softmax activation function, Indicates the first student coding module The transpose of a vector in the attention layer, belongs to [1,3], when hour, Indicates the first student coding module The query vector of the layer, when hour, Indicates the first student coding module The key vector of the layer, when hour, Indicates the first student coding module The value vector of the layer; Indicates the second student coding module A vector of the attention layer, belongs to [1,3], when hour, Indicates the second student coding module The query vector of the layer, when hour, Indicates the second student coding module The key vector of the layer, when hour, Indicates the second student coding module The value vector of the layer; Indicates Attention layer, represents the dimension of the attention layer in the encoding module; since the dimensions of each first encoding module in the student model may be different, learnable weights are required and The dimensions of the first encoding module are aligned.

[0068] Among them, the third vector relationship value of the teacher model can refer to the calculation process of the first vector relationship value of the student model; the fourth vector relationship value of the teacher model can refer to the calculation process of the second vector relationship value of the student model, which will not be repeated here.

[0069] After obtaining the first vector relationship value of the student model and the third vector relationship value of the teacher model, the student model optimization device can use a preset loss function to calculate the loss of the first vector relationship value and the third vector relationship value to obtain the initial relationship difference between the first vector relationship value and the third vector relationship value; since the teacher model and the student model belong to different models, after obtaining the initial relationship difference, it is also necessary to weight the initial relationship difference according to the alignment weight between the student model and the teacher model to obtain the first relationship difference. Among them, the preset loss function can be kldivergence (Kullback-Leibler Divergence, KL divergence), root mean square error (Root Mean SquaredError, RMSE), etc.

[0070] In other embodiments, the student model optimization device may further use a preset loss function to perform loss calculation on the second vector relationship value and the fourth vector relationship value after obtaining the second vector relationship value of the student model and the fourth vector relationship value of the teacher model, so as to obtain an initial relationship difference between the second vector relationship value and the fourth vector relationship value; since the teacher model and the student model belong to different models, after obtaining the initial relationship difference, the initial relationship difference needs to be weighted according to the alignment weight between the student model and the teacher model to obtain the second relationship difference.

[0071] The student model optimization device can determine the target loss value of the student model based on the first relationship difference. Specifically, the sum of the first relationship differences between each first vector relationship value and the corresponding third vector relationship value is calculated, and the sum of the first relationship differences is determined as the target loss value of the student model; the target loss value of the student model can also be determined based on the second relationship difference. Specifically, the sum of the second relationship differences between each second vector relationship value and the corresponding fourth vector relationship value is calculated, and the sum of the second relationship differences is determined as the target loss value of the student model; the target loss value of the student model can also be determined based on the first relationship difference and the second relationship difference. Specifically, the sum of the first relationship difference and the second relationship difference can be determined as the target loss value of the student model. In other embodiments, the target loss value may also include the difference between the predicted result of the student model and the predicted result of the teacher model and the direct difference between the predicted result of the student model and the actual result.

[0072] Finally, as an example, the student model can be a multimodal model, such as GLIP (Grounded Language-Image Pre-training). The structure of distillation learning can be found in Figure 3 , the student model and the teacher model both include an image encoding module, a text encoding module and a fusion encoding module, at least one first encoding module of the student model includes a student image encoding module, a student text encoding module and a student fusion encoding module, at least one second encoding module of the teacher model includes a teacher image encoding module, a teacher text encoding module and a teacher fusion encoding module, the image encoding module can be a swin backbone, the text encoding module can be a BERT (Bidirectional Encoder Representations from Transformers), the fusion encoding module can be a DyHead (Dynamic Head), the fusion encoding module is used to output fusion image features and fusion text features, and ATSS (Adaptive Training Sample Selection) can be used for target detection training for fusion image features.

[0073] The data to be processed includes image data to be processed and text data to be processed. The image data to be processed is input into the student image encoding module for encoding processing to obtain the query vector, key vector and value vector of each first attention layer in the student image encoding module, as well as the image features output by the student image encoding module; the text data to be processed is input into the student text encoding module for encoding processing to obtain the query vector, key vector and value vector of each first attention layer in the student text encoding module, as well as the text features output by the student text encoding module; the image features output by the student image encoding module and the text features output by the student text encoding module are input into the student fusion encoding module to obtain the query vector, key vector and value vector of each first attention layer in the student fusion encoding module. Before the image features are input into the fusion encoding module, they can be processed through the feature pyramid network to obtain image features of different scales.

[0074] The image data to be processed is input into the teacher image encoding module for encoding processing, and the query vector, key vector and value vector of each second attention layer in the teacher image encoding module, as well as the image features output by the teacher image encoding module are obtained; the text data to be processed is input into the teacher text encoding module for encoding processing, and the query vector, key vector and value vector of each second attention layer in the teacher text encoding module, as well as the text features output by the teacher text encoding module are obtained; the image features output by the teacher image encoding module and the text features output by the teacher text encoding module are input into the teacher fusion encoding module, and the query vector, key vector and value vector of each second attention layer in the teacher fusion encoding module are obtained.

[0075] Then, the first vector relationship value of the student image encoding module and the first vector relationship value of the student text encoding module are calculated respectively, and the third vector relationship value of the teacher image encoding module and the third vector relationship value of the teacher text encoding module are calculated respectively. Exemplarily, the calculation of the first vector relationship value and the third vector relationship value satisfies the following formula:

[0076] in, represents a first vector relationship value or a third vector relationship value; represents the number of attention layers, belongs to [1,L]; belongs to [1,3], belongs to [1,3], representing query vector, key vector and value vector respectively; Indicates Layer attention layer vector and vector The vector similarity between ; is a hyperparameter, representing the vector With vector The weight between .

[0077] Calculate the second vector relationship value between the student image encoding module and the student text encoding module, and the fourth vector relationship value between the teacher image encoding module and the teacher text encoding module. Exemplarily, the calculation of the second vector relationship value or the fourth vector relationship value satisfies the following formula:

[0078] in, represents the second vector relation value or the fourth vector relation value; represents the number of attention layers, belongs to [1,L]; belongs to [1,3], belongs to [1,3], representing query vector, key vector and value vector respectively; Represents the image coding module Layer attention layer vector and the text encoding module Layer attention layer vector The vector similarity between ; is a hyperparameter, representing the vector With vector The weight between .

[0079] For the first vector relationship value of the student fusion coding module and the third vector relationship value of the teacher fusion coding module, since the fusion coding module contains not only image features but also text features, its value vector is different from that of the image coding module and the text coding module, and there will be image value vectors of the image data to be processed and text value vectors of the text data to be processed at the same time. Exemplarily, the calculation of the fusion coding module satisfies the following formula:

[0080] in, represents the first vector relationship value of the student fusion coding module or the third vector relationship value of the teacher fusion coding module, represents the number of attention layers of the fusion encoding module, belongs to [1, ]; belongs to [1,4], belongs to [1,4], representing query vector, key vector, image value vector and text value vector respectively; Indicates Layer attention layer vector and vector The vector similarity between ; is a hyperparameter, representing the vector With vector The weight between .

[0081] Furthermore, The calculation satisfies the following formula:

[0082] in, Indicates Layer attention layer vector and vector The vector similarity between ; represents the softmax activation function, Indicates The transpose of a vector in the attention layer, belongs to [1,4], when hour, Indicates The transpose of the query vector in the layer, when hour, Indicates The transpose of the key vector of the layer, when hour, Indicates The transpose of the layer's image value vector; when hour, Indicates The transpose of the layer's vector of text values; Indicates A vector of the attention layer; belongs to [1,4], when hour, Indicates The query vector in the layer, when hour, Indicates The key vector of the layer, when hour, Indicates The image value vector of the layer, when hour, Indicates A vector of text values ​​for the layer; Indicates Attention layer, Represents the dimension of the attention layer in the fusion encoding module.

[0083] After obtaining the first vector relationship value of the student image coding module, the first vector relationship value of the student text coding module, the first vector relationship value of the student fusion coding module, the third vector relationship value of the teacher image coding module, the third vector relationship value of the teacher text coding module and the third vector relationship value of the teacher fusion coding module, as well as the second vector relationship value between the student image coding module and the student text coding module, and the fourth vector relationship value between the teacher image coding module and the teacher text coding module, calculate the first relationship difference between the first vector relationship value of the student image coding module and the third vector relationship value of the teacher image coding module; calculate the first relationship difference between the first vector relationship value of the student text coding module and the third vector relationship value of the teacher text coding module; calculate the first relationship difference between the first vector relationship value of the student fusion coding module and the third vector relationship value of the teacher fusion coding module; calculate the second relationship difference between the second vector relationship value and the fourth vector relationship value; combine these four relationship differences to obtain the target loss value of the student model. Exemplarily, its calculation formula satisfies the following formula:

[0084] in, represents the target loss value, represents the KL divergence calculation, represents the first vector relationship value of the student image encoding module, represents the third vector relationship value of the teacher image encoding module, represents the alignment weight between the student image encoding module and the teacher image encoding module; Represents the first vector relationship value of the student text encoding module, Represents the third vector relationship value of the teacher text encoding module, represents the alignment weight between the student text encoding module and the teacher text encoding module; a second vector relationship value representing the student model; Represents the fourth vector relationship value of the teacher model, represents the alignment weight of the student model and the teacher model; Represents the first vector relationship value of the student fusion coding module, Represents the third vector relationship value of the teacher fusion coding module, represents the alignment weight between the student fusion coding module and the teacher fusion coding module; Represents the weights obtained through multimodal model training.

[0085] See also Figure 4 , Figure 4It is a structural diagram of an exemplary embodiment of the student model optimization device shown in the present application. The student model optimization device 400 includes an input module 410, a relationship value determination module 420, a relationship difference determination module 430, a loss value determination module 440 and an optimization module 450. The input module 410 is used to input the data to be processed into the student model to obtain the query vector, key vector and value vector of each first attention layer in each first encoding module, and input the data to be processed into the teacher model to obtain the query vector, key vector and value vector of each second attention layer in each second encoding module; the relationship value determination module 420 is used to determine the first vector relationship value of each first encoding module and / or the second vector relationship value between any two first encoding modules according to at least one of the query vector, key vector and value vector of each first attention layer, and the query vector, key vector and value vector of each second attention layer according to each second attention layer. At least one of the sum value vectors determines the third vector relationship value of each second encoding module and / or the fourth vector relationship value between any two second encoding modules; the relationship difference determination module 430 is used to determine the first relationship difference between the first vector relationship value of each first encoding module in the student model and the third vector relationship value of the corresponding second encoding module in the teacher model according to the attribute information of the encoding module, and / or, determine the second relationship difference between the second vector relationship value in the student model and the corresponding fourth vector relationship value in the teacher model according to the attribute information of the encoding module; the loss value determination module 440 is used to determine the target loss value of the student model according to the first relationship difference and / or the second relationship difference; the optimization module 450 is used to optimize the student model according to the target loss value to obtain the optimized student model.

[0086] In the above scheme, the student model optimization device determines the first vector relationship value of each first encoding module and / or the second vector relationship value between any two first encoding modules according to at least one of the query vector, key vector and value vector of each attention layer in each first encoding module of the student model; determines the third vector relationship value of each second encoding module and / or the fourth vector relationship value between any two second encoding modules according to at least one of the query vector, key vector and value vector of each attention layer in each second encoding module of the teacher model, thereby not only being able to fully learn the vector relationship of the attention layer within the model, but also being able to learn the vector relationship between multiple encoding modules in the same model; then determining the first relationship difference according to the first vector relationship value and the third vector relationship value and / or determining the second relationship difference according to the second vector relationship value and the fourth vector relationship value; determining the target loss value of the student model according to the first relationship difference and / or the second relationship difference, and optimizing the student model according to the target loss value, thereby optimizing the student model through the difference between the student model and the teacher model in the attention layer, thereby improving the performance of the student model.

[0087] Among them, the functions of each module can be found in the implementation example of the student model optimization method, which will not be repeated here.

[0088] In order to implement the student model optimization method of the above embodiment, this application proposes another electronic device, which can be specifically referred to as Figure 5 , Figure 5 It is a structural schematic diagram of an embodiment of an electronic device provided by the present application.

[0089] The electronic device 500 includes a memory 510 and a processor 520 , wherein the memory 510 and the processor 520 are coupled.

[0090] The memory 510 is used to store program data, and the processor 520 is used to execute the program data to implement the student model optimization method of the above embodiment.

[0091] In this embodiment, the processor 520 may also be referred to as a CPU (Central Processing Unit). The processor 520 may be an integrated circuit chip having signal processing capabilities. The processor 520 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 520 may also be any conventional processor, etc.

[0092] The present application also provides a computer-readable storage medium, such as Figure 6 As shown, the computer-readable storage medium 600 is used to store program data 610. When the program data 610 is executed by the processor, it is used to implement the student model optimization method in the method embodiment of the present application.

[0093] The method involved in the embodiment of the student model optimization method of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0094] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A student model optimization method based on distillation learning, characterized in that: The student model optimization method based on distillation learning is applied to a student model optimization device, the student model optimization device includes a student model and a teacher model, the student model includes at least one first encoding module, each of the first encoding modules includes at least one first attention layer, the teacher model includes at least one second encoding module, each of the second encoding modules includes at least one second attention layer, the method includes: Inputting the data to be processed into the student model to obtain a query vector, a key vector and a value vector of each first attention layer in each first encoding module; Determine a first vector relationship value of each of the first encoding modules and / or a second vector relationship value between any two first encoding modules according to at least one of the query vector, the key vector, and the value vector of each first attention layer; Inputting the data to be processed into the teacher model to obtain a query vector, a key vector and a value vector of each second attention layer in each second encoding module; Determine a third vector relationship value of each of the second encoding modules and / or a fourth vector relationship value between any two second encoding modules according to at least one of the query vector, the key vector, and the value vector of each second attention layer; Determine, according to the attribute information of the encoding module, a first relationship difference between a first vector relationship value of each of the first encoding modules in the student model and a third vector relationship value of the corresponding second encoding module in the teacher model, and / or determine, according to the attribute information of the encoding module, a second relationship difference between a second vector relationship value in the student model and a fourth vector relationship value in the teacher model; Determine a target loss value of the student model according to the first relationship difference and / or the second relationship difference; The student model is optimized according to the target loss value to obtain an optimized student model.

2. The student model optimization method based on distillation learning according to claim 1, characterized in that: The step of determining the first vector relationship value of each first encoding module and / or the second vector relationship value between any two first encoding modules according to at least one of the query vector, the key vector and the value vector of each first attention layer comprises: Determining first target vector similarities of each first attention layer according to vector similarities between query vectors, key vectors, and value vectors of each first attention layer in the first encoding module; Summing the similarities of the first target vectors of each first attention layer to obtain the first vector relationship value of the first encoding module; and / or, Determine the second target vector similarity of each first attention layer according to the vector similarity between the query vector, the key vector and the value vector of each first attention layer of one first encoding module in any two first encoding modules and the query vector, the key vector and the value vector of the corresponding first attention layer of the other first encoding module; The similarities of the second target vectors of each first attention layer are summed to obtain the second vector relationship value corresponding to the two first encoding modules.

3. The student model optimization method based on distillation learning according to claim 2, characterized in that: The step of determining the similarity of the first target vectors of each first attention layer according to the vector similarities between the query vector, the key vector and the value vector of each first attention layer in the first encoding module comprises: At each first attention layer, calculating the sum of vector similarities between the query vector and the query vector, between the query vector and the key vector, and between the query vector and the value vector to obtain the vector similarity of the query vector; At each first attention layer, calculate the sum of vector similarities between the key vector and the query vector, between the key vector and the key vector, and between the key vector and the value vector to obtain the vector similarity of the key vector; At each first attention layer, calculate the sum of vector similarities between the value vector and the query vector, between the value vector and the key vector, and between the value vector and the value vector to obtain the vector similarity of the value vector; The sum of the vector similarities of the query vector, the key vector and the value vector in the same first attention layer is determined as the first target vector similarity corresponding to the first attention layer.

4. The student model optimization method based on distillation learning according to claim 3, characterized in that: The step of calculating the sum of vector similarities between the query vector and the query vector, between the query vector and the key vector, and between the query vector and the value vector at each first attention layer to obtain the vector similarity of the query vector comprises: calculating the vector product between the transpose of the query vector and the query vector, the key vector and the value vector respectively; Perform activation processing on each vector product respectively to obtain the vector similarity between the query vector and the query vector, the vector similarity between the query vector and the key vector, and the vector similarity between the query vector and the value vector; The vector similarity between the query vector and the query vector, the vector similarity between the query vector and the key vector, and the vector similarity between the query vector and the value vector are summed to obtain the vector similarity of the query vector.

5. The student model optimization method based on distillation learning according to claim 2, characterized in that: The step of determining the similarity of the second target vectors of each first attention layer according to the vector similarity between the query vector, the key vector and the value vector of each first attention layer of one first encoding module in any two first encoding modules and the query vector, the key vector and the value vector of the corresponding first attention layer of the other first encoding module comprises: At each first attention layer, obtaining the sum of vector similarities between a query vector in a first encoding module and a query vector, a key vector, and a value vector in another first encoding module to obtain a first cross vector similarity; At each first attention layer, obtaining a vector similarity between a key vector in a first encoding module and a query vector, a key vector, and a value vector in another first encoding module to obtain a second cross vector similarity; At each first attention layer, obtaining a vector similarity between a value vector in a first encoding module and a query vector, a key vector, and a value vector in another first encoding module to obtain a third cross vector similarity; The sum of the first cross-vector similarity, the second cross-vector similarity and the third cross-vector similarity is determined as the second target vector similarity corresponding to the first attention layer in the two first encoding modules.

6. The student model optimization method based on distillation learning according to claim 1, characterized in that: At least one first encoding module of the student model includes a student image encoding module, a student text encoding module and a student fusion encoding module, the data to be processed includes image data to be processed and text data to be processed, and the step of inputting the data to be processed into the student model to obtain the query vector, key vector and value vector of each first attention layer in each first encoding module includes: Input the image data to be processed into the student image encoding module for encoding processing, and obtain the query vector, key vector and value vector of each first attention layer in the student image encoding module, as well as the image features output by the student image encoding module; Input the text data to be processed into the student text encoding module for encoding processing, and obtain the query vector, key vector and value vector of each first attention layer in the student text encoding module, as well as the text features output by the student text encoding module; The image features output by the student image encoding module and the text features output by the student text encoding module are input into the student fusion encoding module to obtain the query vector, key vector and value vector of each first attention layer in the student fusion encoding module.

7. The student model optimization method based on distillation learning according to claim 1, characterized in that: The step of determining the first relationship difference between the first vector relationship value of each of the first encoding modules in the student model and the third vector relationship value of the corresponding second encoding module in the teacher model according to the attribute information of the encoding module comprises: Using a preset loss function to perform loss calculation on the first vector relationship value and the third vector relationship value to obtain an initial relationship difference between the first vector relationship value and the third vector relationship value; The initial relationship difference is weighted according to the alignment weight between the student model and the teacher model to obtain the first relationship difference.

8. The student model optimization method based on distillation learning according to claim 1, characterized in that: The step of determining the target loss value of the student model according to the first relationship difference and / or the second relationship difference comprises: Determine the target loss value of the student model according to the first relationship difference, or, The target loss value of the student model is determined according to the second relationship difference, or, A target loss value for the student model is determined according to the first relationship difference and the second relationship difference.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: include: Program data is stored, and when the program data is executed by a processor, it is used to implement the method according to any one of claims 1 to 8.

Images