Distillation training method and device based on image-text pair, terminal and medium

Abstract

The invention discloses a distillation training method and device based on an image-text pair, a terminal and a medium. According to the method, the image-text pair is adopted for knowledge distillation, image representation and text representation can be aligned, combined distillation of multiple modes is achieved, and the method is more suitable for open set detection. And the model parameters are updated through one or more loss values, so that different training requirements can be met, and the model performance is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of open set target detection technology, and in particular to a distillation training method, device, terminal and medium based on image-text pairs. Background Art

[0002] Object detection technology is an important part of the field of computer vision, which aims to identify and locate target objects in images or videos.

[0003] Existing target detection technologies include closed-set target detection and open-set target detection. Traditional target detection algorithms belong to closed-set target detection algorithms. In closed-set target detection, the model only processes the categories in the training set during the training and testing phases. The open-set target detection model only contacts a part of the data of known category targets during the training phase, but may face unknown category targets during the testing phase. This requires the model to have a certain generalization ability and be able to identify new category targets that do not appear in the training set. Knowledge Distillation is a widely used training method for improving the performance and generalization ability of deep neural networks. In distillation training, the student model can improve its performance with the help of the knowledge of the teacher model. However, the existing distillation training methods based on target detection algorithms have problems such as single training modality and limited supervision signals, which leads to poor distillation training results.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a distillation training method, device, terminal and medium based on image-text pairs in response to the above-mentioned defects of the prior art, aiming to solve the problem that the existing distillation training method based on the target detection algorithm has a single training mode, resulting in poor distillation training effect.

[0006] The technical solution adopted by the present invention to solve the problem is as follows:

[0007] In a first aspect, an embodiment of the present invention provides a distillation training method based on an image-text pair, the method comprising:

[0008] Based on the image-text pairs, a first model is used to obtain a plurality of first image representations and a plurality of first text representations;

[0009] Obtaining an object query according to the plurality of first image representations, wherein the object query includes a query content and a query location;

[0010] Perform a first loss calculation according to the object query to obtain a first loss value;

[0011] Based on the image-text pairs, a second model is used to obtain a plurality of second image representations;

[0012] Perform a second loss calculation based on the first text representation and the second image representation to obtain a second loss value;

[0013] Obtaining a third loss value by performing a third loss calculation based on the query position of the object query;

[0014] Obtaining a comprehensive loss value according to at least one of the first loss value, the second loss value, and the third loss value;

[0015] The parameters of the second model are updated based on the comprehensive loss value.

[0016] In one implementation, obtaining an object query according to the plurality of first image representations, the object query including query content and query location, includes:

[0017] selecting a target image representation from the plurality of first image representations;

[0018] A query content of an object query is determined according to the target image representation, and a query position of the object query is determined according to a position of the target image representation in the plurality of first image representations.

[0019] In one implementation, performing a first loss calculation according to the object query to obtain a first loss value includes:

[0020] De-noising the object query to obtain a de-noised object query;

[0021] A first loss calculation is performed according to the object query and the denoised object query to obtain a first loss value.

[0022] In one implementation, denoising the object query to obtain a denoised object query includes:

[0023] Randomly extracting a content portion of the object query to obtain a query content of the denoised object query;

[0024] The target box of the image-text pair is sampled to obtain a query position of the denoised object query.

[0025] In one implementation, performing a first loss calculation according to the object query and the denoised object query to obtain a first loss value includes:

[0026] Passing the object query and the denoised object query through a decoder of the first model and a decoder of the second model, respectively obtaining a first prediction result and a second prediction result, wherein the prediction result includes a prediction box and a prediction category;

[0027] A first loss calculation is performed according to the first prediction result and the second prediction result to obtain a first loss value.

[0028] In one implementation, performing a second loss calculation according to the first text representation and the second image representation to obtain a second loss value includes:

[0029] fusing the first text representation and the second image representation to obtain a fused representation;

[0030] A second loss calculation is performed according to the fused representation to obtain a second loss value.

[0031] In one implementation, fusing the first text representation and the second image representation to obtain a fused representation includes:

[0032] According to the first text representation and the second image representation, a first cross attention from image to text and a second cross attention from text to image are calculated respectively through a cross attention mechanism;

[0033] A fused representation is obtained based on the first cross-attention and the second cross-attention.

[0034] In one implementation, obtaining a third loss value by calculating a third loss based on the query position of the object query includes:

[0035] Obtaining a third prediction result and a fourth prediction result through the first model and the second model respectively according to the query position of the object query;

[0036] Obtaining a fourth loss value by performing a third loss calculation based on the third prediction result and the fourth prediction result;

[0037] A fifth loss value is obtained by calculating the third loss based on the fourth prediction result and the corresponding data annotation.

[0038] The third loss value is calculated according to the fourth loss value and / or the fifth loss value.

[0039] In a second aspect, an embodiment of the present invention further provides a distillation training device based on image-text pairs, the device comprising:

[0040] A first feature extraction module, configured to obtain a plurality of first image representations and a plurality of first text representations based on the image-text pair through a first model;

[0041] A query generation module, configured to obtain an object query according to the plurality of first image representations, wherein the object query includes a query content and a query position;

[0042] A first loss calculation module, configured to perform a first loss calculation according to the object query to obtain a first loss value;

[0043] A second feature extraction module, configured to obtain a plurality of second image representations based on the image-text pairs through a second model;

[0044] A second loss calculation module, configured to perform a second loss calculation based on the first text representation and the second image representation to obtain a second loss value;

[0045] A third loss calculation module, configured to obtain a third loss value by performing a third loss calculation based on a query position of the object query;

[0046] a comprehensive loss calculation module, configured to obtain a comprehensive loss value according to at least one of the first loss value, the second loss value and the third loss value;

[0047] A parameter updating module is used to update the parameters of the second model based on the comprehensive loss value.

[0048] In a third aspect, an embodiment of the present invention further provides a terminal, comprising a memory and one or more processors; the memory stores one or more programs; the program comprises instructions for executing any of the above-described image-text pair-based distillation training methods; and the processor is used to execute the program.

[0049] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a plurality of instructions are stored, wherein the instructions are suitable for being loaded and executed by a processor to implement the steps of any of the above-described image-text pair-based distillation training methods.

[0050] Beneficial effects of the present invention: The embodiment of the present invention uses image-text pairs for knowledge distillation, which can align image representation and text representation, realize joint distillation of multiple modalities, and is more suitable for open set detection. And by updating model parameters through one or more loss values, different training requirements can be met and model performance can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 It is a flowchart of the image-text pair-based distillation training method provided in an embodiment of the present invention.

[0053] Figure 2 It is a logical schematic diagram of various distillation training methods provided by embodiments of the present invention.

[0054] Figure 3 It is a logical schematic diagram of text-image fusion provided by an embodiment of the present invention.

[0055] Figure 4 This is a graph of experimental results based on the COCO evaluation dataset provided by an embodiment of the present invention.

[0056] Figure 5 It is a diagram of experimental results based on the LVIS evaluation data set provided by an embodiment of the present invention.

[0057] Figure 6 It is a module schematic diagram of a distillation training device based on image-text pairs provided in an embodiment of the present invention.

[0058] Figure 7 It is a principle block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The present invention discloses a distillation training method, device, terminal and medium based on image-text pairs. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0060] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0061] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0062] In view of the above-mentioned defects of the prior art, the present invention provides a distillation training method based on image-text pairs, wherein the method obtains several first image representations and several first text representations through a first model based on the image-text pairs; obtains an object query based on the several first image representations, and the object query includes query content and query position; performs a first loss calculation based on the object query to obtain a first loss value; obtains several second image representations through a second model based on the image-text pairs; performs a second loss calculation based on the first text representation and the second image representation to obtain a second loss value; performs a third loss calculation based on the query position of the object query to obtain a third loss value; obtains a comprehensive loss value based on at least one of the first loss value, the second loss value and the third loss value; and updates the parameters of the second model based on the comprehensive loss value. The present invention uses image-text pairs for knowledge distillation, which can align image representations and text representations, realize joint distillation of multiple modalities, and is more suitable for open set detection. And by updating model parameters through one or more loss values, different training requirements can be met and model performance can be effectively improved.

[0063] like Figure 1 As shown, the method specifically comprises the following steps:

[0064] Step S100: obtaining a plurality of first image representations and a plurality of first text representations based on the image-text pairs through a first model.

[0065] Specifically, in view of the problem of poor generalization ability of the closed set target detection model, this embodiment uses an open set detection model with an encoder and a decoder structure to construct the first model and the second model, wherein the encoder is used to extract image features and the decoder is used to generate prediction results for target detection. The scale difference between the first model and the second model is large. The first model is constructed using a large open set detection model, and the second model is constructed using a streamlined open set detection model. The large first model has high model accuracy, and the small second model has low computational complexity and fast inference speed.

[0066] For example, the Grounding DINO model can be used as an open set detection model. The Grounding DINO model uses the encoder-decoder structure of the Transformer and uses object embedding vectors (object query) to represent the target to be detected. These embedding vectors are used as the input of the decoder and interact with the output of the image encoder. Specifically, the first model can be a teacher model and can use the Grounding DINO model, and the second model can be a student model and can use the Grounding DINO Edge model.

[0067] In order to further improve the network performance and generalization ability of the second model, this embodiment uses the knowledge of the first model to improve its own performance and adopts distillation training to minimize the difference between the second model's own prediction and the prediction of the first model, wherein the input of the distillation training process adopts image-text pairs. During distillation training, the image in the image-text pair is input into the image encoder of the first model for image feature extraction to obtain several image representations (image embedding); and the text in the image-text pair is input into the text encoder of the first model for text feature extraction to obtain several text representations (text embedding).

[0068] Step S200: obtaining an object query according to the plurality of first image representations, wherein the object query includes a query content and a query position.

[0069] Specifically, the first model may select multiple image representations from a plurality of generated image representations as object queries through a detection head module, wherein the object query includes query content and query location.

[0070] In one implementation, obtaining an object query according to the plurality of first image representations, the object query including query content and query location, includes:

[0071] selecting a target image representation from the plurality of first image representations;

[0072] A query content of an object query is determined according to the target image representation, and a query position of the object query is determined according to a position of the target image representation in the plurality of first image representations.

[0073] Specifically, the image representation reflects the content, and the target image representation is an image representation selected from several image representations, wherein the target image representation is used as the query content, and the position of the target image representation in the selected position of several first image representations is used as the query position, so the object query includes the query content and the query position.

[0074] Step S300: Perform a first loss calculation according to the object query to obtain a first loss value.

[0075] Specifically, the detection head of the first model is used to select K image representations from several image representations as object queries. In order to enable the second model to learn the prior knowledge of the first model, this embodiment distributes the object queries to the first model and the second model, and performs a first loss calculation based on the prediction results output by the first model and the second model based on the object queries, respectively, to obtain a first loss value.

[0076] In one implementation, performing a first loss calculation according to the object query to obtain a first loss value includes:

[0077] De-noising the object query to obtain a de-noised object query;

[0078] A first loss calculation is performed according to the object query and the denoised object query to obtain a first loss value.

[0079] Specifically, the process of generating the first loss value can be defined as denoising distill training. In this embodiment, a new object query is constructed by combining the object query generated above with a preset denoising training method, that is, a denoising object query is obtained. Finally, the first loss is calculated by the object query and the denoising object query to obtain the first loss value. The denoising object query is a newly added supervisory signal. By adding an additional supervisory signal, the training convergence speed can be accelerated.

[0080] In one implementation, the number of constructed denoised object queries is equal to the number of object queries. For example, if k object queries are obtained, K denoised object queries are constructed.

[0081] In one implementation, denoising the object query to obtain a denoised object query includes:

[0082] Randomly extracting a content portion of the object query to obtain a query content of the denoised object query;

[0083] The target box of the image-text pair is sampled to obtain a query position of the denoised object query.

[0084] Specifically, taking the construction process of a denoised object query as an example, it is necessary to construct its query content and query position. For the construction of the query content, a query content of the object query of the first model is randomly extracted as the query content of the denoised object query. Since it is a random extraction, the query content of different denoised object queries can be repeated. For the construction of the query position, it can be obtained by sampling the image-text pairs. The sampling method can be, for example, random offset sampling. Given the data annotation of the target box in the image, the data annotation includes the real target box coordinate information and category, and random offset sampling is performed around the target box to obtain the query position of the denoised object query. Since some new detection boxes are constructed by random offset sampling around the target box, the new detection boxes can be used as additional supervision signals to speed up the training convergence.

[0085] In one implementation, sampling the target frame of the image-text pair to obtain the query position of the denoising object query includes:

[0086] The coordinate position of the target frame in a given image includes the horizontal coordinate of the frame center, the vertical coordinate of the frame center, the frame length, and the frame width;

[0087] For each component element, obtain the basic value of the component element according to the target frame;

[0088] Randomly sampling from the deviation range corresponding to the component element to obtain a deviation value, and calculating the value of the component element according to the base value and the deviation value;

[0089] According to the values ​​of all the component elements, the query position of the denoising object query is obtained.

[0090] Specifically, the query position of the denoised object query is obtained by random offset sampling around the target box (GT box, also known as the ground truth box). The query position consists of four components, namely the horizontal coordinate of the box center, the vertical coordinate of the box center, the length of the box, and the width of the box. The basic values ​​of the four components are determined according to the horizontal coordinate of the box center, the vertical coordinate of the box center, the length of the box, and the width of the box. Random offset sampling refers to randomly offsetting the basic values ​​of the four components to obtain new values. Among them, it is necessary to predetermine the deviation ranges corresponding to the four components, and each component needs to be randomly perturbed within its own corresponding deviation range.

[0091] For example, the center position and length and width of the target box are [x, y, w, h] respectively, and the query position of the denoised object query is constructed as [x+Δx, y+Δy, w1, h1], where the horizontal coordinate offset Δx of the center position is randomly sampled in the range [-γw / 2, γw / 2], and the vertical coordinate offset Δy of the center position is randomly sampled in the range [-γh / 2, γ / 2]. The length w1 of the denoised object query is randomly sampled in [(1-λ2)w, (1+λ2)w], and the width h1 of the denoised object query is randomly sampled in [(1-λ2)h, (1+λ2)h]. γ and λ are hyperparameters.

[0092] In one implementation, performing a first loss calculation according to the object query and the denoised object query to obtain a first loss value includes:

[0093] Passing the object query and the denoised object query through a decoder of the first model and a decoder of the second model, respectively obtaining a first prediction result and a second prediction result, wherein the prediction result includes a prediction box and a prediction category;

[0094] A first loss calculation is performed according to the first prediction result and the second prediction result to obtain a first loss value.

[0095] Specifically, all object queries and all denoised objects are used as input data and input into the decoder of the first model and the decoder of the second model respectively. The first model and the second model output prediction results through the detection head module based on the queries decoded by themselves. The prediction results include the prediction boxes (boxes) and the prediction categories (cls) of the detection targets. In this embodiment, the prediction results output by the first model are defined as the first prediction results, and the prediction results output by the second model are defined as the second prediction results. A first loss calculation is performed based on the first prediction results and the second prediction results to obtain a first loss value that can reflect the gap between the two prediction results. When the first loss value is used to update the parameters of the second model, the prediction of the second model can be supervised by the denoising distillation loss to learn the prediction of the first model.

[0096] In one implementation, the first loss value is calculated using two types of losses, one is a prediction box loss, which is used to quantify the gap between the prediction box of the first model and the prediction box of the second model. The prediction box loss can be calculated using L1 loss and / or GIOU loss to supervise the prediction box output by the second model. The other is a prediction category loss, which is used to quantify the gap between the prediction category of the first model and the prediction category of the second model. The prediction category loss can be calculated using cross entropy loss to supervise the prediction category output by the second model.

[0097] Step S400: Obtain several second image representations based on the image-text pairs through a second model.

[0098] Specifically, the image in the image-text pair is input into the encoder of the second model for feature extraction. In this embodiment, the image features extracted by the second model are defined as the second image representation.

[0099] Step S500: perform a second loss calculation based on the first text representation and the second image representation to obtain a second loss value.

[0100] Specifically, a second loss is calculated based on the first prediction result and the second prediction result to obtain a second loss value that can reflect the gap between the two prediction results. When the second loss value is used to update the parameters of the second model, the prediction of the second model based on the image representation can be supervised by the text representation provided by the first model, so that the second model can learn the text representation provided by the first model. In this embodiment, the distillation training of the second model using the first text representation and the second image representation can be called text representation distillation training (text distillation) or multimodal fusion distillation training.

[0101] In one implementation, performing a second loss calculation according to the first text representation and the second image representation to obtain a second loss value includes:

[0102] fusing the first text representation and the second image representation to obtain a fused representation;

[0103] A second loss calculation is performed according to the fused representation to obtain a second loss value.

[0104] Specifically, firstly, a plurality of text representations provided by the first model and a plurality of image representations generated by the second model are multimodally fused to obtain a fused representation, wherein the fused representation includes: a fused text representation and a fused image representation (such as Figure 2 ). The contrast loss is calculated based on the fused representation, and the parameters of the second model are updated based on the calculated contrast loss, so as to supervise the second model to learn to align the image representation with the text representation when the detection target exists, and to separate the image representation without the detection target from the text representation. In practical applications, the second loss calculation can adopt the contrast loss function, input the fused representation into the contrast loss function to calculate the contrast loss, and use the calculated contrast loss value as the second loss value.

[0105] In one implementation, fusing the first text representation and the second image representation to obtain a fused representation includes:

[0106] According to the first text representation and the second image representation, a first cross attention from image to text and a second cross attention from text to image are calculated respectively through a cross attention mechanism;

[0107] A fused representation is obtained based on the first cross-attention and the second cross-attention.

[0108] This embodiment adopts a cross-attention mechanism to fuse the first text representation and the second image representation. The cross-attention mechanism can capture the key information of the text representation and the image representation at the same time, and realize the effective fusion of the text representation and the image representation. Specifically, the cross-attention mechanism is implemented based on the query (Q), key (K) and key value (V), and the key and key value are also called key-value pairs. Figure 3 As shown, in this embodiment, the first text representation is used as a key-value pair and the second image representation is used as a query to calculate the first cross attention from image to text. The first text representation is used as a query and the second image representation is used as a key-value pair to calculate the second cross attention from text to image. Then, the first text representation and the second image representation are fused according to the calculated first cross attention and second cross attention to obtain the fused first text representation and the fused second image representation text representation.

[0109] Step S600: Obtain a third loss value by performing a third loss calculation based on the query position of the object query.

[0110] Specifically, the object query is a query selected from multiple first image representations, and the probability of the detection target existing at its query position is relatively high. Therefore, this embodiment uses the query position of the object query as the detection position, and evaluates the prediction performance of the second model through the third loss calculation to obtain the third loss value. The method of using the query position of the object query to perform distillation training on the second model in this embodiment can be called query selection distill training.

[0111] In one implementation, obtaining a third loss value by performing a third loss calculation based on the query position of the object query includes:

[0112] Obtaining a third prediction result and a fourth prediction result through the first model and the second model respectively according to the query position of the object query;

[0113] Obtaining a fourth loss value by performing a third loss calculation based on the third prediction result and the fourth prediction result;

[0114] A fifth loss value is obtained by calculating the third loss based on the fourth prediction result and the corresponding data annotation.

[0115] The third loss value is calculated according to the fourth loss value and / or the fifth loss value.

[0116] Specifically, this embodiment selects K first image representations from a number of first image representations as query objects, and the positions of the first image representations are used as the most likely positions of the detection targets (query selection), that is, the detection positions are obtained. One way is to distribute the detection positions to the detection head modules of the first model and the second model for prediction, obtain the third prediction result output by the first model and the fourth prediction result output by the second model, and calculate and analyze the gap between the fourth prediction result and the third prediction result through the third loss to obtain the fourth loss value, so as to supervise the second model to learn the prediction of the first model about the detection position, thereby improving the model performance of the second model. Another way is to calculate and analyze the gap between the fourth prediction result and the corresponding data annotation through the third loss to obtain the fifth loss value, so that the second model learns the correct prediction result, converges with the gap between the data annotation, and thereby improves the model performance of the second model. Finally, the fourth loss value, or the fifth loss value, or the fourth loss value and the fifth loss value can be used as the third loss value for query selection distillation training, preferably the fifth loss value calculated based on the true value.

[0117] In actual application, the third loss calculation can adopt the cross entropy loss function, and the prediction result includes the prediction box and the prediction category. The calculation process of the fourth loss value is: the first error value between the prediction box of the third prediction result and the prediction box of the fourth prediction result, the second error value between the prediction category of the third prediction result and the prediction category of the fourth prediction result are calculated by the cross entropy loss function, and the fourth loss value is calculated according to the first error value and the second error value. The calculation process of the fifth loss value is: the third error value between the prediction box of the fourth prediction result and the target box of the data annotation, the fourth error value between the prediction category of the fourth prediction result and the category of the data annotation are calculated by the cross entropy loss function, and the fifth loss value is calculated according to the third error value and the fourth error value.

[0118] Step S700: Obtain a comprehensive loss value according to at least one of the first loss value, the second loss value and the third loss value.

[0119] Step S800: update the parameters of the second model based on the comprehensive loss value.

[0120] Specifically, this embodiment provides a variety of distillation training methods such as denoising distillation training method, text representation distillation training method, query selection distillation training method, etc. Users can select one or more of the distillation training methods according to task requirements. Whichever distillation training method is selected, the final comprehensive loss value of the second model is determined by using the loss value calculated based on the distillation training method, and the model parameters of the second model are updated based on the comprehensive loss value to improve the model performance of the second model.

[0121] Advantages of the present invention include:

[0122] (1) Existing distillation training methods based on target detection algorithms usually only process images and are difficult to apply to open set detection. The text representation distillation training method proposed in the present invention performs knowledge distillation on text representation and image representation respectively, which can align the image representation and text representation of training samples and realize multimodal joint distillation, so as to be better applied to open set detection.

[0123] (2) Existing distillation training methods based on target detection algorithms usually use response distillation, and the supervisory signals available for learning the second model are limited, resulting in slow convergence of distillation training and poor performance. The denoising distillation training method and query selection distillation training method proposed in the present invention take into account the structural characteristics of the detection algorithm based on the encoder and decoder, and add additional supervisory signals, which can effectively accelerate the training convergence speed and improve the model accuracy, thereby achieving better knowledge distillation.

[0124] The technical effect of the present invention proves that:

[0125] like Figure 4 , Figure 5 As shown, experiments were conducted using three training data sets, o365, fliker, and vg. According to the experimental results, it can be found that the distillation training method based on image-text pairs proposed in the present invention has significantly improved model performance compared with the open set detection algorithm that does not use distillation.

[0126] Based on the above embodiments, the present invention also provides a distillation training device based on image-text pairs, such as Figure 6 As shown, the device comprises:

[0127] A first feature extraction module 01, used to obtain a plurality of first image representations and a plurality of first text representations based on the image-text pair through a first model;

[0128] A query generation module 02, configured to obtain an object query according to the plurality of first image representations, wherein the object query includes a query content and a query position;

[0129] A first loss calculation module 03, configured to perform a first loss calculation according to the object query to obtain a first loss value;

[0130] A second feature extraction module 04, used to obtain a plurality of second image representations based on the image-text pair through a second model;

[0131] A second loss calculation module 05, configured to perform a second loss calculation according to the first text representation and the second image representation to obtain a second loss value;

[0132] A third loss calculation module 06, configured to obtain a third loss value by performing a third loss calculation based on the query position of the object query;

[0133] A comprehensive loss calculation module 07, configured to obtain a comprehensive loss value according to at least one of the first loss value, the second loss value and the third loss value;

[0134] The parameter updating module 08 is used to update the parameters of the second model based on the comprehensive loss value.

[0135] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 7 As shown. The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a distillation training method based on image-text pairs is implemented. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0136] Those skilled in the art will understand that Figure 7 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0137] In one implementation, the memory of the terminal stores one or more programs, and is configured to be executed by one or more processors, wherein the one or more programs include instructions for performing a distillation training method based on image-text pairs.

[0138] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0139] In summary, the present invention discloses a distillation training method, device, terminal and medium based on image-text pairs. The method includes: obtaining a plurality of first image representations and a plurality of first text representations through a first model based on the image-text pair; obtaining an object query according to the plurality of first image representations, the object query including query content and query position; performing a first loss calculation according to the object query to obtain a first loss value; obtaining a plurality of second image representations through a second model based on the image-text pair; performing a second loss calculation according to the first text representation and the second image representation to obtain a second loss value; performing a third loss calculation based on the query position of the object query to obtain a third loss value; obtaining a comprehensive loss value according to at least one of the first loss value, the second loss value and the third loss value; and updating the parameters of the second model based on the comprehensive loss value. The present invention adopts image-text pairs for knowledge distillation, which can align image representations and text representations, realize joint distillation of multiple modalities, and is more suitable for open set detection. And updating model parameters through one or more loss values ​​can meet different training requirements and effectively improve model performance.

[0140] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A distillation training method based on image-text pairs, characterized in that: The method comprises: Based on the image-text pairs, a first model is used to obtain a plurality of first image representations and a plurality of first text representations; Obtaining an object query according to the plurality of first image representations, wherein the object query includes a query content and a query location; Perform a first loss calculation according to the object query to obtain a first loss value; Based on the image-text pairs, a second model is used to obtain a plurality of second image representations; Perform a second loss calculation based on the first text representation and the second image representation to obtain a second loss value; Obtaining a third loss value by performing a third loss calculation based on the query position of the object query; Obtaining a comprehensive loss value according to at least one of the first loss value, the second loss value, and the third loss value; The parameters of the second model are updated based on the comprehensive loss value.

2. The image-text pair based distillation training method according to claim 1, characterized in that: Obtaining an object query according to the plurality of first image representations, wherein the object query includes query content and query location, including: selecting a target image representation from the plurality of first image representations; A query content of an object query is determined according to the target image representation, and a query position of the object query is determined according to a position of the target image representation in the plurality of first image representations.

3. The image-text pair based distillation training method according to claim 1, characterized in that: Performing a first loss calculation according to the object query to obtain a first loss value includes: De-noising the object query to obtain a de-noised object query; A first loss calculation is performed according to the object query and the denoised object query to obtain a first loss value.

4. The image-text pair based distillation training method according to claim 3, characterized in that: Denoising the object query to obtain a denoised object query, including: Randomly extracting a content portion of the object query to obtain a query content of the denoised object query; The target box of the image-text pair is sampled to obtain a query position of the denoised object query.

5. The image-text pair based distillation training method according to claim 3, characterized in that: Performing a first loss calculation according to the object query and the denoising object query to obtain a first loss value includes: Passing the object query and the denoised object query through a decoder of the first model and a decoder of the second model to obtain a first prediction result and a second prediction result, respectively, where the prediction result includes a prediction box and a prediction category; A first loss calculation is performed according to the first prediction result and the second prediction result to obtain a first loss value.

6. The image-text pair based distillation training method according to claim 1, characterized in that: Calculating a second loss according to the first text representation and the second image representation to obtain a second loss value includes: fusing the first text representation and the second image representation to obtain a fused representation; A second loss calculation is performed according to the fused representation to obtain a second loss value.

7. The image-text pair based distillation training method according to claim 6, characterized in that: Fusing the first text representation and the second image representation to obtain a fused representation, including: According to the first text representation and the second image representation, a first cross attention from image to text and a second cross attention from text to image are calculated respectively through a cross attention mechanism; A fused representation is obtained based on the first cross-attention and the second cross-attention.

8. The image-text pair based distillation training method according to claim 1, characterized in that: The third loss value is obtained by calculating the query position of the object query through a third loss, including: Obtaining a third prediction result and a fourth prediction result through the first model and the second model respectively according to the query position of the object query; Obtaining a fourth loss value by performing a third loss calculation based on the third prediction result and the fourth prediction result; A fifth loss value is obtained by calculating the third loss based on the fourth prediction result and the corresponding data annotation. The third loss value is calculated according to the fourth loss value and / or the fifth loss value.

9. A distillation training device based on image-text pairs, characterized in that: The device comprises: A first feature extraction module, configured to obtain a plurality of first image representations and a plurality of first text representations based on the image-text pair through a first model; A query generation module, configured to obtain an object query according to the plurality of first image representations, wherein the object query includes a query content and a query position; A first loss calculation module, configured to perform a first loss calculation according to the object query to obtain a first loss value; A second feature extraction module, configured to obtain a plurality of second image representations based on the image-text pairs through a second model; A second loss calculation module, configured to perform a second loss calculation based on the first text representation and the second image representation to obtain a second loss value; A third loss calculation module, configured to obtain a third loss value by performing a third loss calculation based on a query position of the object query; a comprehensive loss calculation module, configured to obtain a comprehensive loss value according to at least one of the first loss value, the second loss value and the third loss value; A parameter updating module is used to update the parameters of the second model based on the comprehensive loss value.

10. A terminal, characterized in that: The terminal includes a memory and one or more processors; the memory stores one or more programs; the program contains instructions for executing the image-text pair based distillation training method as described in any one of claims 1-8; and the processor is used to execute the program.

11. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to implement the steps of the image-text pair based distillation training method as described in any one of claims 1-8.

Images