Image-text pair based distillation training method and device, terminal and medium

Abstract

The application discloses a distillation training method and device based on an image-text pair, a terminal and a medium. The application adopts the image-text pair for knowledge distillation, can align image representation and text representation, realizes joint distillation of multiple modalities, and is more suitable for open set detection. Furthermore, model parameter updating is performed through one or more loss values, different training requirements can be met, and the model performance is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of open set object detection, and particularly relates to a distillation training method and device based on image-text pairs, a terminal and a medium. BACKGROUND

[0002] Object detection technology is an important part of computer vision, aiming to identify and locate target objects in images or videos.

[0003] Existing object detection technologies include closed set object detection and open set object detection. Traditional object detection algorithms belong to closed set object detection algorithms. In closed set object detection, the model only processes classes existing in the training set during the training and testing stages. In open set object detection, the model only encounters a part of the known class data during the training stage, but may face unknown class targets during the testing stage, which requires the model to have certain generalization ability to recognize new class targets not appearing in the training set. Distillation training is a widely used training method for improving the performance and generalization ability of deep neural networks. In distillation training, the student model can improve its performance with the help of the knowledge of the teacher model. However, existing distillation training methods based on object detection algorithms have problems such as single training mode and limited supervision signal, resulting in poor distillation training effect.

[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a distillation training method and device based on image-text pairs, a terminal and a medium to solve the problem of single training mode of existing distillation training methods based on object detection algorithms, resulting in poor distillation training effect.

[0006] The technical solutions adopted by the present application to solve the problem are as follows:

[0007] In a first aspect, the present application provides a distillation training method based on image-text pairs, which comprises:

[0008] Based on the image-text pairs, a first model is used to obtain a plurality of first image representations and a plurality of first text representations;

[0009] An object query is obtained according to the plurality of first image representations, and the object query contains query content and query position;

[0010] A first loss value is obtained by performing first loss calculation according to the object query;

[0011] obtaining, by a second model, a plurality of second image representations based on the image-text pair;

[0012] performing a second loss calculation based on the first text representation and the second image representation to obtain a second loss value;

[0013] performing a third loss calculation based on a query position of the object query to obtain a third loss value;

[0014] obtaining a comprehensive loss value based on at least one of the first loss value, the second loss value and the third loss value;

[0015] performing a parameter update on the second model based on the comprehensive loss value.

[0016] In an implementation, the object query is obtained based on the plurality of first image representations, and the object query includes query content and a query position, including:

[0017] selecting a target image representation from the plurality of first image representations;

[0018] determining the query content of the object query based on the target image representation, and determining the query position of the object query based on a position of the target image representation in the plurality of first image representations.

[0019] In an implementation, the first loss calculation is performed based on the object query to obtain the first loss value, including:

[0020] performing denoising on the object query to obtain a denoised object query;

[0021] performing the first loss calculation based on the object query and the denoised object query to obtain the first loss value.

[0022] In an implementation, the denoised object query is obtained by performing denoising on the object query, including:

[0023] randomly extracting a content part of the object query to obtain query content of the denoised object query;

[0024] sampling a target box of the image-text pair to obtain the query position of the denoised object query.

[0025] In an implementation, the first loss calculation is performed based on the object query and the denoised object query to obtain the first loss value, including:

[0026] passing the object query and the denoised object query through a decoder of the first model and a decoder of the second model to obtain a first prediction result and a second prediction result respectively, and the prediction result includes a prediction box and a prediction category;

[0027] performing a first loss calculation according to the first prediction result and the second prediction result to obtain a first loss value.

[0028] In an implementation, the second loss calculation according to the first text representation and the second image representation to obtain a second loss value comprises:

[0029] fusing the first text representation and the second image representation to obtain a fused representation;

[0030] performing a second loss calculation according to the fused representation to obtain a second loss value.

[0031] In an implementation, the fusing the first text representation and the second image representation to obtain a fused representation comprises:

[0032] calculating, according to the first text representation and the second image representation, a first cross-attention from image to text and a second cross-attention from text to image respectively through a cross-attention mechanism;

[0033] obtaining a fused representation based on the first cross-attention and the second cross-attention.

[0034] In an implementation, the third loss calculation based on the query position of the object query to obtain a third loss value comprises:

[0035] obtaining a third prediction result and a fourth prediction result according to the query position of the object query through the first model and the second model respectively;

[0036] obtaining a fourth loss value through a third loss calculation based on the third prediction result and the fourth prediction result;

[0037] obtaining a fifth loss value through the third loss calculation based on the fourth prediction result and the corresponding data annotation.

[0038] calculating the third loss value according to the fourth loss value and / or the fifth loss value.

[0039] In a second aspect, an embodiment of the present application further provides a distillation training device based on an image-text pair, the device comprising:

[0040] a first feature extraction module configured to obtain a plurality of first image representations and a plurality of first text representations through a first model based on the image-text pair;

[0041] a query generation module configured to obtain an object query according to the plurality of first image representations, the object query comprising query content and a query position;

[0042] The first loss calculation module is configured to perform first loss calculation according to the object query, and obtain a first loss value.

[0043] The second feature extraction module is configured to obtain a plurality of second image representations based on the image-text pair by using a second model.

[0044] The second loss calculation module is configured to perform second loss calculation according to the first text representation and the second image representation, and obtain a second loss value.

[0045] The third loss calculation module is configured to perform third loss calculation based on the query position of the object query, and obtain a third loss value.

[0046] The comprehensive loss calculation module is configured to obtain a comprehensive loss value according to at least one of the first loss value, the second loss value and the third loss value.

[0047] The parameter updating module is configured to perform parameter updating on the second model based on the comprehensive loss value.

[0048] In a third aspect, an embodiment of the present application further provides a terminal, which comprises a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the image-text pair based distillation training method according to any one of the above aspects; and the processor is configured to execute the programs.

[0049] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, which stores a plurality of instructions, and the instructions are adapted to be loaded and executed by a processor to implement the steps of the image-text pair based distillation training method according to any one of the above aspects.

[0050] The present application has the following beneficial effects: the embodiment of the present application performs knowledge distillation based on an image-text pair, can align image representations and text representations, realizes joint distillation of multiple modalities, is more suitable for open set detection, and can meet different training requirements and effectively improve model performance by performing model parameter updating based on one or more loss values. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0052] Figure 1 is a flowchart of the image-text pair based distillation training method provided by the embodiment of the present application.

[0053] Figure 2 is a logical schematic diagram of the plurality of distillation training methods provided by the embodiment of the present application.

[0054] Figure 3 is a logical schematic diagram of the text image fusion provided by the embodiment of the present application.

[0055] Figure 4 is an experimental result graph based on the COCO evaluation data set provided by the embodiment of the present application.

[0056] Figure 5 is an experimental result graph based on the LVIS evaluation data set provided by the embodiment of the present application.

[0057] Figure 6 is a module schematic diagram of the distillation training device based on the image-text pair provided by the embodiment of the present application.

[0058] Figure 7 is a principle block diagram of the terminal provided by the embodiment of the present application. DETAILED DESCRIPTION

[0059] The present application discloses a distillation training method and device based on an image-text pair, a terminal and a medium. In order to make the purpose, technical scheme and effect of the present application more clear and explicit, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0060] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an" and "the" used herein also include the plural forms. It should be further understood that the phrase "comprising" used in the specification of the present application means that the features, integers, steps, operations, elements and / or components exist, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intermediate elements. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0061] Those skilled in the art of the technology can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such herein.

[0062] To overcome the above-mentioned defects of the prior art, the present application provides a distillation training method based on image-text pairs, which comprises the following steps:

[0063] As shown in Figure 1 the method specifically comprises the following steps:

[0064] Step S100, obtaining a plurality of first image representations and a plurality of first text representations based on the image-text pairs through a first model.

[0065] Specifically, in view of the poor generalization ability of the closed-set object detection model, the present embodiment adopts an open-set detection model with an encoder and a decoder structure to construct the first model and the second model, wherein the encoder is used to extract image features, and the decoder is used to generate the prediction result of the object detection. The first model and the second model have a large difference in size, the first model is constructed by using a large open-set detection model, and the second model is constructed by using a simplified open-set detection model. The large first model has high model precision, the small second model has low computational complexity and fast inference speed.

[0066] For example, the Grounding DINO model can be used as the open set detection model. The Grounding DINO model adopts the encoder-decoder structure of the Transformer, uses object query vectors to represent the target to be detected, and these embedding vectors are used as the input of the decoder and interact with the output of the image encoder. Specifically, the first model can be a teacher model, and the Grounding DINO model can be used. The second model can be a student model, and the Grounding DINO Edge model can be used.

[0067] To further improve the network performance and generalization ability of the second model, the embodiment uses distillation training to minimize the difference between the prediction of the second model and the prediction of the first model by improving the performance of the second model with the knowledge of the first model, wherein the input of the distillation training process uses the image-text pair. During the distillation training, the image in the image-text pair is input into the image encoder of the first model to extract image features and obtain a plurality of image embeddings; and the text in the image-text pair is input into the text encoder of the first model to extract text features and obtain a plurality of text embeddings.

[0068] Step S200, obtaining an object query according to the plurality of first image representations, the object query including query content and query position.

[0069] Specifically, the first model can select a plurality of image representations as object queries from the generated plurality of image representations through a detection head module, wherein the object query includes query content and query position.

[0070] In an implementation manner, obtaining an object query according to the plurality of first image representations, the object query including query content and query position, includes:

[0071] selecting a target image representation from the plurality of first image representations;

[0072] determining the query content of the object query according to the target image representation, and determining the query position of the object query according to the position of the target image representation in the plurality of first image representations.

[0073] Specifically, the image representation reflects content, and a target image representation is selected from the plurality of image representations, where the target image representation is taken as query content, and a selected position of the target image representation in the plurality of first image representations is taken as a query position, so that the object query includes the query content and the query position.

[0074] In step S300, first loss calculation is performed according to the object query, to obtain a first loss value.

[0075] Specifically, the K image representations are selected from the plurality of image representations by using a detection head of the first model as the object query, in order to enable the second model to learn prior knowledge of the first model, the object query is distributed to the first model and the second model, and first loss calculation is performed according to prediction results output by the first model and the second model based on the object query, to obtain the first loss value.

[0076] In an implementation manner, the first loss calculation is performed according to the object query, to obtain the first loss value, including:

[0077] The object query is denoised to obtain a denoised object query.

[0078] The first loss calculation is performed according to the object query and the denoised object query, to obtain the first loss value.

[0079] Specifically, the generation process of the first loss value can be defined as denoising distillation, a new object query is constructed by using a preset denoising training method in combination with the foregoing generated object query, that is, a denoised object query is obtained. Finally, the first loss calculation is performed according to the object query and the denoised object query, to obtain the first loss value. The denoised object query is a new supervision signal, and the training convergence speed can be accelerated by adding the additional supervision signal.

[0080] In an implementation manner, the number of the constructed denoised object queries is equal to the number of the object queries, for example, k object queries are obtained, and K denoised object queries are constructed.

[0081] In an implementation manner, the object query is denoised to obtain the denoised object query, including:

[0082] A content part of the object query is randomly extracted to obtain query content of the denoised object query.

[0083] The target frame of the image-text pair is sampled to obtain a query position of the denoised object query.

[0084] Specifically, taking the construction process of a denoising object query as an example, the query content and the query position of the denoising object query need to be constructed. For the construction of the query content, one of the query contents of the object queries of the first model is randomly extracted as the query content of the denoising object query. Since it is randomly extracted, the query contents of different denoising object queries can be repeated. For the construction of the query position, the image-text pair can be sampled to obtain the query position of the denoising object query. The sampling method can use, for example, random offset sampling. Given the data annotation of the target box in the image, the data annotation includes the real target box coordinate information and the category. Random offset sampling is performed around the target box to obtain the query position of the denoising object query. Since random offset sampling around the target box constructs some new detection boxes, the new detection boxes can be used as additional supervision signals to accelerate the training convergence speed.

[0085] In an implementation manner, the target box of the image-text pair is sampled to obtain the query position of the denoising object query, including:

[0086] The coordinate position of the target box in the given image includes the box center horizontal coordinate, the box center vertical coordinate, the box length, and the box width.

[0087] For each component element, the base value of the component element is obtained according to the target box;

[0088] A deviation value is randomly sampled from the deviation range corresponding to the component element, and the numerical value of the component element is calculated according to the base value and the deviation value.

[0089] According to the numerical values of all the component elements, the query position of the denoising object query is obtained.

[0090] Specifically, the query position of the denoising object query is obtained by random offset sampling around the target box (GT boxes, also referred to as ground truth boxes). The component elements of the query position include four kinds, namely the box center horizontal coordinate, the box center vertical coordinate, the box length, and the box width. The base values of the four component elements are determined according to the box center horizontal coordinate, the box center vertical coordinate, the box length, and the box width of the target box. Random offset sampling refers to random offset disturbance of the base values of the four component elements to obtain new numerical values. The deviation range corresponding to each of the four component elements needs to be determined in advance, and each component element needs to be randomly deviated and disturbed in its own corresponding deviation range.

[0091] For example, the center position of the target box and the length and width are [x, y, w, h], and the query position of the denoising object query is constructed as [x+Δx, y+Δy, w1, h1], wherein the center position horizontal coordinate offset Δx is randomly sampled in the range [-γw / 2, γw / 2], and the center position vertical coordinate offset Δy is randomly sampled in the range [-γh / 2, γ / 2]. The length w1 of the denoising object query is randomly sampled in the range [(1-λ2)w, (1+λ2)w], and the width h1 of the denoising object query is randomly sampled in the range [(1-λ2)h, (1+λ2)h]. Wherein γ and λ are hyperparameters.

[0092] In an implementation manner, the first loss value is obtained by performing first loss calculation according to the object query and the denoising object query.

[0093] The object query and the denoising object query are input into the decoder of the first model and the decoder of the second model respectively to obtain first prediction results and second prediction results respectively, and the prediction results include prediction boxes and prediction categories.

[0094] The first loss value is obtained by performing first loss calculation according to the first prediction results and the second prediction results.

[0095] Specifically, all object queries and all denoising objects are input as input data into the decoder of the first model and the decoder of the second model respectively, and the first model and the second model output prediction results through the detection head module based on the decoded queries, and the prediction results include prediction boxes (boxes) and prediction categories (cls) of the detection target. The prediction results output by the first model are defined as the first prediction results, and the prediction results output by the second model are defined as the second prediction results. The first loss value reflecting the gap between the two prediction results is obtained by performing first loss calculation according to the first prediction results and the second prediction results. When the first loss value is used to update the parameters of the second model, the prediction of the second model can be supervised by the denoising distillation loss to learn the prediction of the first model.

[0096] In an implementation manner, the first loss value is obtained by two loss calculations, one is a prediction box loss for quantifying the gap between the prediction box of the first model and the prediction box of the second model, and the prediction box loss can be calculated by using L1 loss and / or GIOU loss to supervise the prediction box output by the second model. The other is a prediction category loss for quantifying the gap between the prediction category of the first model and the prediction category of the second model, and the prediction category loss can be calculated by using cross-entropy loss to supervise the prediction category output by the second model.

[0097] In step S400, a plurality of second image representations are obtained by the second model based on the image-text pair.

[0098] Specifically, the image in the image-text pair is input into an encoder of the second model for feature extraction, and the image feature extracted by the second model is defined as a second image representation in the embodiment.

[0099] In step S500, a second loss calculation is performed according to the first text representation and the second image representation, to obtain a second loss value.

[0100] Specifically, the second loss calculation is performed according to the first prediction result and the second prediction result, to obtain a second loss value that can reflect the gap between the two prediction results. When the second loss value is used to update the parameters of the second model, the prediction of the second model based on the image representation can be supervised by the text representation provided by the first model, so that the second model can learn the text representation provided by the first model. The distillation training of the second model by using the first text representation and the second image representation can be referred to as text distillation or multi-modal fusion distillation training.

[0101] In an implementation manner, the second loss calculation according to the first text representation and the second image representation to obtain the second loss value comprises:

[0102] The first text representation and the second image representation are fused to obtain a fusion representation;

[0103] The second loss calculation is performed according to the fusion representation to obtain the second loss value.

[0104] Specifically, first, the plurality of text representations provided by the first model and the plurality of image representations generated by the second model are fused in a multi-modal manner to obtain a fusion representation, wherein the fusion representation comprises a fused text representation and a fused image representation (as shown in Figure 2 The contrast loss is calculated according to the fusion representation, and the parameters of the second model are updated according to the calculated contrast loss, to supervise the second model to learn to align the image representation and the text representation in which the detection target exists, and to move away the image representation and the text representation in which the detection target does not exist. In actual application, the second loss calculation can use a contrast loss function, input the fusion representation into the contrast loss function to calculate the contrast loss, and use the calculated contrast loss value as the second loss value.

[0105] In an implementation manner, the fusion of the first text representation and the second image representation to obtain the fusion representation comprises:

[0106] According to the first text representation and the second image representation, a first cross-attention from image to text and a second cross-attention from text to image are calculated through a cross-attention mechanism respectively;

[0107] obtain a fusion representation based on the first cross-attention and the second cross-attention.

[0108] The embodiment adopts a cross-attention mechanism to fuse the first text representation and the second image representation. The cross-attention mechanism can capture key information of the text representation and the image representation at the same time, and effectively fuse the text representation and the image representation. Specifically, the cross-attention mechanism is implemented based on a query (Q), a key (K), and a value (V). The key and the value are also referred to as a key-value pair. As shown in the following formula, the embodiment takes the first text representation as the key-value pair and the second image representation as the query to calculate the first cross-attention from the image to the text. The first text representation is taken as the query and the second image representation is taken as the key-value pair to calculate the second cross-attention from the text to the image. Then, the fusion of the first text representation and the second image representation is realized according to the calculated first cross-attention and the second cross-attention to obtain the fused first text representation and the fused second image representation. Figure 3

[0109] In step S600, a third loss value is obtained by third loss calculation based on the query position of the object query.

[0110] Specifically, the object query is a query selected from the plurality of first image representations, and the probability of the query position of the object query being the detection target is relatively high. Therefore, the embodiment takes the query position of the object query as the detection position, and evaluates the prediction performance of the second model by third loss calculation to obtain the third loss value. The way in which the object query is used to perform distillation training on the second model can be referred to as query selection distillation (query selection distill).

[0111] In an implementation manner, the third loss value is obtained by third loss calculation based on the query position of the object query, and the third loss calculation comprises:

[0112] The third prediction result and the fourth prediction result are obtained by the first model and the second model respectively based on the query position of the object query.

[0113] The fourth loss value is obtained by third loss calculation based on the third prediction result and the fourth prediction result.

[0114] The fifth loss value is obtained by the third loss calculation based on the fourth prediction result and the corresponding data annotation.

[0115] The third loss value is calculated according to the fourth loss value and / or the fifth loss value.

[0116] ​Specifically, the embodiment selects K first image representations selected from a plurality of first image representations as a query object, and the positions of the plurality of first image representations are regarded as the positions where the detection target is most likely to exist (query selection), that is, the detection positions are obtained. One way is to distribute the detection positions to the detection head modules of the first model and the second model for prediction, to obtain a third prediction result output by the first model and a fourth prediction result output by the second model, and to analyze the gap between the fourth prediction result and the third prediction result through third loss calculation to obtain a fourth loss value, so as to supervise the second model to learn the prediction of the first model about the detection position, and further improve the model performance of the second model. Another way is to analyze the gap between the fourth prediction result and the corresponding data label through third loss calculation to obtain a fifth loss value, so that the second model learns the correct prediction result and converges the gap with the data label, and further improves the model performance of the second model. Finally, the fourth loss value, or the fifth loss value, or the combination of the fourth loss value and the fifth loss value can be used as the third loss value of the query selection distillation training, and the fifth loss value based on the true value is preferably used.

[0117] In actual application, the third loss calculation can use a cross-entropy loss function, and the prediction result includes a prediction box and a prediction category. The calculation process of the fourth loss value is as follows: the first error value between the prediction box of the third prediction result and the prediction box of the fourth prediction result, and the second error value between the prediction category of the third prediction result and the prediction category of the fourth prediction result are calculated through the cross-entropy loss function, and the fourth loss value is calculated according to the first error value and the second error value. The calculation process of the fifth loss value is as follows: the third error value between the prediction box of the fourth prediction result and the target box of the data label, and the fourth error value between the prediction category of the fourth prediction result and the category of the data label are calculated through the cross-entropy loss function, and the fifth loss value is calculated according to the third error value and the fourth error value.

[0118] Step S700: obtaining a comprehensive loss value according to at least one of the first loss value, the second loss value and the third loss value.

[0119] Step S800: performing parameter updating on the second model based on the comprehensive loss value.

[0120] Specifically, the embodiment provides a plurality of distillation training methods such as the denoising distillation training method, the text representation distillation training method and the query selection distillation training method, and a user can select one or more distillation training methods according to task requirements. If a distillation training method is selected, a loss value calculated based on the distillation training method is used to determine the final comprehensive loss value of the second model, and the model parameters of the second model are updated based on the comprehensive loss value to improve the model performance of the second model.

[0121] Advantages of the present application include:

[0122] (1) The existing distillation training method based on target detection algorithm usually only processes images, which is difficult to apply to open set detection. The text representation distillation training method proposed by the present application respectively performs knowledge distillation on text representation and image representation, can align the image representation and text representation of the training sample, realizes multi-modal joint distillation, and is better applied to open set detection.

[0123] (2) The existing distillation training method based on target detection algorithm usually adopts response distillation, the supervision signal available for the second model learning is limited, which leads to slow distillation training convergence and poor performance. The denoising distillation training method and query selection distillation training method proposed by the present application consider the structural characteristics of the detection algorithm based on the encoder and decoder, and increase the additional supervision signal, which can effectively speed up the training convergence speed and improve the model precision, and realize better knowledge distillation.

[0124] The technical effect of the present application proves:

[0125] As shown in Figure 4 , Figure 5 , o365, fliker, vg three training data sets are used for experiments, and according to the experimental result graph, it can be detected that the distillation training method based on image-text pair proposed by the present application compared with the open set detection algorithm without using distillation, the model performance has been obviously improved.

[0126] Based on the above embodiment, the present application further provides a distillation training device based on image-text pair, as shown in Figure 6 , the device comprises:

[0127] A first feature extraction module 01 is used to obtain a plurality of first image representations and a plurality of first text representations by a first model based on the image-text pair;

[0128] A query generation module 02 is used to obtain an object query according to the plurality of first image representations, the object query containing query content and query position;

[0129] A first loss calculation module 03 is used to calculate a first loss value according to the object query;

[0130] A second feature extraction module 04 is used to obtain a plurality of second image representations by a second model based on the image-text pair;

[0131] A second loss calculation module 05 is used to calculate a second loss value according to the first text representation and the second image representation;

[0132] The third loss calculation module 06 is configured to calculate a third loss value based on the query position of the object query through a third loss calculation.

[0133] The comprehensive loss calculation module 07 is configured to obtain a comprehensive loss value according to at least one of the first loss value, the second loss value and the third loss value.

[0134] The parameter updating module 08 is configured to perform parameter updating on the second model based on the comprehensive loss value.

[0135] Based on the above-mentioned embodiments, the present application further provides a terminal, a principle block diagram of which can be shown as follows. Figure 7 The terminal includes a processor, a memory, a network interface and a display screen connected through a system bus. The processor of the terminal is configured to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the terminal is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement the distillation training method based on image-text pairs. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0136] Those skilled in the art can understand that, Figure 7 The principle block diagram shown in the above-mentioned embodiments is only a block diagram of part of the structure related to the present application scheme, and does not constitute a limitation on the terminal to which the present application scheme is applied. The specific terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0137] In an implementation manner, the memory of the terminal stores more than one program, and is configured to execute the more than one program by more than one processor, which includes instructions for performing the distillation training method based on image-text pairs.

[0138] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of each method can be included. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0139] In summary, the present application discloses a distillation training method and device based on image-text pairs, a terminal and a medium. The method comprises: obtaining a plurality of first image representations and a plurality of first text representations through a first model based on the image-text pairs; obtaining an object query according to the plurality of first image representations, the object query containing query content and query position; performing first loss calculation according to the object query to obtain a first loss value; obtaining a plurality of second image representations through a second model based on the image-text pairs; performing second loss calculation according to the first text representation and the second image representation to obtain a second loss value; performing third loss calculation based on the query position of the object query to obtain a third loss value; obtaining a comprehensive loss value according to at least one of the first loss value, the second loss value and the third loss value; and performing parameter updating on the second model based on the comprehensive loss value. The present application uses image-text pairs for knowledge distillation, can align image representations and text representations, realizes joint distillation of multiple modalities, and is more suitable for open set detection. And the model parameter updating can be performed through one or more loss values, which can meet different training requirements and effectively improve the model performance.

[0140] It should be understood that the application of the present application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all these improvements and changes shall belong to the protection scope of the claims of the present application.

Claims

1. An image-text pair-based distillation training method, characterized in that, The method comprises: obtaining a plurality of first image representations and a plurality of first text representations based on the image-text pair through a first model; obtaining an object query according to the plurality of first image representations, the object query comprising query content and query position; performing first loss calculation according to the object query to obtain a first loss value, comprising: performing denoising on the object query to obtain a denoised object query; passing the object query and the denoised object query through a decoder of the first model and a decoder of a second model to obtain a first prediction result and a second prediction result respectively, the prediction result comprising a prediction box and a prediction category; performing first loss calculation according to the first prediction result and the second prediction result to obtain a first loss value; obtaining a plurality of second image representations based on the image-text pair through a second model; performing second loss calculation according to the first text representation and the second image representation to obtain a second loss value, comprising: fusing the first text representation and the second image representation to obtain a fused representation; performing second loss calculation according to the fused representation to obtain a second loss value; performing third loss calculation based on the query position of the object query to obtain a third loss value, comprising: obtaining a third prediction result and a fourth prediction result according to the query position of the object query through the first model and the second model respectively; performing third loss calculation based on the third prediction result and the fourth prediction result to obtain a fourth loss value; performing third loss calculation based on the fourth prediction result and the corresponding data annotation to obtain a fifth loss value; calculating the third loss value according to the fourth loss value and / or the fifth loss value; obtaining a comprehensive loss value according to at least one of the first loss value, the second loss value and the third loss value; performing parameter update on the second model based on the comprehensive loss value. 2.The image-text pair based distillation training method of claim 1, wherein, obtaining an object query according to the plurality of first image representations, the object query comprising query content and query position, comprising: selecting a target image representation from the plurality of first image representations; determining the query content of the object query according to the target image representation, and determining the query position of the object query according to the position of the target image representation in the plurality of first image representations. 3.The image-text pair based distillation training method of claim 1, wherein, performing denoising on the object query to obtain a denoised object query, comprising: randomly extracting the content part of the object query to obtain the query content of the denoised object query; sampling the target box of the image-text pair to obtain the query position of the denoised object query. 4.The image-text pair based distillation training method of claim 1, wherein, fusing the first text representation and the second image representation to obtain a fused representation, comprising: calculating the first cross-attention from image to text and the second cross-attention from text to image according to the first text representation and the second image representation through the cross-attention mechanism respectively; obtaining the fused representation based on the first cross-attention and the second cross-attention. 5.A distillation training apparatus based on image-text pairs, characterized by, The device comprises: a first feature extraction module for obtaining a plurality of first image representations and a plurality of first text representations based on the image-text pair through a first model; The query generation module is configured to obtain an object query based on the first image representations, the object query including query content and a query location. The first loss calculation module is configured to calculate a first loss value based on the object query, including: denoising the object query to obtain a denoised object query; inputting the object query and the denoised object query into decoders of the first model and the second model to obtain first prediction results and second prediction results, respectively, the prediction results including prediction boxes and prediction categories; and calculating a first loss value based on the first prediction results and the second prediction results. The second feature extraction module is configured to obtain second image representations based on the image-text pairs and the second model. The second loss calculation module is configured to calculate a second loss value based on the first text representation and the second image representations, including: fusing the first text representation and the second image representations to obtain a fused representation; and calculating a second loss value based on the fused representation. The third loss calculation module is configured to calculate a third loss value based on the query location of the object query, including: obtaining third prediction results and fourth prediction results based on the query location of the object query and the first model and the second model, respectively; calculating a fourth loss value based on the third prediction results and the fourth prediction results; calculating a fifth loss value based on the fourth prediction results and corresponding data annotations; and calculating the third loss value based on the fourth loss value and / or the fifth loss value. The comprehensive loss calculation module is configured to calculate a comprehensive loss value based on at least one of the first loss value, the second loss value, and the third loss value. The parameter update module is configured to update the parameters of the second model based on the comprehensive loss value.

6. A terminal, characterized by comprising: The terminal includes a memory and one or more processors. The memory stores one or more programs. The programs include instructions for executing the distillation training method based on image-text pairs according to any one of claims 1-4. The processors are configured to execute the programs.

7. A computer-readable storage medium storing a plurality of instructions thereon, characterized in that, The instructions are suitable for being loaded and executed by the processors to implement the steps of the distillation training method based on image-text pairs according to any one of claims 1-4.

Images