Target detection model training method and device and target detection method and device

Abstract

The invention provides a target detection model training method and device, and relates to the technical field of artificial intelligence, in particular to the technical fields of image processing, natural language processing, deep learning, large models and the like. According to the specific implementation scheme, samples are input into an initial target detection model and a teacher model respectively, and a prediction feature map, a target feature map output by the teacher model and text features are obtained; determining target visual information and a target detection frame of a target feature point in the target feature map based on the labeled detection frame, and determining prediction visual information and a prediction detection frame of a prediction feature point in the prediction feature map; calculating visual information distillation loss based on the predicted visual information and the target visual information; calculating multi-modal distillation loss based on the predicted visual information and the textual features; and training a target detection model based on the visual information distillation loss, the multi-modal distillation loss, the labeling detection frame and the prediction detection frame to obtain a trained target detection model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as image recognition, natural language processing, deep learning, and large models, and can be applied to scenarios such as artificial intelligence content generation, and in particular to a target detection model training method and device, a target detection method and device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Knowledge distillation is one of the core principles of model distillation. It builds a lightweight small model and uses the supervisory information of a large model with better performance to train the small model in order to achieve better performance and accuracy. During the training of the small model, the supervisory information output by the teacher model is called knowledge, and the process of the small model learning to transfer the supervisory information from the teacher model is called distillation.

[0003] Multimodal model distillation is to transfer the multimodal knowledge learned by the teacher model to the small model. For the recognition of targets in images, when learning the small model through distillation, it is necessary to input a small image with only a single target into the teacher model separately so that the teacher model can extract the visual features of the labeled target. This cannot be achieved end-to-end and has a high operating cost. Summary of the invention

[0004] The present disclosure provides a target detection model training method and device, a target detection method and device, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to a first aspect, a target detection model training method is provided, the method comprising: inputting samples into an initial target detection model and a teacher model respectively, obtaining a predicted feature map output by the target detection model, a target feature map output by the teacher model, and text features; determining target visual information and a target detection frame of a target feature point in the target feature map based on a labeled detection frame in the sample, and determining predicted visual information and a predicted detection frame of a predicted feature point in the predicted feature map; calculating a visual information distillation loss based on the predicted visual information and the target visual information; calculating a multimodal distillation loss based on the predicted visual information and the text features; training the target detection model based on the visual information distillation loss, the multimodal distillation loss, the labeled detection frame, and the predicted detection frame to obtain a trained target detection model.

[0006] According to the second aspect, a target detection method is provided, which includes: acquiring an image to be detected; inputting the image to be detected into a target detection model to obtain a detection result of the image to be detected, wherein the target detection model is a trained target detection model obtained by training using the target detection model training method described in any implementation method of the first aspect.

[0007] According to a third aspect, a target detection model training device is provided, which includes: an obtaining unit, configured to input samples into an initial target detection model and a teacher model respectively, to obtain a predicted feature map output by the target detection model, a target feature map output by the teacher model, and text features; a determining unit, configured to determine the target visual information and the target detection frame of the target feature point in the target feature map based on the labeled detection frame in the sample, and to determine the predicted visual information and the predicted detection frame of the predicted feature point in the predicted feature map; a classification calculation unit, configured to calculate the visual information distillation loss based on the predicted visual information and the target visual information; a modal calculation unit, configured to calculate the multimodal distillation loss based on the predicted visual information and the text features; a training unit, configured to train the target detection model based on the visual information distillation loss, the multimodal distillation loss, the labeled detection frame, and the predicted detection frame to obtain a trained target detection model.

[0008] According to the fourth aspect, a target detection device is provided, which includes: an acquisition unit, configured to acquire an image to be detected; an input unit, configured to input the image to be detected into a target detection model to obtain a detection result of the image to be detected, and the target detection model is a trained target detection model obtained by training using a target detection model training device as described in any implementation method of the third aspect.

[0009] According to the fifth aspect, an electronic device is provided, which includes: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation of the first aspect or the second aspect.

[0010] According to a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect or the second aspect.

[0011] According to a seventh aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the method described in any implementation of the first aspect or the second aspect.

[0012] The target detection model training method and device provided by the embodiments of the present disclosure first input samples into the initial target detection model and the teacher model respectively to obtain the predicted feature map output by the target detection model, the target feature map output by the teacher model, and the text features; secondly, based on the labeled detection frame in the sample, the target visual information and the target detection frame of the target feature point in the target feature map are determined, and the predicted visual features and the predicted detection frame of the predicted feature point in the predicted feature map are determined; thirdly, based on the predicted visual information and the target visual information, the visual information distillation loss is calculated; from this, based on the predicted visual information and the text features, the multimodal distillation loss is calculated; finally, based on the visual information distillation loss, the multimodal distillation loss, the labeled detection frame and the predicted detection frame, the target detection model is trained to obtain a trained target detection model. Therefore, by marking the detection box, determining the target visual information of the target feature points in the target feature map, and the predicted visual information in the predicted feature map, the image in the sample can be directly analyzed through the teacher model without having to mark the visual features of the target in the image in advance, thus achieving end-to-end learning of model distillation; through visual information distillation loss, multimodal distillation loss, marking the detection box and predicting the detection box, the powerful effect of the multimodal teacher model can be transferred to the initial target detection model, making the target detection model more powerful and improving the accuracy and reliability of target detection model recognition.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0015] Figure 1 is a flowchart of an embodiment of a method for training a target detection model according to the present disclosure;

[0016] Figure 2 It is a structural diagram of the initial target detection model and the teacher model of the present disclosure for knowledge distillation;

[0017] Figure 3 is a flow chart of an embodiment of a target detection method according to the present disclosure;

[0018] Figure 4 is a structural schematic diagram of an embodiment of a target detection model training device according to the present disclosure;

[0019] Figure 5 is a schematic structural diagram of an embodiment of a target detection device according to the present disclosure;

[0020] Figure 6 It is a block diagram of an electronic device used to implement the target detection model training method or target detection method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] In the existing zero-shot distillation framework, the visual feature extraction branch of the teacher model (for example, CLIP (Contrastive Language-Image Pre-Training, a pre-trained neural network model for matching images and texts)) extracts visual features for each annotated target in the input image, and then aligns them with the visual features of the small model to be trained. The text features in the input prompt words are extracted through the text branch of the teacher model, and then the similarity between the text features and the visual features of the small model is calculated to obtain the classification result.

[0023] The existing teacher model has limited capabilities. For example, it is more suitable for extracting visual features of the entire image and is only used for classification tasks, so it cannot be end-to-end. Therefore, the existing teacher model cannot extract features of candidate targets, cannot help the box regression of the small model, and cannot be end-to-end.

[0024] Based on this, the present invention proposes a target detection model training method. Figure 1 A process 100 according to an embodiment of a method for training a target detection model according to the present disclosure is shown. The method for training a target detection model includes the following steps:

[0025] Step 101, input the samples into the initial target detection model and the teacher model respectively, and obtain the prediction feature map output by the target detection model, the target feature map output by the teacher model, and the text features.

[0026] In this embodiment, the feature map is a data structure commonly used in deep learning, and it is an important concept in convolutional neural networks. The feature map is the result of a series of convolution kernels performing convolution operations on the input image. The feature map can be regarded as an abstract representation of the original image, in which each pixel represents some specific features. Among them, the target feature map is the result of the initial target detection model performing a convolution operation on the image in the sample, and the prediction feature map is the result of the teacher model performing a convolution operation on the image in the sample.

[0027] In this embodiment, the initial target detection model is a pre-built model. After the image is input into the initial target detection model, the initial target detection model can extract the feature map of the image, classify the targets in the feature map, and obtain the predicted detection box of the target in the image in the form of coordinates and the predicted category represented by a probability value.

[0028] In this embodiment, the teacher model is a pre-trained target detection model for processing multimodal information. Specifically, the image and text are input into the teacher model to obtain the detection result of the teacher model for detecting the target in the image. The detection result includes: the target detection box of the target, the target category of the target, the target category is a category obtained based on the input text, and the text is a prompt word to prompt the category requirement. The prompt word is used to prompt the teacher model to select the target in the image that belongs to the category in the prompt word.

[0029] In this embodiment, the knowledge of the teacher model is transferred to the initial target detection model by means of offline distillation, wherein offline distillation means: the teacher model is trained in advance on a known data set, and then when the initial target detection model is trained, the obtained teacher model is used for supervised training to achieve the purpose of distillation, and the training accuracy of the teacher model is higher than that of the initial target detection model. The greater the difference, the more obvious the distillation effect. Generally speaking, the parameters of the teacher model remain unchanged during the distillation training process to achieve the purpose of training the initial target detection model. The loss function of the distillation calculates the difference between the output prediction values ​​of the teacher model and the initial target detection model, and adds it to the loss of the initial target detection as the entire training loss, performs gradient update, and finally obtains a trained target detection model with higher performance and accuracy.

[0030] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of samples involved are carried out after authorization and comply with relevant laws and regulations.

[0031] Step 102, based on the annotated detection frame in the sample, determine the target visual information and the target detection frame of the target feature point in the target feature map, and determine the predicted visual information and the predicted detection frame of the predicted feature point in the predicted feature map.

[0032] In this embodiment, the annotated detection frame is a frame that is pre-annotated for a target (such as a person, an object, etc.) in the sample image, and the shape of the frame can be determined based on the shape of the target.

[0033] In this embodiment, the target feature point is a point that reflects the target feature in the image; the target visual information is information related to the target feature point in the target feature map, and the target visual information includes: target visual features and any one or more of the target categories. The target visual features are the features represented by the position of the target feature point in the target feature map; the position of the target feature point is used to reflect the position of the target in the target feature map, and the target detection box and target category corresponding to the target feature point can be determined through the position of the target feature point.

[0034] In this embodiment, the predicted feature point is a point that reflects the characteristics of the target in the image from the perspective of the target detection model; the predicted visual information is the information related to the predicted feature point in the predicted feature map, and the predicted visual information includes: any one or more of the predicted visual features and the predicted categories. The predicted visual features are the features represented by the position of the predicted feature point in the predicted feature map; the position of the predicted feature point is used to reflect the position of the target in the predicted feature map, and the predicted detection box and predicted category corresponding to the predicted feature point can be determined by the position of the predicted feature point.

[0035] In this embodiment, the predicted visual features in the predicted feature map are the features of the points in the predicted feature map corresponding to the target feature points. The target visual features serve as the targets of the predicted visual features. The smaller the difference between the predicted visual features and the target visual features, the better the effect of knowledge transfer from the teacher model to the target detection model.

[0036] In this embodiment, the above step 102 includes: projecting the labeled detection frame of the target in the sample onto the target feature map and the predicted feature map respectively to obtain a first projection area of ​​the target feature map and a second projection area of ​​the predicted feature map; in response to the first projection area and the second projection area being the same in area, taking the center point of the first projection area as the target feature point to obtain the position of the target feature point, and determining the target detection frame and the target category based on the position of the target feature point; taking the center point of the second projection area as the predicted feature point to obtain the position of the predicted feature point, and determining the predicted detection frame and the predicted category based on the position of the predicted feature point; extracting the features of the target feature point to obtain the target visual features of the target feature point; extracting the features of the predicted feature point to obtain the predicted visual features of the predicted feature point.

[0037] Step 103, based on the predicted visual information and the target visual information, the visual information distillation loss is calculated.

[0038] In this embodiment, in the distillation stage, knowledge is transferred from the teacher model to the initial target detection model by minimizing a loss function. The learning goal of this item is the distribution of category probabilities predicted by the teacher model, and the value calculated by the loss function is the distillation loss.

[0039] In this embodiment, the visual information distillation loss is a value calculated by the visual information distillation loss function set for the teacher model and the initial target detection model. Since the visual information of different feature maps may include visual features and categories, that is, the predicted visual information includes: predicted visual features and predicted categories, and the target prediction information includes: target visual features and target categories, then the visual information distillation loss function includes: visual feature distillation loss function and category distillation loss function, and the visual information distillation loss includes: visual feature distillation loss and category distillation loss. The visual feature distillation loss is a value calculated by the visual feature distillation loss function set for the teacher model and the initial target detection model, wherein the visual feature distillation loss function is the objective function for migrating the knowledge of visual features from the teacher model to the initial target detection model. By setting the visual feature distillation loss function, the target visual features can be taken as the target, and the parameters of the target detection model can be adjusted to make the predicted visual features gradually approach the target visual features. To this end, the visual feature loss calculated by the visual feature distillation loss function is to minimize the loss of the visual features of the same point on the image of the sample predicted by the teacher model and the initial target detection model.

[0040] In this embodiment, the visual feature distillation loss function can adopt the L1 norm loss function, which is also called the minimum absolute deviation. The L1 norm loss function is used to minimize the sum of the absolute differences between the predicted visual features and the target visual features at the corresponding positions of the predicted detection box and the labeled detection box.

[0041] In this embodiment, the classification distillation loss is a value calculated by the classification distillation loss function set for the teacher model and the initial target detection model, wherein the classification distillation loss function is an objective function for migrating the knowledge of target classification in the image from the teacher model to the initial target detection model. By setting the classification distillation loss function, the target category can be used as the target, and the parameters of the target detection model can be adjusted to gradually approach the target category. To this end, the classification distillation loss calculated by the classification distillation loss function minimizes the difference loss of the category of the same point on the image of the sample predicted by the teacher model and the initial target detection model.

[0042] In this embodiment, the classification distillation loss function may adopt a KL divergence loss function, which is used to measure the amount of information lost by the distribution of the predicted category compared to the distribution of the target category.

[0043] Step 104, calculate the multimodal distillation loss based on the predicted visual information and text features.

[0044] In this embodiment, predicting visual information includes: predicting visual features. In each iterative training of the target detection model, the predicted visual features and text features are input into a pre-set multimodal distillation loss function to obtain a multimodal distillation loss, and the multimodal distillation loss is a value calculated by the multimodal distillation loss function set for the teacher model and the initial target detection model, wherein the multimodal distillation loss function is an objective function for migrating the knowledge of the semantic similarity between the text and the image of the target in the image from the teacher model to the initial target detection model. By setting the multimodal distillation loss function, the true value of the text feature can be taken as the target, and the parameters of the target detection model can be adjusted so that the semantic similarity value of the predicted visual features and the text feature gradually approaches the target. For this purpose, the multimodal distillation loss calculated by the multimodal distillation loss function is to minimize the difference loss of the semantic similarity between the text and the image at the same point on the image of the sample predicted by the teacher model and the initial target detection model.

[0045] In this embodiment, the multimodal distillation loss function can use a cross entropy loss function, which can measure the difference between two different probability distributions in the same random variable, which is expressed as the difference between the true probability distribution and the predicted probability distribution in machine learning. The smaller the value of the cross entropy loss function, the better the prediction effect of the target detection model.

[0046] Step 105: Based on the visual information distillation loss, the multimodal distillation loss, the labeled detection box, and the predicted detection box, the target detection model is trained to obtain a trained target detection model.

[0047] In this embodiment, the execution subject on which the target detection model training method runs can perform the training steps from step A to step B below to complete an iterative training of the target detection model. Among them, the sample in step A is a sample selected from the sample set, and the selection method and number of the samples are not limited in the present disclosure, and the number of iterative training of the target detection model is not limited.

[0048] Step A. Calculate the total loss value of the target detection model based on the samples input to the target detection model.

[0049] In this embodiment, the total loss value is determined by visual information distillation loss, multimodal distillation loss, and predicted detection box loss, wherein the predicted detection box loss is calculated based on the labeled detection box and the predicted detection box, and the visual information distillation loss includes: visual feature distillation loss and classification distillation loss.

[0050] In this embodiment, during each iterative training of the target detection model, samples are selected from the sample set and input into the target detection model. Based on the detection results output by the target detection model and the selected samples, the total loss value of the target detection model is calculated.

[0051] In this embodiment, during the iterative training process of the target detection model, a gradient descent algorithm can be used to minimize the loss value of the target detection model, thereby iteratively optimizing the network parameters of the target detection model.

[0052] The original meaning of gradient is a vector, which means that the directional derivative of a loss function at that point reaches the maximum value along that direction, that is, the loss function changes fastest along that direction at that point and the rate of change is the largest. In deep learning, the main task of neural networks is to find the optimal network parameters (weights and biases) during learning. The optimal network parameters are also the parameters when the loss function is minimized.

[0053] Step B. Based on the total loss value of the target detection model, train the target detection model to obtain a trained target detection model.

[0054] In this embodiment, the trained target detection model is a target detection model obtained by inputting the selected samples into the target detection model for iterative training and adjusting the parameters of the target detection model.

[0055] In this embodiment, the total loss value of the target detection model can be used to detect whether the target detection model meets the training completion condition. After the target detection model meets the training completion condition, the trained target detection model is obtained.

[0056] In this embodiment, the training completion condition includes: the total loss value of the target detection model is less than a first loss value threshold. The first loss threshold can be determined based on specific training requirements, for example, the first loss threshold is 0.01.

[0057] Optionally, in this embodiment, in response to the target detection model not meeting the training completion conditions, the relevant parameters in the target detection model are adjusted so that the total loss value of the target detection model converges, and based on the adjusted target detection model, the above-mentioned training steps A-B are continued to be executed.

[0058] In this embodiment, when the target detection model does not meet the training completion conditions, adjusting the relevant parameters of the target detection model helps to help the loss value of the target detection model converge.

[0059] The target detection model training method provided by the embodiment of the present disclosure first inputs samples into the initial target detection model and the teacher model respectively to obtain the predicted feature map output by the target detection model, the target feature map output by the teacher model, and the text features; secondly, based on the labeled detection frame in the sample, the target visual information and the target detection frame of the target feature point in the target feature map are determined, and the predicted visual feature information and the predicted detection frame of the predicted feature point in the predicted feature map are determined; thirdly, based on the predicted visual information and the target visual information, the visual information distillation loss is calculated; from this, based on the predicted visual information and the text features, the multimodal distillation loss is calculated; finally, based on the visual information distillation loss, the multimodal distillation loss, the labeled detection frame and the predicted detection frame, the target detection model is trained to obtain a trained target detection model. Therefore, by marking the detection box, determining the target visual information of the target feature points in the target feature map, and the predicted visual information in the predicted feature map, the image in the sample can be directly analyzed through the teacher model without having to mark the visual features of the target in the image in advance, thus achieving end-to-end learning of model distillation; through visual information distillation loss, multimodal distillation loss, marking the detection box and predicting the detection box, the powerful effect of the multimodal teacher model can be transferred to the initial target detection model, making the target detection model more powerful and improving the accuracy and reliability of target detection model recognition.

[0060] In some optional implementations of the present disclosure, the above-mentioned target detection model training method also includes: calculating the detection box distillation loss based on the predicted detection box and the target detection box; when training the target detection model, also based on the detection box distillation loss, obtaining the trained target detection model.

[0061] In this embodiment, the positions corresponding to the predicted detection frame and the target detection frame are input into the detection frame distillation loss function to obtain the detection frame distillation loss, wherein the detection frame distillation loss function is the objective function for migrating the knowledge of the target detection frame in the image from the teacher model to the initial target detection model. By setting the detection frame distillation loss function, the target detection frame can be used as the target, and the parameters of the target detection model can be adjusted to gradually bring the predicted detection frame closer to the target detection frame. To this end, the detection frame distillation loss calculated by the detection frame distillation loss function is the loss that minimizes the difference between the detection frames of the same point on the image of the sample predicted by the teacher model and the initial target detection model.

[0062] In this embodiment, the detection box distillation loss function can use mean squared error (MSE), which is the most commonly used loss function in the field of machine learning and is used to measure the error between the output result (final predicted value) and the expected true value. It can minimize the loss of the model by reducing the difference between the predicted value and the true value during training, thereby improving the performance of the model.

[0063] like Figure 2 As shown in the figure, the teacher model can use a large model for the detection task, which can ensure that the large model and the target detection model can be truly aligned in terms of visual features, which is more conducive to the learning of distillation. Specifically, the teacher model can use the OWL model, which includes: a visual basic model, a visual abstraction module, and a pre-trained language model. The visual basic model is used to convert the input image into a visual token, and the visual abstraction module summarizes the longer and fine-grained image features into a small number of learnable tokens, thereby achieving efficient modeling of visual information. The generated visual tokens and learnable tokens are input into the language model together with the text query to generate the corresponding response.

[0064] In this embodiment, since the same sample is input into the large model and the target detection model when training the target detection model, end-to-end learning is truly realized. That is, the picture input into the teacher model is the whole picture, rather than the pre-processed small target picture. Therefore, distillation learning can be realized instantly by using the same sample input into the large model and the target detection model.

[0065] The target detection model training method provided in the present disclosure is as follows: Figure 2 As shown in Figure 2, the target matching module maps the coordinates of the annotation box to the target feature map of the teacher model ( Figure 2 (not shown) to determine the position and features of the target feature points of the teacher model, so that the visual feature distillation loss, detection box distillation loss, classification distillation loss, and multimodal distillation loss can be calculated with the true value in the sample. Among them, the visual feature distillation loss is Figure 2 S is used to represent it.

[0066] The target detection model training method provided in this embodiment utilizes the regression characteristics of the teacher model, calculates the detection box distillation loss based on the predicted detection box and the target detection box, and trains the target detection model based on the detection box distillation loss. This allows the regression of the target detection model to be assisted by the teacher model, thereby improving the generalization ability of the target detection model.

[0067] In some optional implementations of the present disclosure, the target visual information includes: target visual features and target categories, the predicted visual information includes: predicted visual features and predicted categories, and the method of determining the target visual information and target detection frame of the target feature point in the target feature map based on the labeled detection frame in the sample, and determining the predicted visual information and predicted detection frame of the predicted feature point in the predicted feature map includes: mapping the labeled detection frame in the sample to the target feature map to obtain a first mapping detection frame on the target feature map; determining the target feature point on the target feature map based on the first mapping detection frame; extracting the features of the target feature point to obtain the target visual features of the target feature point; determining the target detection frame and target category of the target based on the position of the first mapping detection frame; mapping the labeled detection frame to the predicted feature map to obtain a second mapping detection frame on the predicted feature map; determining the predicted feature point of the target on the predicted feature map based on the second mapping detection frame; extracting the features of the predicted feature point to obtain the predicted visual features of the predicted feature point; and determining the predicted detection frame and predicted category of the target based on the position of the second mapping detection frame.

[0068] In this optional implementation, the target feature point is a point on the target feature map that represents the target (an object in the sample image, such as a person, object, etc.), the target visual feature is the image feature of the target, and can be extracted by an image feature extraction unit. The position of the target feature point is the core position of the target on the target feature map, for example, the position of the target feature point is the position of the center of the target on the target feature map.

[0069] In this optional implementation, the target feature point is a point on the target feature map that represents the target (an object in the sample image, such as a person, object, etc.), the target visual feature is the image feature of the target, and can be extracted by an image feature extraction unit. The position of the target feature point is the core position of the target on the target feature map, for example, the position of the target feature point is the position of the center of the target on the target feature map.

[0070] In this optional implementation, based on the position of the first mapping detection frame, determining the target detection frame and the target category of the target includes: based on the position of the first mapping detection frame, expanding a preset range to obtain a target detection frame; based on the features in the target feature map corresponding to the target detection frame, classifying the target category through a classification module, wherein the classification module can be a fully connected layer.

[0071] In this optional implementation, based on the position of the second mapping detection frame, determining the predicted detection frame and predicted category of the target includes: based on the position of the second mapping detection frame, expanding the preset range to obtain the predicted detection frame; based on the features in the predicted feature map corresponding to the predicted detection frame, obtaining the predicted category through classification module classification.

[0072] This optional implementation provides a method for obtaining target feature points, target visual features, target detection boxes, target categories, predicted visual features, predicted detection boxes, and predicted categories, maps the labeled detection boxes to the target feature map, and maps the labeled detection boxes to the predicted feature map, providing a reliable implementation method for obtaining target visual features and predicted visual features, thereby ensuring the accuracy of the distillation loss calculation.

[0073] In some optional implementations of the present disclosure, the target visual information includes: target visual features and target categories, the predicted visual information includes: predicted visual features and predicted categories, and the target visual information and target detection frame of the target feature point in the target feature map based on the labeled detection frame in the sample, and the predicted visual information and predicted detection frame of the predicted feature point in the predicted feature map include: determining the center point of the labeled detection frame in the sample; mapping the center point of the labeled detection frame to the target feature map to obtain the target feature point on the target feature map; extracting the features of the target feature point to obtain the target visual features of the target feature point; determining the target detection frame and the target category based on the position where the center point of the labeled detection frame is mapped to the target feature map; mapping the center point of the labeled detection frame to the predicted feature map to obtain the predicted feature point of the target on the predicted feature map; extracting the features of the predicted feature point to obtain the predicted visual features of the predicted feature point; determining the predicted detection frame and the predicted category based on the position where the center point of the labeled detection frame is mapped to the predicted feature map.

[0074] In this optional implementation, the specific methods of the target detection box, target category, predicted detection box, and predicted category are the same as those in the above optional implementation, and will not be repeated here.

[0075] This optional implementation provides a method for obtaining target feature points, target visual features, target detection boxes, target categories, predicted visual features, predicted detection boxes, and predicted categories, determines the center point of the labeled detection box, maps the center point on the target feature map, obtains the target feature point, extracts the features of the target feature point on the target feature map, and obtains the target visual features; maps the center point on the predicted feature map, obtains the predicted feature point, extracts the features of the predicted feature point, and obtains the predicted visual features, which provides another reliable implementation method for obtaining the target visual features and the predicted visual features, and ensures the accuracy of the distillation loss calculation.

[0076] Optionally, the target visual information includes: target visual features and target categories, the predicted visual information includes: predicted visual features and predicted categories, and the target visual information and target detection frame of the target feature point in the target feature map are determined based on the labeled detection frame in the sample, and the predicted visual information and predicted detection frame of the predicted feature point in the predicted feature map are determined, including: when the labeled frame in the sample is a closed frame, determining the diagonal intersection of the labeled detection frame; mapping the diagonal intersection of the labeled detection frame to the target feature map to obtain the target feature point on the target feature map; extracting the features of the target feature point to obtain the target visual features of the target feature point; determining the target detection frame and the target category based on the position where the center point of the labeled detection frame is mapped to the target feature map; mapping the diagonal intersection of the labeled detection frame to the predicted feature map to obtain the predicted feature point of the target on the predicted feature map; extracting the features of the predicted feature point to obtain the predicted visual features of the predicted feature point; determining the predicted detection frame and the predicted category based on the position where the center point of the labeled detection frame is mapped to the predicted feature map.

[0077] In some optional implementations of the present disclosure, the target visual information includes: target visual features and target categories, the predicted visual information includes: predicted visual features and predicted categories, the visual information distillation loss includes: visual feature distillation loss and classification distillation loss, and the calculation of the visual information distillation loss based on the predicted visual information and the target visual information includes: calculating the visual feature distillation loss based on the predicted visual features and the target visual features; and calculating the classification distillation loss based on the predicted category, the target category and the position.

[0078] In this optional implementation, the above-mentioned calculation of the visual feature distillation loss based on the predicted visual features and the target visual features includes: calculating the visual feature distillation loss based on the predicted visual features, the target visual features and a pre-set visual feature distillation loss function.

[0079] In this optional implementation, the calculation of the classification distillation loss based on the predicted category and the target category includes: calculating the classification distillation loss based on the predicted category, the target category and a pre-set classification distillation loss function.

[0080] This optional implementation provides a method for obtaining visual feature distillation loss and classification distillation loss. The visual feature distillation loss is calculated based on the predicted visual features and the target visual features; the classification distillation loss is calculated based on the predicted categories and the target categories, providing a reliable implementation method for obtaining the visual feature distillation loss and the classification distillation loss.

[0081] In some optional implementations of the present disclosure, the predicted visual information includes: predicted visual features, and the calculation of the multimodal distillation loss based on the predicted visual information and text features includes: calculating the similarity between the text features and the predicted visual features to obtain a similarity value; and calculating the multimodal distillation loss based on the similarity value and the true value of the text in the sample.

[0082] In this optional implementation, an image encoder may be used to encode the predicted visual features into a vector a; a text encoder may be used to encode the text features into a vector b; a·b is calculated,

[0083] If a and b come from a pair of paired images and texts, the pre-set multimodal distillation loss function will make a·b close to 1 (the true value of the text in the sample); if a and b come from unpaired images and texts, the multimodal distillation loss function will make a·b close to 0.

[0084] It should be noted that when there are multiple targets in the sample, the predicted visual features can be the features of multiple targets, and the vector corresponding to the predicted visual features can be represented as a matrix. When the text feature includes multiple features, the vector corresponding to the text feature can be represented as a matrix, and the matrix-matrix product obtains the final similarity matrix. The similarity value of each target can be obtained by averaging the values ​​of each row of the similarity matrix.

[0085] The method for calculating the multimodal distillation loss provided by this optional implementation calculates the similarity between text features and predicted visual features, and calculates the multimodal distillation loss through the similarity value, thereby improving the accuracy of the multimodal distillation loss.

[0086] In some optional implementations of the present disclosure, the target detection model is trained based on the visual information distillation loss, the multimodal distillation loss, the labeled detection box and the predicted detection box to obtain the trained target detection model, including: calculating the predicted detection box loss based on the predicted detection box and the labeled detection box; training the target detection model based on the visual information distillation loss, the multimodal distillation loss and the predicted detection box loss to obtain the trained target detection model.

[0087] In this optional implementation, the positions corresponding to the predicted detection box and the labeled detection box are input into a pre-set predicted detection box loss function to obtain the predicted detection box loss, wherein the predicted detection box loss function is a function that controls the predicted detection box to approach the labeled detection box. By setting the predicted detection box loss function, the labeled detection box can be used as the target, and the parameters of the target detection model can be adjusted to make the predicted detection box gradually approach the labeled detection box.

[0088] In this embodiment, the predicted detection box loss function may adopt the smoothL1 function, which is the L1 norm loss function after smoothing.

[0089] The method for training a target detection model provided by this optional implementation calculates the predicted detection box loss based on the predicted detection box and the labeled detection box, and trains the target detection model based on the visual information distillation loss, the multimodal distillation loss, and the predicted detection box loss, thereby improving the reliability of the trained target detection model.

[0090] In some optional implementations of the present disclosure, the target detection model is trained based on the visual information distillation loss, the multimodal distillation loss and the predicted detection box loss to obtain the trained target detection model, including: weighted summing the visual information distillation loss, the multimodal distillation loss and the predicted detection box loss to obtain the total model loss; based on the total model loss, the target detection model is trained to obtain the trained target detection model.

[0091] In this embodiment, when the visual information distillation loss includes: visual feature distillation loss and classification distillation loss, the visual feature distillation loss, the classification distillation loss, the multimodal distillation loss and the predicted detection box loss can be weighted and summed to obtain the total model loss.

[0092] The method for training a target detection model provided by this optional implementation performs weighted summation of the visual information distillation loss, the multimodal distillation loss, and the predicted detection box loss to obtain the total model loss. Based on the total model loss, the target detection model is trained, thereby improving the reliability of the trained target detection model.

[0093] Optionally, the target detection model is trained based on the visual information distillation loss, the multimodal distillation loss and the predicted detection box loss to obtain the trained target detection model, including: summing the visual information distillation loss, the multimodal distillation loss and the predicted detection box loss to obtain a first total loss; based on the first total loss, the target detection model is trained to obtain the trained target detection model.

[0094] In some optional implementations of the present disclosure, the above-mentioned inputting samples into the initial target detection model and the teacher model respectively to obtain the predicted feature map output by the target detection model, the target feature map output by the teacher model, and the text features include: obtaining a sample set, the sample set includes at least one sample, and the sample includes: an image and a search text corresponding to the image; selecting a sample from the sample set; inputting the image in the sample into the initial target detection model to obtain the predicted feature map of the target detection model predicting the image in the sample; inputting the image and text in the sample into the teacher model to obtain the target feature map of the teacher model predicting the image in the sample and the text features predicting the text.

[0095] In this optional implementation, the search text is a prompt word for searching for a specific category of targets, such as Figure 2 The prompt word T in the image can be used to determine the category of the target that needs to be screened out in the image by searching the text target detection model and the teacher model. For this purpose, the predicted category and the target category can both be the categories recorded in the search text.

[0096] In this optional implementation, the initial target detection model is an image (such as Figure 2 The image I) is used as the target detection model, and the image is input into the initial target detection model, and the predicted detection box of the target in the image and the predicted category of the target in the image output by the target detection model can be obtained. It should be noted that the trained target detection model is obtained by training the initial target detection model, and the trained target detection model can change the actual value in the predicted category based on the knowledge obtained by the teacher model migration.

[0097] This optional implementation provides a method for obtaining the output of the target detection model and the teacher model, selects samples from the sample set, inputs the images in the samples into the target detection model, and inputs the samples into the teacher model, so that the teacher model can fully recognize the characteristics of the samples, and provides a reliable implementation method for the transfer of knowledge to the target detection model.

[0098] Furthermore, based on the target detection model training method provided in the above embodiment, the present disclosure also provides an embodiment of a target detection method. The target detection method of the present disclosure combines artificial intelligence fields such as image processing and deep learning.

[0099] See also Figure 3 , shows a process 300 according to an embodiment of the target detection method of the present disclosure. The target detection method provided in this embodiment includes the following steps:

[0100] Step 301: Acquire an image to be detected.

[0101] In this embodiment, the image to be detected is an image of a target to be detected, wherein the target may be a person, an object, etc. in the image. The target detection model is used to perform target detection on the image to be detected to obtain a detection result of the target in the image to be detected.

[0102] In this embodiment, the execution subject of the target detection method can obtain the image to be detected in a variety of ways. For example, the execution subject can obtain the image to be detected stored in the database server through a wired connection or a wireless connection. For another example, the execution subject can also receive the image to be detected collected in real time by a terminal or other device.

[0103] Step 302: input the image to be detected into the target detection model to obtain the detection result of the image to be detected.

[0104] In this embodiment, the execution subject may input the image to be detected obtained in step 301 into the target detection model to obtain the detection result of the image to be detected output by the target detection model.

[0105] In this embodiment, the target detection model can be adopted as described above Figure 1 The trained target detection model is obtained by training the method described in the embodiment. For the specific training process, please refer to Figure 1 The relevant description of the illustrated embodiment will not be repeated here.

[0106] In this embodiment, the detection result of the image to be detected includes: a predicted detection frame of the target in the image to be detected and a predicted category of the target, wherein the predicted detection frame of the target is used to characterize the position of the target in the image to be detected, and the predicted category of the target is used to characterize the type to which the target belongs. The ability of the current target detection model to determine the predicted category of the target can be the ability transferred from the teacher model. For this reason, when the text input to the teacher model includes multiple types, the target detection model can detect targets of the multiple categories.

[0107] The target detection method provided by the embodiment of the present disclosure obtains an image to be detected, inputs the image to be detected into a target detection model generated by a target detection model training method, and obtains a detection result of the image to be detected. Thus, the detection result is generated by using the target detection model, which improves the reliability and accuracy of the detection result.

[0108] Further references Figure 4 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a target detection model training device, which is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0109] like Figure 4As shown, the target detection model training device 400 provided in this embodiment includes: an obtaining unit 401, a determining unit 402, a classification calculation unit 403, a modal calculation unit 404, and a training unit 405. Among them, the obtaining unit 401 can be configured to input the sample into the initial target detection model and the teacher model respectively, and obtain the predicted feature map output by the target detection model, the target feature map output by the teacher model, and the text feature. The determining unit 402 can be configured to determine the target visual information and the target detection frame of the target feature point in the target feature map based on the labeled detection frame in the sample, and determine the predicted visual information and the predicted detection frame of the predicted feature point in the predicted feature map. The classification calculation unit 403 can be configured to calculate the visual information distillation loss based on the predicted visual information and the target visual information. The modal calculation unit 404 can be configured to calculate the multimodal distillation loss based on the predicted visual information and the text feature. The training unit 405 can be configured to train the target detection model based on the visual information distillation loss, the multimodal distillation loss, the labeled detection frame and the predicted detection frame to obtain a trained target detection model.

[0110] In this embodiment, in the target detection model training device 400, the specific processing of the obtaining unit 401, the determining unit 402, the classification calculation unit 403, the modal calculation unit 404, and the training unit 405 and the technical effects thereof can be referred to respectively. Figure 1 The relevant descriptions of step 101, step 102, step 103, step 104, and step 105 in the corresponding embodiments are not repeated here.

[0111] In some optional implementations of this embodiment, the above-mentioned device 400 also includes: a detection frame calculation unit (not shown in the figure), which can be configured to calculate the detection frame distillation loss based on the predicted detection frame and the target detection frame; when training the target detection model, the trained target detection model is obtained based on the detection frame distillation loss.

[0112] In some optional implementations of the present embodiment, the target visual information includes: target visual features and target categories, the predicted visual information includes: predicted visual features and predicted categories, and the determination unit 402 is further configured to: map the annotated detection frame in the sample to the target feature map to obtain a first mapped detection frame on the target feature map; extract features of the target feature points to obtain target visual features of the target feature points; determine the target detection frame and target category of the target based on the position of the first mapped detection frame as the position of the target feature point; map the annotated detection frame to the predicted feature map to obtain a second mapped detection frame on the predicted feature map; determine the predicted feature points of the target on the predicted feature map based on the second mapped detection frame; extract features of the predicted feature points to obtain predicted visual features of the predicted feature points; and determine the predicted detection frame and predicted category of the target based on the position of the second mapped detection frame.

[0113] In some optional implementations of the present embodiment, the target visual information includes: target visual features and target categories, the predicted visual information includes: predicted visual features and predicted categories, and the determination unit 402 is further configured to: determine the center point of the annotated detection box in the sample; map the center point of the annotated detection box to the target feature map to obtain the target feature point on the target feature map; extract the features of the target feature point to obtain the target visual features of the target feature point; determine the target detection box and the target category based on the position where the center point of the annotated detection box is mapped to the target feature map; map the center point of the annotated detection box to the predicted feature map to obtain the predicted feature point of the target on the predicted feature map; extract the features of the predicted feature point to obtain the predicted visual features of the predicted feature point; determine the predicted detection box and the predicted category based on the position where the center point of the annotated detection box is mapped to the predicted feature map.

[0114] In some optional implementations of the present embodiment, the target visual information includes: target visual features and target categories, the predicted visual information includes: predicted visual features and predicted categories, the visual information distillation loss includes: visual feature distillation loss and classification distillation loss, and the classification calculation unit 403 is configured to: calculate the visual feature distillation loss based on the predicted visual features and the target visual features; and calculate the classification distillation loss based on the predicted categories and the target categories.

[0115] In some optional implementations of this embodiment, the above-mentioned predicted visual information includes: predicted visual features, and the above-mentioned modal calculation unit 404 is configured to: calculate the similarity between the text features and the predicted visual features to obtain a similarity value; and calculate the multimodal distillation loss based on the similarity value and the true value of the text in the sample.

[0116] In some optional implementations of this embodiment, the training unit 405 is configured to: calculate the predicted detection box loss based on the predicted detection box and the labeled detection box; train the target detection model based on the visual information distillation loss, the multimodal distillation loss and the predicted detection box loss to obtain a trained target detection model.

[0117] In some optional implementations of this embodiment, the training unit 405 is further configured to: weighted sum the visual information distillation loss, the multimodal distillation loss, and the predicted detection box loss to obtain the total model loss; based on the total model loss, train the target detection model to obtain a trained target detection model.

[0118] In some optional implementations of the present embodiment, the obtaining unit 401 is configured to: obtain a sample set, the sample set includes at least one sample, and the sample includes: an image and a search text corresponding to the image; select a sample from the sample set; input the image in the sample into the initial target detection model to obtain a prediction feature map of the target detection model predicting the image in the sample; input the image and text in the sample into the teacher model to obtain a target feature map of the teacher model predicting the image in the sample and a text feature of the teacher model predicting the text.

[0119] The target detection model training device provided by the embodiment of the present disclosure, first, the obtaining unit 401 inputs the samples into the initial target detection model and the teacher model respectively, and obtains the predicted feature map output by the target detection model, the target feature map output by the teacher model, and the text features; secondly, the determining unit 402 determines the target visual information and the target detection frame of the target feature point in the target feature map based on the labeled detection frame in the sample, and determines the predicted visual information and the predicted detection frame of the predicted feature point in the predicted feature map; thirdly, the classification calculation unit 403 calculates the visual information distillation loss based on the predicted visual information and the target visual information; from this, the modal calculation unit 404 calculates the multimodal distillation loss based on the predicted visual information and the text features; finally, the training unit 405 trains the target detection model based on the visual information distillation loss, the multimodal distillation loss, the labeled detection frame and the predicted detection frame to obtain the trained target detection model. Therefore, by marking the detection box, determining the target visual information of the target feature points in the target feature map, and the predicted visual information in the predicted feature map, the image in the sample can be directly analyzed through the teacher model without having to mark the visual features of the target in the image in advance, thereby achieving end-to-end learning of model distillation; through visual information distillation loss and multimodal distillation loss, marking the detection box and predicting the detection box, the powerful effect of the multimodal teacher model can be transferred to the initial target detection model, making the target detection model more powerful and improving the accuracy and reliability of target detection model recognition.

[0120] Further references Figure 5 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a target detection device, which is similar to Figure 3 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0121] like Figure 5 As shown, the target detection device 500 provided in this embodiment includes: an acquisition unit 501 and an input unit 502. The acquisition unit can be configured to acquire an image to be detected; the input unit 502 can be configured to input the image to be detected into a target detection model to obtain a detection result of the image to be detected, and the target detection model is a trained target detection model obtained by the target detection model training device of the above embodiment.

[0122] In the target detection device 500, the specific processing of the acquisition unit 501 and the input unit 502 and the technical effects thereof can be referred to in Figure 3 The relevant descriptions of step 301 and step 302 in the corresponding embodiment are not repeated here.

[0123] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0124] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0125] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0126] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0127] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0128] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a target detection model training method or a target detection method. For example, in some embodiments, the target detection model training method or the target detection method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the target detection model training method or the target detection method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the target detection model training method or the target detection method in any other suitable manner (e.g., by means of firmware).

[0129] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0130] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable target detection model training device or a target detection device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0131] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0132] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0133] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0134] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0135] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0136] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A target detection model training method, the method comprising: Input the samples into the initial target detection model and the teacher model respectively, and obtain the prediction feature map output by the target detection model, the target feature map output by the teacher model, and the text features; Determine target visual information and a target detection frame of a target feature point in the target feature map based on the annotated detection frame in the sample, and determine predicted visual information and a predicted detection frame of a predicted feature point in the predicted feature map; Calculating a visual information distillation loss based on the predicted visual information and the target visual information; Calculating a multimodal distillation loss based on the predicted visual information and the text features; Based on the visual information distillation loss, the multimodal distillation loss, the labeled detection box, and the predicted detection box, the object detection model is trained to obtain a trained object detection model.

2. The method according to claim 1, further comprising: Calculating a detection box distillation loss based on the predicted detection box and the target detection box; When the target detection model is trained, a trained target detection model is obtained based on the detection box distillation loss.

3. The method according to claim 1 or 2, wherein: The target visual information includes: target visual features and target categories, the predicted visual information includes: predicted visual features and predicted categories, the determining of the target visual information and target detection frame of the target feature point in the target feature map based on the labeled detection frame in the sample, and determining the predicted visual information and predicted detection frame of the predicted feature point in the predicted feature map includes: Mapping the annotated detection frame in the sample to the target feature map to obtain a first mapped detection frame on the target feature map; Determining target feature points on the target feature map based on the first mapping detection frame; Extracting the features of the target feature points to obtain target visual features of the target feature points; Determine an object detection frame and an object category of the object based on the position of the first mapping detection frame; Mapping the labeled detection frame onto the predicted feature map to obtain a second mapped detection frame on the predicted feature map; Determining predicted feature points of the target on the predicted feature map based on the second mapping detection frame; Extracting features of the predicted feature points to obtain predicted visual features of the predicted feature points; Based on the position of the second mapped detection box, a predicted detection box and a predicted category of the target are determined.

4. The method according to claim 1 or 2, wherein: The target visual information includes: target visual features and target categories, the predicted visual information includes: predicted visual features and predicted categories, the determining of the target visual information and target detection frame of the target feature point in the target feature map based on the labeled detection frame in the sample, and determining the predicted visual information and predicted detection frame of the predicted feature point in the predicted feature map includes: Determine the center point of the marked detection box in the sample; Mapping the center point of the labeled detection frame to the target feature map to obtain the target feature point on the target feature map; Extracting the features of the target feature points to obtain target visual features of the target feature points; Determine the target detection frame and the target category based on the position of the target feature map mapped to the center point of the marked detection frame; Mapping the center point of the labeled detection frame to the predicted feature map to obtain the predicted feature point of the target on the predicted feature map; Extracting features of the predicted feature points to obtain predicted visual features of the predicted feature points; Based on the mapping of the center point of the labeled detection box to the position of the predicted feature map, a predicted detection box and a predicted category are determined.

5. The method according to claim 1 or 2, wherein: The target visual information includes: target visual features and target categories, the predicted visual information includes: predicted visual features and predicted categories, the visual information distillation loss includes: visual feature distillation loss and classification distillation loss, and the calculation of the visual information distillation loss based on the predicted visual information and the target visual information includes: Calculating a visual feature distillation loss based on the predicted visual feature and the target visual feature; Based on the predicted category and the target category, a classification distillation loss is calculated.

6. The method according to claim 1 or 2, wherein: The predicting visual information includes: predicting visual features, and the calculating the multimodal distillation loss based on the predicted visual information and the text features includes: Calculating the similarity between the text feature and the predicted visual feature to obtain a similarity value; Based on the similarity value and the true value of the text in the sample, a multimodal distillation loss is calculated.

7. The method according to claim 1 or 2, wherein: The training of the target detection model based on the visual information distillation loss, the multimodal distillation loss, the labeled detection frame, and the predicted detection frame to obtain the trained target detection model includes: Calculating a predicted detection box loss based on the predicted detection box and the labeled detection box; Based on the visual information distillation loss, the multimodal distillation loss, and the predicted detection box loss, the object detection model is trained to obtain a trained object detection model.

8. The method according to claim 7, wherein: The training of the target detection model based on the visual information distillation loss, the multimodal distillation loss, and the predicted detection box loss to obtain the trained target detection model includes: The visual information distillation loss, the multimodal distillation loss and the predicted detection box loss are weighted summed to obtain the total model loss; Based on the total loss of the model, the target detection model is trained to obtain a trained target detection model.

9. The method according to claim 1 or 2, wherein: The samples are input into the initial target detection model and the teacher model respectively to obtain the prediction feature map output by the target detection model, the target feature map output by the teacher model, and the text features include: Acquire a sample set, the sample set including at least one sample, the sample including: an image and a search text corresponding to the image; Select samples from the sample set; Inputting the image in the sample into an initial target detection model to obtain a prediction feature map of the target detection model for predicting the image in the sample; The images and texts in the sample are input into the teacher model to obtain the target feature map for predicting the images in the sample and the text features for predicting the texts by the teacher model.

10. A target detection method, the method comprising: Acquire the image to be detected; The image to be detected is input into a target detection model to obtain a detection result of the image to be detected, wherein the target detection model is a trained target detection model obtained by using the target detection model training method according to any one of claims 1 to 9.

11. A target detection model training device, the device comprising: An obtaining unit is configured to input the samples into an initial target detection model and a teacher model respectively, and obtain a prediction feature map output by the target detection model, a target feature map output by the teacher model, and text features; a determination unit configured to determine target visual information and a target detection frame of a target feature point in the target feature map based on the labeled detection frame in the sample, and to determine predicted visual information and a predicted detection frame of a predicted feature point in the predicted feature map; A classification calculation unit, configured to calculate a visual information distillation loss based on the predicted visual information and the target visual information; A modality calculation unit configured to calculate a multimodal distillation loss based on the predicted visual information and the text features; The training unit is configured to train the target detection model based on the visual information distillation loss, the multimodal distillation loss, the labeled detection box and the predicted detection box to obtain a trained target detection model.

12. The device according to claim 11, further comprising: A detection box calculation unit, configured to calculate a detection box distillation loss based on the predicted detection box and the target detection box; When the target detection model is trained, a trained target detection model is obtained based on the detection box distillation loss.

13. The device according to claim 11 or 12, wherein: The target visual information includes: target visual features and target categories, the predicted visual information includes: predicted visual features and predicted categories, and the determination unit is further configured to: map the annotated detection frame in the sample to the target feature map to obtain a first mapped detection frame on the target feature map; extract the features of the target feature points to obtain the target visual features of the target feature points; determine the target detection frame and target category of the target based on the position of the first mapped detection frame; map the annotated detection frame to the predicted feature map to obtain a second mapped detection frame on the predicted feature map; determine the predicted feature points of the target on the predicted feature map based on the second mapped detection frame; extract the features of the predicted feature points to obtain the predicted visual features of the predicted feature points; and determine the predicted detection frame and predicted category of the target based on the position of the second mapped detection frame.

14. The device according to claim 11 or 12, wherein: The target visual information includes: target visual features and target categories, the predicted visual information includes: predicted visual features and predicted categories, and the determination unit is further configured to: determine the center point of the annotated detection box in the sample; map the center point of the annotated detection box to the target feature map to obtain the target feature point on the target feature map; extract the features of the target feature point to obtain the target visual features of the target feature point; determine the target detection box and the target category based on the position where the center point of the annotated detection box is mapped to the target feature map; map the center point of the annotated detection box to the predicted feature map to obtain the predicted feature point of the target on the predicted feature map; extract the features of the predicted feature point to obtain the predicted visual features of the predicted feature point; determine the predicted detection box and the predicted category based on the position where the center point of the annotated detection box is mapped to the predicted feature map.

15. The device according to claim 11 or 12, wherein: The target visual information includes: target visual features and target categories, the predicted visual information includes: predicted visual features and predicted categories, the visual information distillation loss includes: visual feature distillation loss and classification distillation loss, and the classification calculation unit is configured to: calculate the visual feature distillation loss based on the predicted visual features and the target visual features; calculate the classification distillation loss based on the predicted categories and the target categories.

16. The device according to claim 11 or 12, wherein: The predicted visual information includes: predicted visual features, and the modal calculation unit is configured to: calculate the similarity between the text features and the predicted visual features to obtain a similarity value; and calculate the multimodal distillation loss based on the similarity value and the true value of the text in the sample.

17. The device according to claim 11 or 12, wherein: The training unit is configured to: calculate the predicted detection box loss based on the predicted detection box and the labeled detection box; train the target detection model based on the visual information distillation loss, the multimodal distillation loss and the predicted detection box loss to obtain a trained target detection model.

18. The device according to claim 17, wherein: The training unit is further configured to: perform weighted summation of the visual information distillation loss, the multimodal distillation loss, and the predicted detection box loss to obtain a total model loss; and train the target detection model based on the total model loss to obtain a trained target detection model.

19. The device according to claim 11 or 12, wherein: The obtaining unit is configured to: obtain a sample set, the sample set includes at least one sample, and the sample includes: an image and a search text corresponding to the image; select a sample from the sample set; input the image in the sample into an initial target detection model to obtain a prediction feature map of the target detection model predicting the image in the sample; input the image and text in the sample into a teacher model to obtain a target feature map of the teacher model predicting the image in the sample and a text feature of the teacher model predicting the text.

20. A target detection device, comprising: An acquisition unit, configured to acquire an image to be detected; The input unit is configured to input the image to be detected into the target detection model to obtain the detection result of the image to be detected. The target detection model is a trained target detection model obtained by using the target detection model training device described in any one of claims 11-19.

21. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 10.

23. A computer program product, comprising a computer program, which, when executed by a processor, implements the method of any one of claims 1 to 10.

Images