Fine-tuning Method, Device, Electronic Device and Medium of Open-domain Object Detection Model

Abstract

The present application provides a fine-tuning method, device, electronic device and medium for an open-domain object detection model. The method includes: obtaining a pre-trained open-domain object detection model and object detection training data in a specified business scenario; adding a first decoder detection head matching the specified business scenario to the open-domain object detection model to obtain an updated open-domain object detection model; freezing the parameters of other model parts of the updated open-domain object detection model except the first decoder detection head, and training the updated open-domain object detection model based on the object detection training data to obtain a predicted object detection result of the object detection training data; calculating a loss value of the updated open-domain object detection model based on the predicted object detection result and the annotation result of the object detection training data; and obtaining a final open-domain object detection model when the loss value indicates that the updated open-domain object detection model has converged.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and particularly to a fine-tuning method, device, electronic device, and medium for an open-domain object detection model. Background Art

[0002] Open-domain object detection is a computer vision technology aimed at identifying and locating various objects in images or videos, and is not limited to predefined object categories. Different from traditional object detection, open-domain object detection faces dynamically changing and unknown object categories, requiring the algorithm to have high generalization ability and adaptability. Common open-domain object detection models include Grounding-dino, Yolo-World, DetClip v2, etc.

[0003] When an open-domain object detection model is initially trained, it usually learns the ability to process various novel objects and scenes through large-scale and diverse datasets. This training enables the model to exhibit high robustness and adaptability when facing unknown objects and concepts. However, currently, fine-tuning the full parameters of an open-domain object detection model using small sample data will have an adverse impact on the detection ability of the model. When comprehensively fine-tuning the model to adapt to a new task or dataset, it will cause changes in the original feature representation and learning ability of the model. This change will damage the performance of the model in open-domain scenarios and reduce its ability to process emerging or unseen objects, thus affecting the accuracy and comprehensiveness of detection. In addition, when an open-domain object detection model undertakes multiple object detection tasks simultaneously, it may encounter interference between tasks. In this case, the parameters and features of the model may need to be balanced between different tasks, resulting in the performance of some tasks not meeting expectations. This interference between tasks makes it difficult for the model to ensure high-precision performance for each task when processing multiple tasks simultaneously. In practical applications, various complex scenarios are often faced. After a model is trained with industry data, it can ensure the detection accuracy of relevant categories while also having the ability to detect other targets that have not been trained. Summary of the Invention

[0004] The technical problem to be solved by the embodiments of this application is to provide a fine-tuning method, device, electronic device, and medium for an open-domain object detection model, so as to train an open-domain object detection model with open-domain detection ability and high-precision object detection ability for different industries' businesses.

[0005] In a first aspect, the embodiments of this application provide a fine-tuning method for an open-domain object detection model, and the method includes:

[0006] Obtain a pre-trained open-domain object detection model and target detection training data for object detection in a specified business scenario;

[0007] Add a first decoder detection head that matches the specified business scenario to the open-domain object detection model to obtain an updated open-domain object detection model, and the first decoder detection head is connected in parallel with the original second decoder detection head in the open-domain object detection model;

[0008] Freeze the parameters of other model parts of the updated open-domain object detection model except the first decoder detection head, and perform model training on the updated open-domain object detection model based on the object detection training data to obtain the predicted object detection results of the object detection training data output by the updated open-domain object detection model;

[0009] Calculate the loss value of the updated open-domain object detection model based on the predicted object detection results and the annotation results of the object detection training data;

[0010] When the loss value indicates that the updated open-domain object detection model has converged, obtain the final open-domain object detection model.

[0011] Optionally, the open-domain object detection model further includes: an image feature extraction module, a text feature extraction module, and a feature fusion module,

[0012] The performing model training on the updated open-domain object detection model based on the object detection training data to obtain the predicted object detection results of the object detection training data output by the updated open-domain object detection model includes:

[0013] Call the image feature extraction module to extract features from the sample images in the object detection training data to obtain the image features of the sample images;

[0014] Call the text feature extraction module to extract features from the text in the object detection training data to obtain the text features of the text;

[0015] Call the feature fusion module to perform interaction and fusion processing on the image features and the text features to obtain fusion features;

[0016] Call the first decoder detection head to process the fusion features to obtain the predicted object detection results.

[0017] Optionally, the calling the first decoder detection head to process the fusion features to obtain the predicted object detection results includes:

[0018] Call the first decoder detector to process the fused features, and obtain the predicted annotation boxes in the sample images in the target detection training data and the predicted categories corresponding to the predicted annotation boxes;

[0019] Use the predicted annotation boxes and the predicted categories corresponding to the predicted annotation boxes as the predicted target detection results.

[0020] Optionally, calculating the loss value of the updated open-domain target detection model based on the predicted target detection results and the annotation results of the target detection training data includes:

[0021] Calculate the GIoU loss value and the L1 loss value based on the predicted annotation boxes and the corresponding ground truth annotation boxes;

[0022] Calculate the contrast loss value based on the predicted categories and the corresponding ground truth annotation categories;

[0023] Calculate the weighted average of the GIoU loss value, the L1 loss value, and the contrast loss value, and use this weighted average as the loss value.

[0024] Optionally, after calculating the loss value of the updated open-domain target detection model based on the predicted target detection results and the annotation results of the target detection training data, it further includes:

[0025] In the case where the loss value indicates that the updated open-domain target detection model has not converged, minimize the loss value based on the gradient descent method to obtain the model prediction error;

[0026] Backpropagate the model prediction error to the first decoder detector to update the parameters of the first decoder detector, and iteratively execute the step of freezing the parameters of other model parts of the updated open-domain target detection model except the first decoder detector until the updated open-domain target detection model converges.

[0027] Optionally,

[0028] When the specified business scenario is multiple business scenarios, each of the specified business scenarios corresponds to a first decoder detector; or

[0029] When the specified business scenario is multiple business scenarios, multiple of the specified business scenarios correspond to a first decoder detector.

[0030] In a second aspect, an embodiment of the present application provides a fine-tuning device for an open-domain target detection model, and the device includes:

[0031] A target training data acquisition module, configured to acquire a pre-trained open-domain object detection model and object detection training data for object detection in a specified business scenario;

[0032] An updated model acquisition module, configured to add a first decoder detection head matching the specified business scenario to the open-domain object detection model to obtain an updated open-domain object detection model, where the first decoder detection head is connected in parallel with a second decoder detection head originally in the open-domain object detection model;

[0033] A predicted detection result acquisition module, configured to freeze the parameters of other model parts of the updated open-domain object detection model except the first decoder detection head, and perform model training on the updated open-domain object detection model based on the object detection training data to obtain a predicted object detection result of the object detection training data output by the updated open-domain object detection model;

[0034] A loss value calculation module, configured to calculate a loss value of the updated open-domain object detection model based on the predicted object detection result and the annotation result of the object detection training data;

[0035] An object detection model acquisition module, configured to obtain a final open-domain object detection model when the loss value indicates that the updated open-domain object detection model has converged.

[0036] Optionally, the open-domain object detection model further includes: an image feature extraction module, a text feature extraction module, and a feature fusion module,

[0037] The predicted detection result acquisition module includes:

[0038] An image feature acquisition unit, configured to call the image feature extraction module to extract features from a sample image in the object detection training data to obtain an image feature of the sample image;

[0039] A text feature acquisition unit, configured to call the text feature extraction module to extract features from the text in the object detection training data to obtain a text feature of the text;

[0040] A fused feature acquisition unit, configured to call the feature fusion module to perform interaction and fusion processing on the image feature and the text feature to obtain a fused feature;

[0041] A predicted detection result acquisition unit, configured to call the first decoder detection head to process the fused feature to obtain the predicted object detection result.

[0042] Optionally, the prediction detection result acquisition unit includes:

[0043] A prediction category acquisition subunit, configured to process the fusion feature by invoking the first decoder detection head, and obtain a predicted annotation box in the sample image of the target detection training data and a prediction category corresponding to the predicted annotation box;

[0044] A prediction detection result acquisition subunit, configured to use the predicted annotation box and the prediction category corresponding to the predicted annotation box as the predicted target detection result.

[0045] Optionally, the loss value calculation module includes:

[0046] An L1 loss calculation unit, configured to calculate a GIoU loss value and an L1 loss value based on the predicted annotation box and the true annotation box corresponding to the predicted annotation box;

[0047] A contrast loss calculation unit, configured to calculate a contrast loss value based on the predicted category and the corresponding true annotation category;

[0048] A loss value calculation unit, configured to calculate a weighted average of the GIoU loss value, the L1 loss value, and the contrast loss value, and use the weighted average as the loss value.

[0049] Optionally, the apparatus further includes:

[0050] A prediction error acquisition module, configured to, when the loss value indicates that the updated open-domain target detection model has not converged, minimize the loss value based on the gradient descent method to obtain a model prediction error;

[0051] A target detection model adjustment module, configured to backpropagate the model prediction error to the first decoder detection head to update the parameters of the first decoder detection head, and iteratively execute the prediction detection result acquisition module and the loss value calculation module until the updated open-domain target detection model converges.

[0052] Optionally,

[0053] when the specified service scenario is multiple service scenarios, each of the specified service scenarios corresponds to one of the first decoder detection heads; or

[0054] when the specified service scenario is multiple service scenarios, multiple of the specified service scenarios correspond to one of the first decoder detection heads.

[0055] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0056] A processor, a memory, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the method for fine-tuning the open-domain object detection model described in any one of the above is implemented.

[0057] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the method for fine-tuning the open-domain object detection model described in any one of the above.

[0058] Compared with the prior art, the embodiments of the present application include the following advantages:

[0059] In the embodiments of the present application, by obtaining a pre-trained open-domain object detection model and object detection training data for object detection in a specified business scenario. A first decoder detection head matching the specified business scenario is added to the open-domain object detection model to obtain an updated open-domain object detection model, and the first decoder detection head is connected in parallel with the original second decoder detection head in the open-domain object detection model. Freeze the parameters of other model parts of the updated open-domain object detection model except the first decoder detection head, and perform model training on the updated open-domain object detection model based on the object detection training data to obtain a predicted object detection result of the object detection training data output by the updated open-domain object detection model. Based on the predicted object detection result and the annotation result of the object detection training data, calculate the loss value of the updated open-domain object detection model. When the loss value indicates that the updated open-domain object detection model has converged, obtain the final open-domain object detection model. In the embodiments of the present application, by adding a decoder detection head matching the specified business scenario to the open-domain object detection model and training the pre-trained open-domain object detection model in combination with the training data of multiple business scenarios, the trained open-domain object detection model can not only have the open-domain detection ability, but also achieve high-precision object detection ability in the business scenarios of multiple industries. This method is not only applicable to high-precision open-domain detection, but also can meet the existing object detection requirements of the industry, improving the generalization of the open-domain object detection model.

[0060] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. Description of the Drawings

[0061] Figure 1 It is a flowchart of the steps of a method for fine-tuning an open-domain object detection model provided by an embodiment of the present application;

[0062] Figure 2 It is a model structure diagram of an open-domain object detection model provided by an embodiment of the present application;

[0063] Figure 3 Schematic diagram of a fine-tuned open-domain object detection model provided by an embodiment of the present application;

[0064] Figure 4 Structural diagram of a decoder detection head provided by an embodiment of the present application;

[0065] Figure 5 Structural schematic diagram of a fine-tuning device for an open-domain object detection model provided by an embodiment of the present application;

[0066] Figure 6 Structural schematic diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0067] To make the above objectives, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below with reference to the accompanying drawings and specific implementation manners.

[0068] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms "a", "said", and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0069] Open-Domain Object Detection is an advanced task in computer vision, aiming to solve the performance problems of traditional object detection models when encountering unknown categories.

[0070] Compared with traditional closed-domain object detection models, the main difference of open-domain object detection models is that they can handle new categories that were not seen during training. This requires the model to have a certain generalization ability to make reasonable responses when facing new categories, such as recognizing that this is an unknown category instead of misclassifying it as a known category. After being trained on industry data, the open-domain detection model can, while ensuring the detection accuracy of relevant categories, also have the ability to detect other targets that have not been trained. Existing methods need to fine-tune all parameters of the open-domain object detection model, which will change the weights and parameters of the original model, thus losing or reducing the open-domain detection ability. And there will be interference problems when multiple tasks are trained simultaneously, making it difficult to ensure high accuracy for multiple tasks at the same time.

[0071] To solve the above problems, the embodiments of the present application aim to train the model on data from multiple industries so that it not only has the open-domain detection ability but also achieves high-precision object detection ability in the businesses of multiple industries. This method is applicable to both high-precision open-domain detection and can meet the existing object detection requirements of the industry.

[0072] Next, the technical solution of the present application will be described in detail in combination with specific embodiments.

[0073] Referring to Figure 1 , a flowchart of steps of a fine-tuning method for an open-domain object detection model provided by an embodiment of the present application is shown. As Figure 1 shown, the fine-tuning method for the open-domain object detection model may include: step 101, step 102, step 103, step 104, and step 105.

[0074] Step 101: Obtain a pre-trained open-domain object detection model and object detection training data for object detection in a specified business scenario.

[0075] Embodiments of the present application can be applied to the scenario of fine-tuning a pre-trained open-domain object detection model by combining object detection training data for object detection in a specified business scenario.

[0076] The specified business scenario refers to a business scenario that requires object detection within a certain industry. In this example, the specified business scenario may be, but is not limited to, a security monitoring scenario, an intelligent transportation scenario, a remote sensing image analysis scenario, etc.

[0077] The object detection training data refers to model training data for fine-tuning a pre-trained open-domain object detection model. In this example, the object detection training data may be composed of images, texts, etc. within the specified business scenario.

[0078] In practical applications, annotation data related to the object detection training data may also be prepared. These data need to contain clear object box annotations, including the location and category of the detection target, and be converted into a format suitable for training.

[0079] When enabling the pre-trained open-domain object detection model to have a high-precision object detection function for a specified business scenario, a pre-trained open-domain object detection model and object detection training data for object detection in the specified business scenario can be obtained.

[0080] After obtaining the pre-trained open-domain object detection model and the object detection training data for object detection in the specified business scenario, step 102 is executed.

[0081] Step 102: Add a first decoder detection head that matches the specified business scenario to the open-domain object detection model to obtain an updated open-domain object detection model. The first decoder detection head is connected in parallel with the original second decoder detection head in the open-domain object detection model.

[0082] The second decoder detection head refers to the original decoder detection head of the pre-trained open-domain object detection model.

[0083] The first decoder detection head refers to the newly added decoder detection head in the pre-trained open-domain object detection model for object detection in a specified business scenario.

[0084] After obtaining the pre-trained open-domain object detection model, a first decoder detection head matching the specified business scenario can be added to the open-domain object detection model to obtain an updated open-domain object detection model, in which the first decoder detection head and the second decoder detection head are in parallel.

[0085] The method of adding a first decoder detection head matching the specified business scenario to the open-domain object detection model can be as follows: deeply analyze the architecture of the current open-domain object detection model, especially the decoder part, to determine the functions, inputs and outputs of the existing decoder detection head (i.e., the second decoder detection head in this example) and its interaction with other model components. According to the specific requirements of the business scenario, clarify the target functions of the new decoder detection head, such as detecting specific types of objects, improving detection accuracy, etc., and consider the complementarity between the new decoder detection head and the existing decoder detection head to avoid overlapping functions. Select a suitable network architecture for the new decoder detection head (i.e., the first decoder detection head in this example), such as convolutional neural network (CNN), Transformer, etc., and design the input and output layers of the new decoder detection head to ensure compatibility with the existing model. On the basis of the existing model, add the relevant layers of the new decoder detection head to ensure that the connection between the new decoder detection head and the existing model is correct.

[0086] In this example, when the specified business scenario is multiple business scenarios, each specified business scenario corresponds to a first decoder detection head. For example, if the specified business scenarios include: monitoring business scenario and intelligent transportation business scenario, then two first decoder detection heads can be added to the open-domain object detection model: decoder detection head 1 and decoder detection head 2. Decoder detection head 1 is used to process data in the monitoring business scenario, and decoder detection head 2 is used to process data in the intelligent transportation business scenario, etc.

[0087] When the specified business scenario is multiple business scenarios, multiple specified business scenarios correspond to a first decoder detection head. For example, if the specified business scenarios include: monitoring business scenario and intelligent transportation business scenario, then a decoder detection head, i.e., decoder detection head 1, can be added to the open-domain object detection model. Decoder detection head 1 can be used to process data in the monitoring business scenario and the intelligent transportation business scenario, etc.

[0088] Understandably, the above examples are only examples listed for better understanding the technical solutions of the embodiments of the present application, and are not the only limitation to this embodiment.

[0089] After adding a first decoder detection head matching the specified business scenario to the open-domain object detection model to obtain an updated open-domain object detection model, step 103 is executed.

[0090] Step 103: Freeze the parameters of other model parts of the updated open-domain object detection model except the first decoder detection head, and perform model training on the updated open-domain object detection model based on the object detection training data to obtain the predicted object detection results of the object detection training data output by the updated open-domain object detection model.

[0091] After adding a first decoder detection head matching the specified business scenario to the open-domain object detection model to obtain an updated open-domain object detection model, the parameters of other model parts of the updated open-domain object detection model except the first decoder detection head can be frozen, and the updated open-domain object detection model can be trained based on the object detection training data to obtain the predicted object detection results of the object detection training data output by the updated open-domain object detection model.

[0092] In the open-domain object detection model, freezing the parameters of other model parts of the updated open-domain object detection model except the first decoder detection head is to keep the feature extraction ability of this part of the network unchanged, and at the same time only train other parts of the model. In a specific implementation, first, it is necessary to clarify which layers of other model parts of the updated open-domain object detection model except the first decoder detection head are to be frozen, and then these layers can be frozen by setting the requires_grad attribute of the parameters to False.

[0093] The model fine-tuning process of the pre-trained open-domain object detection model can be described in detail in combination with the following specific implementation manners.

[0094] In a specific implementation of the present application, the open-domain object detection model further includes: an image feature extraction module, a text feature extraction module, and a feature fusion module. The above step 103 may include:

[0095] Sub-step A1: Call the image feature extraction module to extract features from the sample images in the object detection training data to obtain the image features of the sample images.

[0096] In this embodiment, the open-domain object detection model further includes: an image feature extraction module, a text feature extraction module, and a feature fusion module. The structure of the open-domain object detection model can be as Figure 2As shown in the figure, it includes: an Image Backbone (image feature extraction module), a Text Encoder (text feature extraction module), a FeatureEnhancer (feature fusion module), and a Decoder (decoder detection head module).

[0097] After obtaining the object detection training data in a specified business scenario, the Image Backbone can be called first to extract features from the sample images in the object detection training data, obtaining the image features of the sample images. Specifically, the Image Backbone can use a convolutional neural network or other visual feature extractors (such as Vision Transformer, etc.) for processing to generate high-dimensional Image Features, which contain spatial and context information.

[0098] Sub-step A2: Call the Text Encoder to extract features from the text in the object detection training data, obtaining the text features of the text.

[0099] The Text Encoder can extract features from the text in the object detection training data, obtaining the text features of the text. Specifically, the Text Encoder processes the text for query (such as Figure 2 the text "A man with a knife" shown in the figure) through a language model (such as the text encoder of BERT or CLIP, etc.) to generate text features (Text Features) that are aligned with the visual features, and these features express the semantic information of the query.

[0100] Sub-step A3: Call the FeatureEnhancer to perform interaction and fusion processing on the image features and the text features, obtaining fused features.

[0101] After obtaining the image features and the text features, the FeatureEnhancer can be called to perform interaction and fusion processing on the image features and the text features to obtain fused features. As Figure 2 shown in the figure, the FeatureEnhancer can achieve the interaction and fusion of visual and text features through a Transformer structure. Specifically, the Transformer encoder can receive the image features and the text features and use the self-attention mechanism to align the image features and the text features spatially and semantically. This process helps to detect the objects in the image that are relevant to the query text.

[0102] After calling the FeatureEnhancer to perform interaction and fusion processing on the image features and the text features to obtain fused features, execute Sub-step A4.

[0103] Sub-step A4: Invoke the first decoder detection head to process the fused features to obtain the predicted target detection results.

[0104] After invoking the feature fusion module to interact and fuse the image features and text features to obtain the fused features, the first decoder detection head can be invoked to process the fused features to obtain the predicted target detection results. Specifically, the main task of the decoder detection head module is to process the fused features from the encoder and generate the target detection boxes and classification results.

[0105] The processing process of the first decoder detection head for the fused features can be described in detail in combination with the following specific implementation manners.

[0106] In another specific implementation of the present application, the above sub-step A4 may include:

[0107] Sub-step B1: Invoke the first decoder detection head to process the fused features to obtain the predicted annotation boxes in the sample image in the target detection training data and the predicted categories corresponding to the predicted annotation boxes.

[0108] In this embodiment, the structure of the first decoder detection head may be as Figure 4 shown. The first decoder detection head may be composed of a self-attention layer, a cross-attention layer, and a feed-forward network (FFN). The self-attention layer enables the query embeddings to interact with each other through the self-attention mechanism, capturing the relationships between different targets, thereby helping the model better distinguish different objects in a multi-target scenario. The cross-attention mechanism of the cross-attention layer allows the query embeddings to interact with the visual and text features, enabling the model to generate detection boxes and classification predictions related to the text query based on the text semantics and visual information of the image. The feed-forward network is a two-layer fully connected network using the ReLU activation function. After each attention mechanism, the feed-forward network further optimizes the query features to improve the detection performance of the model. These decoder detection heads are designed to output the positions and related features of the target boxes. The number and structure of the decoder detection heads can be adjusted according to the complexity of the task and the diversity of the targets. In this way, the detection heads can be customized for each task to improve the detection performance of the model on specific tasks and output the target box positions and features.

[0109] The first decoder detection head can be invoked to process the fused features to obtain the predicted annotation boxes in the sample image in the target detection training data and the predicted categories corresponding to the predicted annotation boxes.

[0110] Sub-step B2: Use the predicted annotation boxes and the predicted categories corresponding to the predicted annotation boxes as the predicted target detection results.

[0111] Then, the predicted bounding box and the predicted category corresponding to the predicted bounding box can be used as the predicted object detection result. In this example,

[0112] After training the updated open-domain object detection model based on the object detection training data to obtain the predicted object detection result of the object detection training data output by the updated open-domain object detection model, step 104 is executed.

[0113] Step 104: Calculate the loss value of the updated open-domain object detection model based on the predicted object detection result and the annotation result of the object detection training data.

[0114] After training the updated open-domain object detection model based on the object detection training data to obtain the predicted object detection result of the object detection training data output by the updated open-domain object detection model, the loss value of the updated open-domain object detection model can be calculated based on the predicted object detection result and the annotation result of the object detection training data. In this example, the loss value can be obtained by weighted summation of the GIoU loss value, the L1 loss value, and the contrastive loss value. Among them, the contrastive loss is a loss function used to measure the similarity of paired samples and is widely used in metric learning and similarity learning tasks. Its goal is to learn feature embeddings by pulling closer the distances of similar samples and pulling farther the distances of dissimilar samples. In the open-domain object detection model, it is usually used to reduce the feature difference between the query embedding and the text feature, so that the text description better matches the objects in the image. This loss helps the model better distinguish similar categories and accurately locate the text-related objects. The calculation process of the loss value can be described in detail in combination with the following specific implementation.

[0115] In a specific implementation of the present application, the above step 104 may include:

[0116] Sub-step C1: Calculate the GIoU loss value and the L1 loss value based on the predicted bounding box and the corresponding ground truth bounding box of the predicted bounding box.

[0117] In this embodiment, the GIoU (Generalized Intersection over Union) loss is a loss function used to measure the relationship between the predicted bounding box and the ground truth bounding box. It is developed based on the IoU (Intersection over Union) loss and aims to address the problem that the IoU loss cannot effectively reflect the distance between the predicted bounding box and the ground truth bounding box when they do not intersect or have a low overlap degree. The GIoU loss introduces an additional penalty term, which is the difference between the area of the minimum bounding rectangle of the predicted bounding box and the ground truth bounding box and the area of their union, to more comprehensively measure the relationship between the two.

[0118] The L1 loss, also known as the Mean Absolute Error (MAE), is a loss function used to measure the difference between the predicted value and the true value. It calculates the loss value by summing and averaging the absolute differences between the predicted value and the true value.

[0119] After obtaining the predicted annotation bounding box, the GIoU loss value and the L1 loss value can be calculated based on the predicted annotation bounding box and the corresponding ground truth annotation bounding box.

[0120] Sub-step C2: Calculate the contrast loss value based on the predicted class and the corresponding ground truth annotation class.

[0121] Meanwhile, the contrast loss value can be calculated based on the predicted class and the corresponding ground truth annotation class. For example, the input of the first decoder detection head includes query features (Query Features) with a feature dimension of (900, 256), as well as image features (Image Features) and text features (Text Features). The output includes class embedding (ClassEmbedding) and box embedding (Box Embedding), where the feature dimension of the class embedding is (900, 256), and the feature dimension of the box embedding is (900, 4), representing the center coordinates (cx, cy) of the detection box and its width and height (w, h) respectively. For the regression of the bounding box, the model optimizes the position of the detection box by calculating the GIoU loss and the L1 loss between the GT box and the box embedding. The query features and the text features reduce the difference between the features through the contrastive loss (Contrastive Loss), thereby improving the model's ability to distinguish different classes.

[0122] Sub-step C3: Calculate the weighted average of the GIoU loss value, the L1 loss value, and the contrast loss value, and use this weighted average as the loss value.

[0123] After calculating the GIoU loss value, L1 loss value, and contrast loss value, the weighted average of the GIoU loss value, L1 loss value, and contrast loss value can be calculated, and this weighted average is used as the final loss value.

[0124] Step 105: When the loss value indicates that the updated open-domain object detection model has converged, obtain the final open-domain object detection model.

[0125] After calculating the loss value of the updated open-domain object detection model, it can be determined whether the updated open-domain object detection model has converged based on this loss value.

[0126] If the loss value indicates that the updated open-domain object detection model has converged, the final open-domain object detection model can be obtained, that is, the trained updated open-domain object detection model is used as the final open-domain object detection model, and this final open-domain object detection model has the object detection ability in the specified business scenario.

[0127] If the loss value indicates that the updated open-domain object detection model has not converged, the loss value can be minimized based on the gradient descent method to obtain the model prediction error, and the model prediction error is backpropagated to the first decoder detection head to update the parameters of the first decoder detection head. Specifically, the backpropagation algorithm can be used to calculate the gradient of the loss with respect to the model parameters layer by layer starting from the loss function. These gradients indicate how to adjust the parameters to reduce the loss, and during the backpropagation process, ensure that the gradients are correctly passed to the parameters of the first decoder detection head. Furthermore, based on the calculated gradients, the parameters of the first decoder detection head can be updated using the gradient descent method (or its variants, such as Stochastic Gradient Descent SGD, Adam, etc.). When updating the parameters, a learning rate is usually multiplied to control the step size of the parameter update.

[0128] Furthermore, the parameters of the second decoder detection head of the frozen updated open-domain object detection model can be iteratively updated until the step of calculating the loss value of the updated open-domain object detection model is reached, until the updated open-domain object detection model converges. Finally, a detection model suitable for a specific task is obtained. This model not only retains the wide applicability of the open-domain object detection model but also improves the detection accuracy and performance on specific tasks through the newly added decoder detection head. The model can accurately identify and detect objects in a specific task, meeting industry requirements.

[0129] In practical applications, when there are new tasks, repeat the above steps 101 to 105. Prepare new task annotation data, add corresponding decoder detection heads, calculate the loss, and update the parameters. Through this iterative process, the model can gradually adapt to multiple tasks and form a multi-task adapted detection model. In this way, the model not only performs well in a single task but also can handle various different detection requirements, with higher application universality and flexibility. As Figure 3 shown, the model structure can include: decoder detection head 1, decoder detection head 2,..., decoder detection head N (N is a positive integer) for performing object detection tasks in different business scenarios.

[0130] The embodiment of the present application can retain the original open-domain detection ability: The integrity and stability of the original model architecture and weights are maintained during design, which ensures that the detection ability of the model in the open domain is retained and exerted, which is crucial for ensuring the adaptability of the model in different tasks and environments.

[0131] At the same time, it can adapt to different tasks: This embodiment introduces additional decoder detection heads. On the premise of adding less computational cost, the detection accuracy and efficiency of specific tasks can be significantly improved. By adding these dedicated detection heads, the model can more accurately identify and classify specific types of targets. This method not only optimizes resource utilization but also improves the flexibility and accuracy of the detection system, and multiple tasks avoid interfering with each other during training.

[0132] Moreover, an open-domain object detection model adapted to complex industry scenarios is trained. After being trained with industry data, the model can, while ensuring the detection accuracy of relevant categories, also have the ability to detect other targets that have not been trained. This makes the model more flexible and reliable when applied to actual industry problems and demonstrates high application generalization.

[0133] The fine-tuning method of the open-domain object detection model provided by the embodiment of the present application includes obtaining a pre-trained open-domain object detection model and object detection training data for object detection in a specified business scenario. A first decoder detection head matching the specified business scenario is added to the open-domain object detection model to obtain an updated open-domain object detection model. The first decoder detection head is connected in parallel with the original second decoder detection head in the open-domain object detection model. The parameters of other model parts except the first decoder detection head in the updated open-domain object detection model are frozen, and the updated open-domain object detection model is trained based on the object detection training data to obtain the predicted object detection results of the object detection training data output by the updated open-domain object detection model. Based on the predicted object detection results and the annotation results of the object detection training data, the loss value of the updated open-domain object detection model is calculated. When the loss value indicates that the updated open-domain object detection model has converged, the final open-domain object detection model is obtained. By adding a decoder detection head matching the specified business scenario to the open-domain object detection model and training the pre-trained open-domain object detection model in combination with the training data of multiple business scenarios, the open-domain object detection model trained by the embodiment of the present application not only has the open-domain detection ability, but also realizes the high-precision object detection ability in the business scenarios of multiple industries. This method is suitable for both high-precision open-domain detection and can meet the existing object detection requirements of the industry, improving the generalization of the open-domain object detection model.

[0134] Referring to Figure 5 , a schematic structural diagram of a fine-tuning device for an open-domain object detection model provided by an embodiment of the present application is shown. As Figure 5 shown, the fine-tuning device 500 for the open-domain object detection model may include the following modules:

[0135] An object training data acquisition module 510, configured to obtain a pre-trained open-domain object detection model and object detection training data for object detection in a specified business scenario;

[0136] An updated model acquisition module 520, configured to add a first decoder detection head matching the specified business scenario to the open-domain object detection model to obtain an updated open-domain object detection model, where the first decoder detection head is connected in parallel with the original second decoder detection head in the open-domain object detection model;

[0137] The prediction detection result acquisition module 530 is configured to freeze the parameters of other model parts of the updated open-domain object detection model except the first decoder detection head, and perform model training on the updated open-domain object detection model based on the object detection training data to obtain the predicted object detection result of the object detection training data output by the updated open-domain object detection model;

[0138] The loss value calculation module 540 is configured to calculate the loss value of the updated open-domain object detection model based on the predicted object detection result and the annotation result of the object detection training data;

[0139] The object detection model acquisition module 550 is configured to obtain the final open-domain object detection model when the loss value indicates that the updated open-domain object detection model has converged.

[0140] Optionally, the open-domain object detection model further includes: an image feature extraction module, a text feature extraction module, and a feature fusion module.

[0141] The prediction detection result acquisition module includes:

[0142] The image feature acquisition unit is configured to call the image feature extraction module to extract features from the sample image in the object detection training data to obtain the image features of the sample image;

[0143] The text feature acquisition unit is configured to call the text feature extraction module to extract features from the text in the object detection training data to obtain the text features of the text;

[0144] The fusion feature acquisition unit is configured to call the feature fusion module to perform interaction and fusion processing on the image features and the text features to obtain fusion features;

[0145] The prediction detection result acquisition unit is configured to call the first decoder detection head to process the fusion features to obtain the predicted object detection result.

[0146] Optionally, the prediction detection result acquisition unit includes:

[0147] The prediction class acquisition subunit is configured to call the first decoder detection head to process the fusion features to obtain the predicted annotation box in the sample image in the object detection training data and the prediction class corresponding to the predicted annotation box;

[0148] The prediction detection result acquisition subunit is configured to use the predicted annotation box and the prediction class corresponding to the predicted annotation box as the predicted object detection result.

[0149] Optionally, the loss value calculation module includes:

[0150] An L1 loss calculation unit, configured to calculate a GIoU loss value and an L1 loss value based on the predicted bounding box and the corresponding ground truth bounding box;

[0151] A contrast loss calculation unit, configured to calculate a contrast loss value based on the predicted category and the corresponding ground truth category;

[0152] A loss value calculation unit, configured to calculate a weighted average of the GIoU loss value, the L1 loss value, and the contrast loss value, and use the weighted average as the loss value.

[0153] Optionally, the apparatus further includes:

[0154] A prediction error acquisition module, configured to, when the loss value indicates that the updated open-domain object detection model has not converged, minimize the loss value based on the gradient descent method to obtain a model prediction error;

[0155] An object detection model adjustment module, configured to backpropagate the model prediction error to the first decoder detection head to update the parameters of the first decoder detection head, and iteratively execute the prediction detection result acquisition module and the loss value calculation module until the updated open-domain object detection model converges.

[0156] Optionally,

[0157] when the specified service scenario is multiple service scenarios, each specified service scenario corresponds to one first decoder detection head; or

[0158] when the specified service scenario is multiple service scenarios, multiple specified service scenarios correspond to one first decoder detection head.

[0159] The fine-tuning device of the open domain target detection model provided in the embodiment of the present application obtains a pre-trained open domain target detection model and target detection training data for target detection in a specified business scenario. A first decoder detection head that matches the specified business scenario is added to the open domain target detection model to obtain an updated open domain target detection model, and the first decoder detection head is connected in parallel with the original second decoder detection head in the open domain target detection model. The parameters of the other model parts of the updated open domain target detection model except the first decoder detection head are frozen, and the updated open domain target detection model is trained based on the target detection training data to obtain the predicted target detection results of the target detection training data output by the updated open domain target detection model. Based on the predicted target detection results and the labeling results of the target detection training data, the loss value of the updated open domain target detection model is calculated. When the loss value indicates that the updated open domain target detection model has converged, the final open domain target detection model is obtained. The embodiments of the present application add a decoder detection head that matches the specified business scenario to the open domain target detection model, and train the pre-trained open domain target detection model in combination with training data from multiple business scenarios. This allows the trained open domain target detection model to not only have open domain detection capabilities, but also achieve high-precision target detection capabilities in business scenarios in multiple industries. This approach is suitable for high-precision open domain detection, and can also meet the industry's existing target detection needs, thereby improving the generalization of the open domain target detection model.

[0160] An embodiment of the present application also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the above-mentioned method for fine-tuning the open domain object detection model.

[0161] Figure 6 FIG. 6 is a schematic structural diagram of an electronic device 600 according to an embodiment of the present invention. Figure 6 As shown, electronic device 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 602 or computer program instructions loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of electronic device 600 can also be stored in RAM 603. CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0162] Multiple components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, a microphone, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, an optical disc, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0163] Each of the processes and treatments described above can be executed by the processing unit 601. For example, the method of any of the above embodiments can be implemented as a computer software program, which is tangibly contained in a computer-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the CPU 601, one or more actions in the method described above can be executed.

[0164] Additionally, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the fine-tuning method of the above-mentioned open-domain object detection model is implemented.

[0165] Each embodiment in this specification is described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other.

[0166] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a device, or a computer program product. Therefore, the embodiments of the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the method, the terminal (system), and the computer program product according to the embodiments of the present application. It should be understood that each process and / or block in the flowcharts and / or block diagrams, and the combination of the processes and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminals to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminals are used to implement the process inFigure 1 means for the functions specified in one or more processes and / or blocks Figure 1 or blocks.

[0168] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement the functions specified in one Figure 1 or more processes and / or blocks Figure 1 or blocks.

[0169] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one Figure 1 or more processes and / or blocks Figure 1 or blocks.

[0170] Although the preferred embodiments of the present application have been described, additional changes and modifications can be made by those skilled in the art once they learn of the basic inventive concept. Therefore, the appended claims are intended to be construed to cover the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0171] Finally, it should also be noted that, in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variation thereof is intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or apparatus. Without further limitation, an element defined by the phrase "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or apparatus that comprises the element.

[0172] The above has introduced in detail a fine-tuning method for an open-domain object detection model, a fine-tuning device for an open-domain object detection model, an electronic device, and a computer-readable storage medium provided by this application. Specific examples are used in this article to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation on this application.

Claims

1. A fine-tuning method for an open-domain object detection model, characterized in that, The method includes: Obtaining a pre-trained open-domain object detection model and object detection training data for object detection in a specified business scenario; Adding a first decoder detection head matching the specified business scenario to the open-domain object detection model to obtain an updated open-domain object detection model, where the first decoder detection head is connected in parallel with a second decoder detection head originally in the open-domain object detection model; Freezing the parameters of other model parts of the updated open-domain object detection model except the first decoder detection head, and training the updated open-domain object detection model based on the object detection training data to obtain a predicted object detection result of the object detection training data output by the updated open-domain object detection model; Calculating a loss value of the updated open-domain object detection model based on the predicted object detection result and the annotation result of the object detection training data; In the case where the loss value indicates that the updated open-domain object detection model has not converged, minimizing the loss value based on the gradient descent method to obtain a model prediction error, and based on the model prediction error, calculating the loss value of the updated open-domain object detection model again until the loss value indicates that the updated open-domain object detection model has converged; In the case where the loss value indicates that the updated open-domain object detection model has converged, obtaining a final open-domain object detection model; The calculating the loss value of the updated open-domain object detection model based on the predicted object detection result and the annotation result of the object detection training data includes: Calculating a GIoU loss value and an L1 loss value based on the predicted bounding box and the corresponding ground truth bounding box; Calculating a contrast loss value based on the predicted class and the corresponding ground truth annotation class; Calculating a weighted average of the GIoU loss value, the L1 loss value, and the contrast loss value, and using the weighted average as the loss value.

2. The method according to claim 1, characterized in that The open-domain object detection model further includes: an image feature extraction module, a text feature extraction module, and a feature fusion module. The training the updated open-domain object detection model based on the object detection training data to obtain a predicted object detection result of the object detection training data output by the updated open-domain object detection model includes: Invoking the image feature extraction module to extract features from the sample image in the object detection training data to obtain image features of the sample image; Invoking the text feature extraction module to extract features from the text in the object detection training data to obtain text features of the text; Invoking the feature fusion module to perform interaction and fusion processing on the image features and the text features to obtain fused features; Invoking the first decoder detection head to process the fused features to obtain the predicted object detection result.

3. The method according to claim 2, wherein The invoking the first decoder detection head to process the fused features to obtain the predicted object detection result includes: Invoke the first decoder detection head to process the fusion features, and obtain the predicted annotation boxes in the sample images in the target detection training data and the predicted categories corresponding to the predicted annotation boxes; Use the predicted annotation boxes and the predicted categories corresponding to the predicted annotation boxes as the predicted target detection results.

4. The method according to claim 1, wherein The recalculating the loss value of the updated open-domain target detection model based on the model prediction error until the loss value indicates that the updated open-domain target detection model converges includes: Backpropagate the model prediction error to the first decoder detection head to update the parameters of the first decoder detection head, and iteratively execute the step of freezing the parameters of other model parts of the updated open-domain target detection model except the first decoder detection head until the loss value of the updated open-domain target detection model is calculated until the updated open-domain target detection model converges.

5. The method according to claim 1, wherein when the specified business scenario is multiple business scenarios, each of the specified business scenarios corresponds to one of the first decoder detection heads; or when the specified business scenario is multiple business scenarios, multiple of the specified business scenarios correspond to one of the first decoder detection heads.

6. A fine-tuning device for an open-domain object detection model, characterized in that, The apparatus includes: A target training data acquisition module, configured to acquire a pre-trained open-domain target detection model and target detection training data for target detection in a specified business scenario; An updated model acquisition module, configured to add a first decoder detection head matching the specified business scenario to the open-domain target detection model to obtain an updated open-domain target detection model, where the first decoder detection head is connected in parallel with the original second decoder detection head in the open-domain target detection model; A predicted detection result acquisition module, configured to freeze the parameters of other model parts of the updated open-domain target detection model except the first decoder detection head, and perform model training on the updated open-domain target detection model based on the target detection training data to obtain the predicted target detection results of the target detection training data output by the updated open-domain target detection model; A loss value calculation module, configured to calculate the loss value of the updated open-domain target detection model based on the predicted target detection results and the annotation results of the target detection training data; A predicted error acquisition module, configured to minimize the loss value based on the gradient descent method to obtain a model prediction error when the loss value indicates that the updated open-domain target detection model has not converged; A target detection model adjustment module, configured to recalculate the loss value of the updated open-domain target detection model based on the model prediction error until the loss value indicates that the updated open-domain target detection model converges; A target detection model acquisition module, configured to obtain a final open-domain target detection model when the loss value indicates that the updated open-domain target detection model has converged; The loss value calculation module includes: The L1 loss calculation unit is configured to calculate a GIoU loss value and an L1 loss value based on the predicted bounding box and the corresponding ground truth bounding box; The contrastive loss calculation unit is configured to calculate a contrastive loss value based on the predicted class and the corresponding ground truth class; The loss value calculation unit calculates a weighted average of the GIoU loss value, the L1 loss value, and the contrastive loss value, and uses this weighted average as the loss value.

7. The device according to claim 6, wherein The open-domain object detection model further includes: an image feature extraction module, a text feature extraction module, and a feature fusion module. The predicted detection result acquisition module includes: An image feature acquisition unit, configured to call the image feature extraction module to extract features from the sample image in the object detection training data, obtaining the image features of the sample image; A text feature acquisition unit, configured to call the text feature extraction module to extract features from the text in the object detection training data, obtaining the text features of the text; A fused feature acquisition unit, configured to call the feature fusion module to perform interaction and fusion processing on the image features and the text features, obtaining fused features; A predicted detection result acquisition unit, configured to call the first decoder detection head to process the fused features, obtaining the predicted object detection result.

8. An electronic device, characterized in that, Including: A processor, a memory, and a computer program stored on the memory and executable on the processor, where when the processor executes the program, it implements the fine-tuning method of the open-domain object detection model according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is enabled to execute the fine-tuning method of the open-domain object detection model according to any one of claims 1 to 5.

Images