An incremental training method for an object detection model, a server, and an object detection system.

Abstract

This application discloses an incremental training method, server, and object detection system for an object detection model. The method includes: training an original OVD visual backbone network, an OVD predefined detection subnetwork, and a language model using a large image description dataset to obtain frozen first network parameters of the OVD visual backbone network; training the original OVD visual backbone network and the incremental subnetwork based on a pre-trained dataset to obtain an updated OVD visual backbone network and a trained incremental subnetwork, using the parameters of the trained incremental subnetwork as second network parameters; loading the frozen first network parameters of the original OVD visual backbone network and the second network parameters of the incremental subnetwork, and fine-tuning the incremental subnetwork using gradient backpropagation based on user incremental data to obtain third network parameters of the incremental subnetwork; and generating a fusion network of the OVD predefined detection subnetwork and the incremental subnetwork corresponding to the third network parameters as an object detection model.

Description

Technical Field

[0001] This application relates to the field of image detection technology, and in particular to an incremental training method for an object detection model, a server, and an object detection system. Background Technology

[0002] With the development of object detection technology, object recognition in many images relies on deep learning models. However, when using deep learning models to identify objects that differ significantly from the training samples, the accuracy is often low, meaning that the detection effect on new categories of objects is not good. Summary of the Invention

[0003] This application provides an incremental training method for object detection models to address the problem of poor detection performance of deep learning models for new object categories.

[0004] In a first aspect, embodiments of this application provide an incremental training method for an object detection model, including:

[0005] The object detection model includes a language model, an Open Vocabulary Detection (OVD) visual backbone network, an OVD predefined detection subnetwork, and an incremental subnetwork. The language model is obtained through pre-training on text samples.

[0006] The original OVD visual backbone network, the OVD predefined detection subnetwork, and the language model are trained using the Image Caption large dataset. The original OVD visual backbone network, the language model, and the OVD predefined detection subnetwork are then frozen to obtain the first network parameters of the frozen OVD visual backbone network.

[0007] The original OVD visual backbone network and incremental sub-network are trained based on the pre-trained dataset to obtain the updated OVD visual backbone network and the trained incremental sub-network. The parameters of the trained incremental sub-network are used as the second network parameters.

[0008] Load the first network parameters of the frozen original OVD visual backbone network and the second network parameters of the incremental sub-network, and fine-tune the incremental sub-network based on user incremental data using gradient backpropagation to obtain the third network parameters of the incremental sub-network.

[0009] A fusion network is generated, consisting of the OVD predefined detection subnetwork and the incremental subnetwork corresponding to the third network parameters, as the target detection model.

[0010] Optionally, before loading the first network parameters of the frozen original OVD visual backbone network and the second network parameters of the incremental subnetwork, the method further includes:

[0011] Obtain the Zero-shot capability evaluation result of the frozen OVD predefined detection subnetwork, and based on the Zero-shot capability evaluation result, determine the organization of user incremental data in the training scheme of the incremental subnetwork.

[0012] Optionally, based on incremental user data, the incremental sub-network is fine-tuned using gradient backpropagation to obtain the third network parameters of the incremental sub-network, including:

[0013] The images from the user's incremental data are input into the frozen original OVD visual backbone network and the incremental subnetwork to obtain visual features;

[0014] The language model is used to transform the categorical text in the incremental user data into text features.

[0015] Based on the incremental subnetwork, the visual features and the text features are interactively calculated to obtain the prediction box;

[0016] Based on the loss between the predicted bounding box and the true annotations in the user incremental data, gradient backpropagation optimizes the incremental sub-network to obtain the third network parameters.

[0017] Optionally, the method further includes:

[0018] Based on the Zero-shot capability evaluation results, the organization of user incremental data in the training scheme of the incremental sub-network is determined, including:

[0019] Based on the Zero-shot capability assessment results, determine the long-tail categories and / or emerging categories in user needs relative to the Image Caption large dataset;

[0020] By combining the long-tail categories and / or emerging categories, corresponding calibration data is collected as the incremental user data.

[0021] Optionally, a fusion network is generated, consisting of the OVD predefined detection subnetwork and the incremental subnetwork corresponding to the third network parameters, as the target detection model. The method further includes:

[0022] Based on the incremental user data, obtain the Zero-shot capability evaluation result of the OVD predefined detection subnetwork corresponding to the current category of the incremental user data.

[0023] If the Zero-shot capability assessment result is greater than the satisfaction threshold, then the predicted value of the current category by the OVD predefined detection subnetwork is selected as the output detection result.

[0024] If the Zero-shot capability assessment result of the current category is less than the basic capability threshold, the prediction value of the current category by the incremental sub-network corresponding to the third network parameter is selected as the output detection result.

[0025] If the Zero-shot capability assessment result is greater than the basic capability threshold and less than the satisfaction threshold, the weighted fusion of the predicted values ​​of the current category by the OVD predefined detection subnetwork and the incremental subnetwork corresponding to the third network parameters is selected as the output detection result, wherein the satisfaction threshold is greater than the basic capability threshold and less than 1.

[0026] Secondly, embodiments of this application provide a target detection method based on a visual model, wherein the target detection model is trained using the above method; the target detection model includes: an OVD predefined detection subnetwork and an incremental subnetwork;

[0027] The target to be detected is input into the target detection model, and forward inference is performed based on the OVD predefined detection subnetwork and the incremental subnetwork to obtain at least two first-class prediction results.

[0028] The at least two first-category prediction results are sorted according to their confidence levels to obtain a first list;

[0029] The first prediction result with the highest confidence in the first list is used as the query box, and the IoU is calculated with other first category prediction results.

[0030] The prediction results in the first list whose IoU is greater than the IoU threshold are used as the second list. The second prediction result with the highest confidence in the second list is used as the matching box. The confidence of the matching box and the query box are weighted and fused to obtain the confidence of the target detection result. The coordinates of the query box are output as the coordinates of the target detection result. The first prediction result and the second list are deleted.

[0031] If there is no prediction result with an IoU greater than the IoU threshold in the first list, then the first prediction result is output as the target detection result and the first prediction result is deleted.

[0032] Optionally, the method further includes:

[0033] After deleting the first prediction result, update the first list;

[0034] Return to the step of using the first prediction result with the highest confidence in the first list as the query box, until all prediction results in the first list are deleted.

[0035] Optionally, the method further includes:

[0036] After all the first category prediction results in the first list are deleted, at least two second category prediction results obtained by forward inference from the OVD predefined detection subnetwork and the incremental subnetwork are sorted according to their confidence to obtain the first list;

[0037] Return to the step of using the first prediction result with the highest confidence in the first list as the query box and calculating the IoU with other first category prediction results, until the prediction results of all categories performed by the OVD predefined detection subnetwork and the incremental subnetwork for forward inference have been processed.

[0038] Thirdly, embodiments of this application provide a server, the server including a cloud processor, the cloud processor executing computer-readable code to implement the above-described incremental training method for the target detection model, so as to obtain the target detection model.

[0039] Fourthly, embodiments of this application provide a target detection system, the system comprising: a cloud processor and a user terminal.

[0040] The cloud processor is used to train an object detection model based on the above method;

[0041] The user terminal is used to deploy the trained object detection model, and the user terminal also includes a display screen:

[0042] The display screen is used to display the detection results in the image to be detected, and the detection results include at least one of the following: predicted bounding box, target category, coordinates, and confidence score.

[0043] This application embodiment freezes the original OVD visual backbone network, the language model, and the OVD predefined detection subnetwork, backing up the global parameters of the target detection model. This avoids catastrophic forgetting of the OVD predefined detection subnetwork. The incremental subnetwork addresses the need for category increments, and the fusion network of the OVD predefined detection subnetwork and the incremental subnetwork can meet the needs for data distribution increments. This allows the target detection model to retain general knowledge while having strong open set detection capabilities. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the incremental training method for the target detection model provided in the embodiments of this application;

[0045] Figure 2 A schematic diagram of the modules and training process of the target detection model provided in the embodiments of this application;

[0046] Figure 3 This is a schematic diagram of the incremental training process provided in an embodiment of this application;

[0047] Figure 4 A schematic diagram and flowchart of the forward inference process of the target detection model provided in the embodiments of this application. Detailed Implementation

[0048] The present application will be described in detail below with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present application. Any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the protection scope of the present application.

[0049] OVD (Open-Vocabulary Object Detection) is a technique that detects objects in an image that do not belong to a predefined, fixed category. Since the objects in an OVD task are not predefined in the training set, they need to be located and identified during inference based on natural language descriptions or an open-vocabulary dataset. The training set for OVD can use data from the visible class (base class). The unpredictable objects that need to be detected by OVD are called data from the unseen class, or target data.

[0050] OVD (On-Vehicle Visualization) has broad application prospects in scenarios involving unpredictable objects. For example, in the field of autonomous driving, vehicles need to identify new types of obstacles and respond accordingly. In the field of medical image analysis, medical systems need to identify new case features in reports. And in the field of intelligent robotics, intelligent robots may need to identify new types of objects and perform corresponding actions.

[0051] Because of its high flexibility and scalability, OVD allows users to define the categories and corresponding descriptive terms of new objects according to their application scenarios and needs. Users can define these new objects and corresponding descriptive terms without manually modifying the internal structure or parameters of OVD. However, this also means that OVD technology faces technical challenges such as data scarcity, uneven data distribution, inaccurate location of unpredictable objects, inaccurate description of cross-semantic granularity terms, and high complexity of visual-language matching. These technical challenges also limit the performance of OVD technology in practical applications.

[0052] To address the aforementioned issues, this application provides a training method for a visual model based on the OVD visual backbone network. The OVD visual backbone network is pre-trained using a large ImageCaption dataset. The ImageCaption dataset can be a collection of massive image data of the visible class (e.g., a massive image dataset with a scale of tens of millions).

[0053] In one optional embodiment of this application, the visual model may include an Open Vocabulary Object Detection (OVD) visual backbone network and an incremental subnetwork. In this way, the visual model can learn the proprietary knowledge of the incremental sample data provided by the user, while retaining the general knowledge learned during pre-training on a large-scale dataset.

[0054] See Figure 1 As shown, the visual model provided in this application embodiment includes an OVD visual backbone network, an OVD predefined detection subnetwork, and an incremental subnetwork.

[0055] The model training method provided in this application includes the following steps:

[0056] S101: Pre-train the original OVD visual backbone network, the OVD predefined detection subnetwork, and the language model using the Image Caption large dataset, freeze the original OVD visual backbone network, the language model, and the OVD predefined detection subnetwork, and obtain the first network parameters of the frozen OVD visual backbone network.

[0057] S102: Train the original OVD visual backbone network and incremental sub-network based on the pre-trained dataset to obtain the updated OVD visual backbone network and the trained incremental sub-network, and use the parameters of the trained incremental sub-network as the second network parameters.

[0058] S103: Load the first network parameters of the frozen original OVD visual backbone network and the second network parameters of the incremental sub-network, and fine-tune the incremental sub-network based on user incremental data by gradient backpropagation to obtain the third network parameters of the incremental sub-network.

[0059] S104: Generate a fusion network of the OVD predefined detection subnetwork and the incremental subnetwork corresponding to the third network parameters as the target detection model.

[0060] If the original network structure of OVD remains unchanged, and methods such as data replay, model regularization, and knowledge distillation are used to constrain OVD and retain the model's memory of old knowledge from the pre-training stage, there will still be some loss of old knowledge.

[0061] Therefore, in this embodiment of the application, the original OVD visual backbone network, the language model, and the OVD predefined detection subnetwork are frozen, the global parameters of the target detection model are backed up, and then the OVD visual backbone network and the incremental subnetwork are trained based on the pre-trained dataset in step S102. In S102, the OVD predefined detection subnetwork and the language model are still frozen, and the parameters of the trained incremental subnetwork are used as the second network parameters W2.

[0062] In this way, freezing the original OVD visual backbone network, the language model, and the OVD predefined detection subnetwork, and backing up the global parameters of the target detection model, can prevent catastrophic forgetting of the OVD predefined detection subnetwork. The incremental subnetwork can be used to address the need for category increments, and the fusion network of the OVD predefined detection subnetwork and the incremental subnetwork can meet the needs for data distribution increments. This allows the target detection model to retain general knowledge while having strong open set detection capabilities.

[0063] See Figure 2 As shown, the object detection model can include an OVD predefined module, an incremental module, and a fusion module. In the OVD predefined module, text from the pre-training dataset is input into the language model, which transforms the categorical text in the incremental user data into text features. Images from the pre-training dataset are input into the OVD visual backbone network and then processed by the incremental sub-network to obtain visual features. Based on the incremental sub-network, the visual features and text features are interactively computed to obtain the predicted bounding box (Bbox). During the training of the OVD predefined module on the pre-training dataset, the language model, the OVD visual backbone network, and the OVD predefined detection sub-network are all in a frozen state.

[0064] In the incremental module, the output of the OVD visual backbone network is input into the incremental sub-network to obtain visual features. The visual features output by the incremental sub-network and the text features output by the language model are interactively calculated to obtain the predicted bounding box (Bbox), class, and confidence score.

[0065] In the fusion module, an ensemble decision is performed on the predicted bounding box coordinates, class, and score output by the predefined detection subnetwork and incremental subnetwork of OVD to generate the output predicted bounding box coordinates, class, and score.

[0066] Optionally, in this embodiment of the application, before loading the first network parameters of the frozen original OVD visual backbone network and the second network parameters of the incremental sub-network, the zero-shot capability evaluation result of the frozen OVD predefined detection sub-network is obtained, and based on the zero-shot capability evaluation result, the organization of user incremental data in the training scheme of the incremental sub-network is determined.

[0067] Therefore, the aforementioned incremental module can determine the long-tail categories and / or emerging categories in user needs relative to the Image Caption large dataset based on the Zero-shot capability evaluation results;

[0068] By combining the long-tail categories and / or emerging categories, corresponding calibration data is collected as the incremental user data.

[0069] For example, for tail categories and tail scenarios not covered by the OVD predefined modules, the task size is significantly smaller than the OVD predefined scope.

[0070] See Figure 3 As shown, the incremental training process of the incremental module may include the following steps:

[0071] After inputting the image from the user's incremental data into the frozen original OVD visual backbone network, it is then input into the incremental subnetwork and the OVD predefined detection subnetwork to obtain two visual features.

[0072] The language model is used to transform the categorical text in the incremental user data into text features.

[0073] The incremental subnetwork is used to perform interactive calculations on each visual feature and text feature to obtain a prediction box.

[0074] The loss between the predicted bounding box output by the incremental subnetwork and the ground truth annotation in the user's incremental data is used as a gradient backpropagation method to optimize the incremental subnetwork, resulting in the third network parameters, which may include:

[0075] The predicted bounding boxes output by the incremental subnetwork are processed using a weighted matrix.

[0076] Figure 3 The detection results output by the OVD predefined detection subnetwork are compared with the labeled ground truth. The sample difficulty coefficient of the image in the user incremental data relative to the OVD predefined detection subnetwork is calculated, and a weighted coefficient matrix is ​​generated.

[0077] The formula for generating the weighted coefficient matrix can be as follows;

[0078] difficulty = 1 - target_porbs;

[0079] Where target_porbs represents the prediction confidence of the OVD predefined detection subnetwork, and Difficulty represents the sample difficulty coefficient;

[0080] dynamic_weights = alpha * (1+ difficulty),

[0081] Where dynamic_weights are the dynamic weights, and alpha is an adjustable coefficient, with the default alpha value being 1.0.

[0082] Figure 3 The predicted candidate regions of the incremental subnetwork are compared with the labeled ground truth to obtain the loss function of the incremental subnetwork. Then, the loss function is multiplied by the weighted coefficient matrix to obtain the final loss function.

[0083] cls_loss = FocalLoss(inc_target_porbs)

[0084] final_cls_loss = dynamic_weights * cls_loss

[0085] Where `inc_target_porbs` represents the prediction confidence of the incremental sub-network, `cls_loss` represents the classification loss of the incremental sub-network, `final_cls_loss` represents the final loss function, `dynamic_weights` represents the dynamic weights, and `FocalLoss` represents the improved loss function based on cross-entropy. For the incremental categories of user incremental data, the following two cases can be set:

[0086] When the number of incremental categories in the incremental user data is 0 (all incremental user data are non-emerging categories or long-tail categories), the prediction results of the OVD predefined detection subnetwork are directly adopted and are not included in the Zero-shot capability evaluation.

[0087] When the number of users belonging to the incremental category in the incremental data is greater than 0 (there are emerging categories or long-tail categories), the Zero-shot capability evaluation result of the OVD predefined detection subnetwork corresponding to the current category is compared with the basic capability threshold.

[0088] The OVD predefined detection subnetwork corresponds to case 1: if the Zero-shot capability assessment result is greater than the satisfaction threshold, then the predicted value of the OVD predefined detection subnetwork for the current category is selected as the output detection result.

[0089] The OVD predefined detection subnetwork corresponds to case 2: the Zero-shot capability assessment result is less than the basic capability threshold, and the prediction value of the incremental subnetwork corresponding to the third network parameter for the current category is selected as the output detection result.

[0090] The OVD predefined detection subnetwork corresponds to case 3: If the Zero-shot capability assessment result is greater than the basic capability threshold and less than the satisfaction threshold, the weighted fusion of the predicted values ​​of the current category by the OVD predefined detection subnetwork and the incremental subnetwork corresponding to the third network parameters is selected as the output detection result, wherein the satisfaction threshold is greater than the basic capability threshold and less than 1.

[0091] Referring to Table 1, after generating the fusion network of the OVD predefined detection subnetwork and the incremental subnetwork corresponding to the parameters of the third network as the target detection model, the training method may further include the following steps to implement the dual-branch best matching weighted target fusion strategy of the fusion module:

[0092] Based on the incremental user data, obtain the Zero-shot capability evaluation result of the OVD predefined detection subnetwork corresponding to the current category of the incremental user data.

[0093] If the Zero-shot capability assessment result is greater than the satisfaction threshold, then the predicted value of the current category by the OVD predefined detection subnetwork is selected as the output detection result.

[0094] If the Zero-shot capability assessment result of the current category is less than the basic capability threshold, the prediction value of the current category by the incremental sub-network corresponding to the third network parameter is selected as the output detection result.

[0095] If the Zero-shot capability assessment result is greater than the basic capability threshold and less than the satisfaction threshold, the weighted fusion of the predicted values ​​of the current category by the OVD predefined detection subnetwork and the incremental subnetwork corresponding to the third network parameters is selected as the output detection result, wherein the satisfaction threshold is greater than the basic capability threshold and less than 1.

[0096] For example, the three possibilities mentioned above can be adapted to the following scenarios as shown in the table below.

[0097]

[0098] Table 1

[0099] Optionally, the incremental subnetwork can be fine-tuned with a small learning rate of 0.1*A in the above steps.

[0100] In this embodiment of the application, after the target detection model is trained based on the above training method, target detection can be performed in, but is not limited to, the following ways:

[0101] like Figure 4 As shown, the target detection model includes: an OVD predefined detection subnetwork and an incremental subnetwork;

[0102] S401: Input the target to be detected into the target detection model, and perform forward inference based on the OVD predefined detection subnetwork and the incremental subnetwork respectively to obtain at least two first-class prediction results;

[0103] S402: Sort the at least two first-category prediction results according to their confidence levels to obtain a first list;

[0104] S403: Use the first prediction result with the highest confidence in the first list as the query box, and calculate the IoU with other first category prediction results.

[0105] For example, IoU (Intersection over Union), also known as the Jaccard index, is a metric that measures the degree of overlap between two bounding boxes (BBoxes) or segmented regions. For instance, the IoU between a query box and other first-class predictions. Based on the co-detection capability represented by the IoU of the two boxes, the decision is made to either perform S404 to obtain a weighted fusion result, or directly output the prediction with the highest confidence in S405.

[0106] S404: Take the prediction results of IoU greater than IoU threshold in the first list as the second list, take the second prediction result with the highest confidence in the second list as the matching box, perform weighted fusion of the confidence of the matching box and the query box as the confidence of the target detection result, output the coordinates of the query box as the coordinates of the target detection result, and delete the first prediction result and the second list.

[0107] S405: If there is no prediction result with IoU greater than the IoU threshold in the first list, then the first prediction result is output as the target detection result and the first prediction result is deleted.

[0108] After deleting the first prediction result, the target detection method may further include the following intra-class small loop process:

[0109] S406: Update the first list;

[0110] Return to step S403 until all prediction results in the first list are deleted.

[0111] This object detection method can also include the following large loop processes for different classifications:

[0112] S407: After all the first category prediction results in the first list are deleted, the at least two second category prediction results obtained by forward inference from the OVD predefined detection subnetwork and the incremental subnetwork respectively are sorted according to their confidence to obtain the first list;

[0113] Return to S403 until the prediction results of all categories for forward inference performed by the OVD predefined detection subnetwork and the incremental subnetwork have been processed.

[0114] In step S405, the confidence level of the first prediction result can be weighted and processed to serve as the final confidence level output of the predicted bounding box in the target detection result.

[0115] In this embodiment, the OVD visual backbone network is trained using the Image Caption large dataset to achieve feature detection of open words. Then, by back-feeding the classification loss and localization loss respectively, an incremental sub-network with more accurate classification and localization capabilities is obtained. When the OVD predefined detection sub-network and the incremental sub-network can perform co-detection, it is determined to add an incremental sub-network to the OVD visual backbone network, thereby achieving a more accurate detection effect for open words.

[0116] In an optional embodiment of this application, for the same incremental image to be detected, multiple prediction boxes with different confidence levels output by the OVD predefined detection subnetwork are stored in a third list, and multiple prediction boxes with different confidence levels output by the incremental subnetwork are stored in a fourth list.

[0117] Using the prediction box with the highest confidence in the third or fourth list as the query box, iterate through the prediction boxes in different lists that correspond to the query box, and calculate the inter-box IoU (IoU) between them and the query box:

[0118] If the inter-frame IOU is greater than the matching threshold, it is determined that the OVD predefined detection subnetwork and the incremental subnetwork have co-detection capability.

[0119] If the inter-frame IOU is less than the matching threshold, the query box is deleted; the process returns to using the prediction box with the highest confidence in the third or fourth list as the query box, traversing the prediction boxes in different lists from the query box, and calculating their inter-frame IOU with the query box, until all prediction boxes in the third or fourth list are deleted as query boxes, then it is determined that the OVD predefined detection subnetwork and the incremental subnetwork do not have co-detection capability.

[0120] If the OVD predefined detection subnetwork and incremental subnetwork do not have co-detection capability.

[0121] After training the incremental subnetwork, the target detection model in this embodiment can delete the language model, retaining only the OVD visual backbone network, the OVD predefined detection subnetwork, and the incremental subnetwork. When identifying newly added types of targets in an image, the image to be identified can be input into the OVD visual backbone network to obtain the visual features and open vocabulary of the image. The visual features and open vocabulary are then input into the OVD predefined detection subnetwork and the incremental subnetwork to obtain the predicted bounding boxes output by the OVD predefined detection subnetwork and the incremental subnetwork. The predicted bounding boxes of the OVD predefined detection subnetwork are placed in the third list, and the predicted bounding boxes of the incremental subnetwork are placed in the fourth list.

[0122] Step 1: Take the prediction box with the highest confidence in the two lists as the query box Q-Bbox, and iterate through the IoU values ​​of other prediction boxes belonging to different lists.

[0123] Step 2: If the IoU between Q-Bbox and other predicted boxes is greater than the IoU threshold, it is considered that there are matching predicted boxes in branch 1 (OVD predefined detection sub-network) and branch 2 (incremental sub-network) (duplicate predicted boxes in the same list have been filtered in the NMS nonmaximum suppression stage). At this time, the query box Q-Bbox and the predicted box BM-Bbox with the highest confidence among other predicted boxes belonging to different lists are taken out. The confidence and coordinates of Q-Bbox and BM-Bbox are weighted and fused as the fusion result. Q-Bbox and BM-Bbox are then removed from the two lists.

[0124] Step 3: If there are no cases where the IoU between the Q-Bbox and other predicted boxes is greater than the IoU threshold, then the predicted box is considered to be a detection capability unique to branch 1 or branch 2. Branch 1 and branch 2 do not have co-detection capabilities for the corresponding target. The confidence score of the Q-Bbox can be directly multiplied by the weighting coefficient to obtain the output confidence score, and then the Q-Bbox is removed from its list. Return to step 1 until the fourth list is empty.

[0125] Step 4: For each category's prediction bounding box, repeat steps 1 to 3 until the prediction results of all categories, which are forward inferenced by the OVD predefined detection subnetwork and the incremental subnetwork respectively, are processed, thus completing the target detection function for all images to be detected.

[0126] This application embodiment also provides a server including a cloud processor, which executes computer-readable code to implement the above-described incremental training method for the target detection model, so as to obtain the target detection model.

[0127] This application also provides a target detection system, including: a cloud processor and a user terminal.

[0128] The cloud processor is used to train the target detection model based on the above-mentioned incremental training method for the target detection model.

[0129] The user terminal is used to deploy the trained object detection model, and the user terminal also includes a display screen:

[0130] The display screen is used to display the detection results in the image to be detected, and the detection results include at least one of the following: predicted bounding box, target category, coordinates, and confidence score.

[0131] In this embodiment, the OVD visual backbone network is trained using the Image Caption large dataset to achieve feature detection of open words. Based on the frozen OVD, targeted incremental learning is performed with a small learning rate to obtain a predefined OVD detection subnetwork with more accurate detection capabilities. Then, by backpropagating the classification loss and localization loss respectively, an incremental subnetwork with more accurate classification and localization capabilities is obtained. When the predefined OVD detection subnetwork and the incremental subnetwork can perform co-detection, it is determined to add an incremental subnetwork to the OVD visual backbone network, thereby achieving a more accurate detection effect for open words.

[0132] Based on the same inventive concept, this application provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method for detecting content played on a display screen.

[0133] This application provides an incremental training device for an object detection model. The object detection model includes a language model, an Open Vocabulary Detection (OVD) visual backbone network, an OVD predefined detection subnetwork, and an incremental subnetwork. The language model is obtained through pre-training with text samples.

[0134] The incremental training device for the object detection model includes:

[0135] The first training module is used to train the original OVD visual backbone network, the OVD predefined detection subnetwork and the language model using the Image Caption large dataset, freeze the original OVD visual backbone network, the language model and the OVD predefined detection subnetwork, and obtain the first network parameters of the frozen OVD visual backbone network.

[0136] The second training module is used to train the original OVD visual backbone network and incremental sub-network based on the pre-training dataset to obtain the updated OVD visual backbone network and the trained incremental sub-network, and to use the parameters of the trained incremental sub-network as the second network parameters.

[0137] The fine-tuning module is used to load the first network parameters of the frozen original OVD visual backbone network and the second network parameters of the incremental sub-network, and fine-tune the incremental sub-network based on user incremental data in the manner of gradient backpropagation to obtain the third network parameters of the incremental sub-network.

[0138] The generation module is used to generate a fusion network of the OVD predefined detection subnetwork and the incremental subnetwork corresponding to the third network parameters as a target detection model.

[0139] The fine-tuning module is also used for:

[0140] Obtain the Zero-shot capability evaluation result of the frozen OVD predefined detection subnetwork, and based on the Zero-shot capability evaluation result, determine the organization of user incremental data in the training scheme of the incremental subnetwork.

[0141] The fine-tuning module is also used for:

[0142] The images from the user's incremental data are input into the frozen original OVD visual backbone network and the incremental subnetwork to obtain visual features;

[0143] The language model is used to transform the categorical text in the incremental user data into text features.

[0144] Based on the incremental subnetwork, the visual features and the text features are interactively calculated to obtain the prediction box;

[0145] Based on the loss between the predicted bounding box and the true annotations in the user incremental data, gradient backpropagation optimizes the incremental sub-network to obtain the third network parameters.

[0146] The fine-tuning module is also used for:

[0147] Based on the Zero-shot capability evaluation results, the organization of user incremental data in the training scheme of the incremental sub-network is determined, including:

[0148] Based on the Zero-shot capability assessment results, determine the long-tail categories and / or emerging categories in user needs relative to the Image Caption large dataset;

[0149] By combining the long-tail categories and / or emerging categories, corresponding calibration data is collected as the incremental user data.

[0150] The generation module is also used for:

[0151] Based on the incremental user data, obtain the Zero-shot capability evaluation result of the OVD predefined detection subnetwork corresponding to the current category of the incremental user data.

[0152] If the Zero-shot capability assessment result is greater than the satisfaction threshold, then the predicted value of the current category by the OVD predefined detection subnetwork is selected as the output detection result.

[0153] If the Zero-shot capability assessment result of the current category is less than the basic capability threshold, the prediction value of the current category by the incremental sub-network corresponding to the third network parameter is selected as the output detection result.

[0154] If the Zero-shot capability assessment result is greater than the basic capability threshold and less than the satisfaction threshold, the weighted fusion of the predicted values ​​of the current category by the OVD predefined detection subnetwork and the incremental subnetwork corresponding to the third network parameters is selected as the output detection result, wherein the satisfaction threshold is greater than the basic capability threshold and less than 1.

[0155] In this embodiment of the application, after training the target detection video model as described above, the target detection model can be deployed to the user terminal. The target detection model may include: an OVD predefined detection subnetwork and an incremental subnetwork; the user terminal can use the trained target detection model to perform the following target detection methods:

[0156] The target to be detected is input into the target detection model, and forward inference is performed based on the OVD predefined detection subnetwork and the incremental subnetwork to obtain at least two first-class prediction results.

[0157] The at least two first-category prediction results are sorted according to their confidence levels to obtain a first list;

[0158] The first prediction result with the highest confidence in the first list is used as the query box, and the IoU is calculated with other first category prediction results.

[0159] The prediction results in the first list whose IoU is greater than the IoU threshold are used as the second list. The second prediction result with the highest confidence in the second list is used as the matching box. The confidence of the matching box and the query box are weighted and fused to obtain the confidence of the target detection result. The coordinates of the query box are output as the coordinates of the target detection result. The first prediction result and the second list are deleted.

[0160] If there is no prediction result with an IoU greater than the IoU threshold in the first list, then the first prediction result is output as the target detection result and the first prediction result is deleted.

[0161] Optionally, the method further includes:

[0162] After deleting the first prediction result, update the first list;

[0163] Return to the step of using the first prediction result with the highest confidence in the first list as the query box, until all prediction results in the first list are deleted.

[0164] Optionally, the method further includes:

[0165] After all the first category prediction results in the first list are deleted, at least two second category prediction results obtained by forward inference from the OVD predefined detection subnetwork and the incremental subnetwork are sorted according to their confidence to obtain the first list;

[0166] Return to the step of using the first prediction result with the highest confidence in the first list as the query box and calculating the IoU with other first category prediction results, until the prediction results of all categories performed by the OVD predefined detection subnetwork and the incremental subnetwork for forward inference have been processed.

[0167] This application provides a display screen, including:

[0168] The display and processor are provided, wherein the processor is used to execute the above method, and the display is used to display the above image and the prediction box output by the above visual model.

[0169] In optional embodiments of this application, the display screen may include, but is not limited to, components such as a processor, touch panel, voice input device, speaker, and monitor.

[0170] The memory can be random access memory, read-only memory, non-volatile, programmable ROM, erasable PROM, electrically erasable memory, flash memory, optical memory, and registers, etc. The processor can be a general-purpose processor, which is a processor that performs specific steps and / or operations by reading and executing computer programs stored in the memory. The general-purpose processor may use the memory during the execution of these steps and / or operations. The general-purpose processor can be a central processing unit, ASIC, and FPGA, etc. In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor.

[0171] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0172] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0173] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The above descriptions are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for incrementally training a target detection model, characterized in that, The method comprises: The target detection model comprises a language model, an original open vocabulary target detection (OVD) visual stem network, an OVD predefined detection subnetwork, and an incremental subnetwork, the language model is obtained by pre-training a text sample, The original OVD visual stem network, the OVD predefined detection subnetwork, and the language model are trained through an image description (Image Caption) big data set, the original OVD visual stem network, the language model, and the OVD predefined detection subnetwork are frozen, and first network parameters of the frozen OVD visual stem network are obtained; The original OVD visual stem network and the incremental subnetwork are trained based on a pre-training data set, and updated OVD visual stem network and trained incremental subnetwork are obtained, and parameters of the trained incremental subnetwork are used as second network parameters; The first network parameters of the frozen original OVD visual stem network and the second network parameters of the incremental subnetwork are loaded, the incremental subnetwork is fine-tuned in a gradient backpropagation manner based on user incremental data, and third network parameters of the incremental subnetwork are obtained; A fusion network of the OVD predefined detection subnetwork and the incremental subnetwork corresponding to the third network parameters is generated as a target detection model.

2. The method of claim 1, wherein, Before the first network parameters of the frozen original OVD visual stem network and the second network parameters of the incremental subnetwork are loaded, the method further comprises: Zero-shot capability evaluation results of the frozen OVD predefined detection subnetwork are obtained, and based on the Zero-shot capability evaluation results, an organization of user incremental data in a training scheme of the incremental subnetwork is determined.

3. The method of claim 2, wherein, The incremental subnetwork is fine-tuned in a gradient backpropagation manner based on user incremental data, and third network parameters of the incremental subnetwork are obtained, comprising: Images in the user incremental data are input into the frozen original OVD visual stem network and the incremental subnetwork to obtain visual features; Class text in the user incremental data is converted into text features by the language model; The visual features and the text features are interactively calculated based on the incremental subnetwork to obtain a prediction box; The incremental subnetwork is optimized in a gradient backpropagation manner based on a loss between the prediction box and real labels in the user incremental data, and third network parameters are obtained.

4. The method of claim 2, wherein, The method further comprises: Based on the Zero-shot capability evaluation results, an organization of user incremental data in a training scheme of the incremental subnetwork is determined, comprising: Based on the Zero-shot capability evaluation results, long-tail categories and / or emerging categories relative to the Image Caption big data set in user demand are determined; In combination with the long-tail categories and / or emerging categories, corresponding labeled data is collected as the user incremental data.

5. The method of claim 1, wherein, The method further comprises: According to the user incremental data, obtain a Zero-shot capability evaluation result of an OVD predefined detection sub-network corresponding to a current category of the user incremental data, If the Zero-shot capability evaluation result is greater than a satisfaction threshold, select a prediction value of the current category of the OVD predefined detection sub-network as a detection result of output; If the Zero-shot capability evaluation result of the current category is less than a basic capability threshold, select a prediction value of the current category of the incremental sub-network corresponding to the third network parameter as the detection result of output; If the Zero-shot capability evaluation result is greater than the basic capability threshold and less than the satisfaction threshold, select a weighted fusion of prediction values of the current category of the OVD predefined detection sub-network and the incremental sub-network corresponding to the third network parameter as the detection result of output, wherein the satisfaction threshold is greater than the basic capability threshold and less than 1. 6.A target detection method based on a target detection model, characterized in that, The target detection model is trained by the method in any one of claims 1-5; the target detection model comprises: an OVD predefined detection sub-network and an incremental sub-network; input a target to be detected into the target detection model, and perform forward reasoning based on the OVD predefined detection sub-network and the incremental sub-network respectively to obtain at least two first category prediction results; sort the at least two first category prediction results according to confidence to obtain a first list; take a first prediction result with the highest confidence in the first list as a query box, and calculate an IoU with other first category prediction results; take a prediction result with an IoU greater than an IoU threshold in the first list as a second list, take a second prediction result with the highest confidence in the second list as a matching box, and perform weighted fusion on confidences of the matching box and the query box to obtain a confidence of a target detection result, output coordinates of the query box as coordinates of the target detection result, and delete the first prediction result and the second list; if there is no prediction result with an IoU greater than the IoU threshold in the first list, take the first prediction result as the target detection result, and delete the first prediction result.

7. The method of claim 6, wherein, The method further comprises: after deleting the first prediction result, update the first list; return to the step of taking a first prediction result with the highest confidence in the first list as a query box until all prediction results in the first list are deleted.

8. The method of claim 7, wherein, The method further comprises: after all first category prediction results in the first list are deleted, sort at least two second category prediction results obtained by performing forward reasoning by the OVD predefined detection sub-network and the incremental sub-network respectively according to confidence to obtain a first list; return to the step of taking a first prediction result with the highest confidence in the first list as a query box, and calculating an IoU with other first category prediction results until all category prediction results obtained by performing forward reasoning by the OVD predefined detection sub-network and the incremental sub-network are processed.

9. A server, characterized by The server comprises a cloud processor, which executes computer readable code to implement the target detection model incremental training method in claim 1 to obtain the target detection model.

10. A target detection system characterized by, The system comprises a cloud processor and a user terminal, The cloud processor is configured to train a target detection model based on the method in claim 1. The user terminal is configured to deploy the trained target detection model, and the user terminal further comprises a display screen. The display screen is configured to display a detection result in a to-be-detected image, and the detection result comprises at least one of a prediction box, a target category, coordinates, and a confidence level.

Images