Target detection training method, target detection method, device, equipment and medium

Abstract

The application belongs to the technical field of image processing, and particularly relates to a target detection training method, a target detection method, a device, equipment and a medium. The method of the application trains a to-be-trained model by using second type data not belonging to a category corresponding to a classification task of the to-be-trained model, such as inaccurate classification labeling or no specific classification labeling, and first type data with accurate classification, and can automatically distinguish the loss calculation modes of the first type data and the second type data in the training process, so as to accurately and effectively realize the training of the to-be-trained model. Compared with using only the first type data, the trained target detection model has more accurate target detection effect, so as to provide a good basis for the classification of detected targets, thereby improving the accuracy of target classification.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a target detection training method, target detection method, apparatus, device and medium. Background Technology

[0002] Currently, deep learning-based object detection has gradually become a core technology in fields such as autonomous driving, video surveillance, machining, and intelligent robots. Data and its volume are of paramount importance in deep learning. However, in some application scenarios, due to annotation issues, some of the massive amounts of data are labeled with specific categories (i.e., class-aware), while other data are only labeled with foreground categories, meaning that all categories are classified as a single foreground element (i.e., class-agnostic).

[0003] Class-aware data is meaningful for training the classification ability of object detection models, while class-agnostic data cannot help with the classification of object detection models. Class-agnostic data contains bounding boxes but does not have specific classifications of the bounding boxes. It can be used to some extent to train the model's ability to locate bounding boxes.

[0004] Existing models trained with class-aware data have a certain ability to detect targets and classify them. However, their training is mainly used to improve the accuracy of classification. Since the target is the basis for classification, the effect of using class-aware data on improving the accuracy of the target detection is limited.

[0005] Therefore, how to make reasonable use of the characteristics of known data to optimize the accuracy of the model in detecting targets, and combine classification detection to improve the model's classification ability while improving its detection ability, has become an urgent problem to be solved. Summary of the Invention

[0006] In view of this, embodiments of this application provide a target detection training method, target detection method, apparatus, device, and medium to address the problem of how to reasonably utilize the characteristics of known data to optimize the accuracy of target detection by the model, and to improve the model's classification ability while simultaneously improving its detection ability by combining classification detection.

[0007] In a first aspect, embodiments of this application provide a target detection training method, the target detection training method comprising:

[0008] Acquire a first type of data and a second type of data, wherein the first type of data includes a first image, a first bounding box label corresponding to the first image, and a category-related label corresponding to the first bounding box label, wherein the category-related label belongs to the category corresponding to the classification task of the model to be trained; the second type of data includes a second image, a second bounding box label corresponding to the second image, and a category-independent label corresponding to the second bounding box label, wherein the category-independent label does not belong to the category corresponding to the classification task.

[0009] The first image is input into the model to be trained for object detection, and the first bounding box result and the first classification result corresponding to the first image are obtained. The second image is input into the model to be trained for object detection, and the second bounding box result and the second classification result corresponding to the second image are obtained.

[0010] Based on the first target bounding box result and first target bounding box annotation of the first image, and the second target bounding box result and second target bounding box annotation of the second image, the target bounding box loss is determined. Based on the first classification result and category-related annotation of the first image, the classification loss is determined. The category-independent annotation is used to indicate that the second classification result of the corresponding second image is not included in the calculation of the classification loss.

[0011] Based on the bounding box loss and the classification loss, the parameters of the model to be trained are adjusted, and the process of inputting the first image into the model to be trained is repeated to obtain the first bounding box result and the first classification result corresponding to the first image, until training is completed and a trained object detection model is obtained.

[0012] Secondly, embodiments of this application provide a target detection method, the target detection method comprising:

[0013] Obtain the image to be processed;

[0014] The image to be processed is input into the trained object detection model obtained by the object detection training method described in the first aspect, and the object detection result corresponding to the image to be processed is output. The object detection result includes the bounding box result and / or the classification category of the corresponding bounding box result.

[0015] Thirdly, according to an embodiment of this application, a target detection training device is provided, the target detection training device comprising:

[0016] The first acquisition module is used to acquire a first type of data and a second type of data. The first type of data includes a first image, a first bounding box label corresponding to the first image, and a category-related label corresponding to the first bounding box label. The category-related label belongs to the category corresponding to the classification task of the model to be trained. The second type of data includes a second image, a second bounding box label corresponding to the second image, and a category-independent label corresponding to the second bounding box label. The category-independent label does not belong to the category corresponding to the classification task.

[0017] The first object detection module is used to input the first image into the model to be trained for object detection, and obtain a first object box result and a first classification result corresponding to the first image; and to input the second image into the model to be trained for object detection, and obtain a second object box result and a second classification result corresponding to the second image.

[0018] The first loss calculation module is used to determine the target box loss based on the first target box result and the first target box annotation of the first image, the second target box result and the second target box annotation of the second image, and to determine the classification loss based on the first classification result and the category-related annotation of the first image. The category-independent annotation is used to indicate that the second classification result of the corresponding second image does not participate in the calculation of the classification loss.

[0019] The first iterative training module is used to adjust the parameters of the model to be trained according to the target box loss and the classification loss, and return to the step of inputting the first image into the model to be trained to obtain the first target box result and the first classification result corresponding to the first image, until the training is completed and a trained target detection model is obtained.

[0020] Fourthly, embodiments of this application provide a target detection device, the target detection device comprising:

[0021] The second acquisition module is used to acquire the image to be processed;

[0022] The object detection module is used to input the image to be processed into the trained object detection model obtained after executing the object detection training method described in the first aspect, and output the object detection result corresponding to the image to be processed. The object detection result includes the bounding box result and / or the classification category of the corresponding bounding box result.

[0023] Fifthly, embodiments of this application provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the target detection training method as described in the first aspect, or the target detection method as described in the second aspect.

[0024] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the target detection training method as described in the first aspect, or the target detection method as described in the second aspect.

[0025] The beneficial effects of this application embodiment compared with the prior art are as follows: This application obtains first type data and second type data, the classification and labeling accuracy of the first type data and the second type data are different. The first image is input into the model to be trained for target detection, and the first target box result and the first classification result corresponding to the first image are obtained. The second image is input into the model to be trained for target detection, and the second target box result and the second classification result corresponding to the second image are obtained. Based on the first target box result and the first target box label of the first image, and the second target box result and the second target box label of the second image, the target box loss is determined. Based on the first classification result and the category-related label of the first image, the classification loss is determined. The category-independent label is used to indicate that the second classification result of the corresponding second image is not included in the calculation of the classification loss. Based on the target box loss and the classification loss, the parameters of the model to be trained are adjusted. The process is repeated, and the first image is input into the model to be trained to obtain the first target box result and the first classification result corresponding to the first image, until the training is completed and a trained target detection model is obtained.

[0026] By using second-class data (which may have inaccurate or no classification labels, or data that does not belong to the classification task of the model to be trained) and first-class data (which have accurate classification labels), the model to be trained is trained. During the training process, the loss calculation methods for first-class and second-class data can be automatically distinguished, thereby accurately and effectively training the model. Compared with using only first-class data, the trained object detection model has a more accurate object detection effect, thus providing a good foundation for the classification of detected objects and improving the accuracy of object classification. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of an object detection training method or an application environment of an object detection method provided in Embodiment 1 of this application;

[0029] Figure 2This is a flowchart illustrating a target detection training method provided in Embodiment 2 of this application;

[0030] Figure 3 This is a flowchart illustrating a target detection training method provided in Embodiment 3 of this application;

[0031] Figure 4 This is a schematic flowchart of a target detection method provided in Embodiment 4 of this application;

[0032] Figure 5 This is a schematic diagram of the structure of a target detection training device provided in Embodiment 5 of this application;

[0033] Figure 6 This is a schematic diagram of the structure of a target detection device provided in Embodiment Six of this application;

[0034] Figure 7 This is a schematic diagram of the structure of a computer device provided in Embodiment 7 of this application. Detailed Implementation

[0035] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0036] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0037] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0038] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0039] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0040] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0041] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0042] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0043] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0044] To illustrate the technical solution of this application, specific embodiments are described below.

[0045] The object detection training method and object detection method provided in Embodiment 1 of this application can be applied to, for example, Figure 1In the application environment, both the object detection training method and the object detection method can be applied to the server. The server is used to deploy and implement the object detection training method and the corresponding model. The model corresponding to the object detection training method is an untrained model, which is trained by acquiring image data from the client or database. The model corresponding to the object detection method is a model trained using the object detection training method, which is used to acquire image data from the client or database to perform object detection, object classification, etc.

[0046] The client devices include, but are not limited to, handheld computers, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, server-side computer equipment, and personal digital assistants (PDAs). The server can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0047] See Figure 2 This is a flowchart illustrating a target detection training method provided in Embodiment 2 of this application. The above-described target detection training method is applied to... Figure 1 The server-side in [the context]. For example... Figure 2 As shown, the object detection training method may include the following steps:

[0048] Step S201: Obtain the first type of data and the second type of data.

[0049] The first type of data (i.e., class-aware data) includes the first image, the first bounding box annotation corresponding to the first image, and the class-related annotation corresponding to the first bounding box annotation. The class-related annotation belongs to the category corresponding to the classification task of the model to be trained. The second type of data (i.e., class-agnostic data) includes the second image, the second bounding box annotation corresponding to the second image, and the class-agnostic annotation corresponding to the second bounding box annotation. The class-agnostic annotation does not belong to the category corresponding to the classification task.

[0050] The difference between the first type of data and the second type of data is that the first type of data can be directly used for model classification training. It contains bounding boxes and classification labels for each bounding box, which correspond to the model's classification task. However, since the first type of data is relatively small, it requires a lot of manpower and resources to classify and label it. Therefore, the second type of data does not contain classification labels, but it has information such as bounding boxes, which can play a positive role in the model's processing of images to obtain bounding boxes.

[0051] To meet the needs of the model being trained, the second type of data contains classification labels (i.e., category-independent labels) for the corresponding bounding boxes. These category-independent labels do not belong to the categories of the classification task; their categories belong to the foreground category and correspond to a superclass. The categories corresponding to the category-related labels correspond to a subclass. The relationship between the superclass and the subclass can be one of complete inclusion, partial inclusion, or no inclusion. For example, the superclass could be "human body," and the subclasses could be specific human bodies such as men, women, the elderly, and children; or human bodies in different postures such as sitting, standing, and lying down.

[0052] For example, consider a human body detection model. This model processes image data such as pictures and video streams to identify target human bodies. To identify a "man" in an image, the training of this human body detection model would use two types of data: first, image data containing the bounding box "male"; and second, image data containing the bounding box "human body". Ultimately, this human body detection model can be used to track image data of "male". The first type of data includes the image, the bounding box, and the bounding box classification (i.e., male), while the second type of data includes the image, the bounding box, and the bounding box classification (i.e., human body). The classification task of the human body detection model is a gender task; therefore, the first type of data corresponds to the classification task, while the second type of data does not.

[0053] For example, consider a human pose detection model. This model is also used to process image data such as pictures and video streams to identify the target pose of a human body. If the goal is to identify a "standing" pose in an image, the first type of data in the training of this human pose detection model consists of image data containing a bounding box indicating a "standing" pose, and the second type of data consists of image data containing a bounding box indicating a "human body". Ultimately, this human pose detection model can be used to identify image data of a "standing" pose. The first type of data includes the image, the bounding box, and the classification of the bounding box (i.e., standing), while the second type of data includes the image, the bounding box, and the classification of the bounding box (i.e., human body). The classification task of the human detection model is a pose task; therefore, the first type of data corresponds to the classification task, while the second type of data does not.

[0054] Images can be acquired by corresponding image acquisition devices. If there are target regions formed by targets such as faces in the images, target detection can be performed on the images to capture the targets within the target regions. For example, when capturing targets, the corresponding targets can be captured by performing target detection on the real-time acquired images, which can then be used for target recognition, target tracking and other scenarios.

[0055] Optionally, the second type of data can be obtained, including:

[0056] Obtain the second image and the corresponding second bounding box annotations for the second image;

[0057] Using preset values, category labels are applied to the second target box annotations to obtain category-independent labels for the corresponding second target box annotations;

[0058] The second image, the corresponding second bounding box annotation of the second image, and the category-independent annotation of the corresponding second bounding box annotation are used to form the second type of data.

[0059] In constructing the second type of data, the second image and the corresponding second target box annotation can be obtained. That is, when this part of the data is collected, it only contains the image and the target box annotation. Therefore, for each target box, a preset value is used as its category annotation to form a category-independent annotation, so that the structure of the second type of data is the same as that of the first type of data, so as to meet the use of the model.

[0060] In the first type of data, the index of any image corresponding to a category is one of all categories in the classification task. In the second type of data, the index of any image corresponding to a category is the preset value mentioned above, which can be 0, -1, etc.

[0061] Step S202: Input the first image into the model to be trained for object detection to obtain the first bounding box result and the first classification result corresponding to the first image; input the second image into the model to be trained for object detection to obtain the second bounding box result and the second classification result corresponding to the second image.

[0062] The model to be trained is an image object detection and classification model. After an image is input into the model, it can output object detection results and classification results, i.e., the bounding box result and classification result for each image. The model to be trained includes, but is not limited to, proposal-based two-stage Convolutional Neural Networks (CNN) object detection frameworks (i.e., Faster R-CNN series networks), pixel-based single-stage CNN object detection frameworks (i.e., YOLO series networks), and query-based Transformer object detection frameworks (i.e., DETR series networks), etc.

[0063] For example, the spatial position of the target box in the image [xmin, ymin, xmax, ymax] can be obtained based on the coordinate points and spatial feature values ​​[x1, y1, x2, y2], where (xmin, ymin) represents the coordinates of the top left vertex of the target box, and (xmax, ymax) represents the coordinates of the bottom right vertex of the target box.

[0064] For example, the classification result can be calculated by similarity matching, and the target category can be output as the index of the category corresponding to the maximum similarity.

[0065] In the target detection process of the model to be trained, the processing method for the first image and the second image is the same, and both can output target box results and classification results. Therefore, there is no need to set up a separate training process for the second image, which can effectively reduce the workload of training design and improve training efficiency.

[0066] Step S203: Determine the target box loss based on the first target box result and first target box annotation of the first image, the second target box result and second target box annotation of the second image, and determine the classification loss based on the first classification result and category-related annotation of the first image.

[0067] The category-independent label is used to indicate that the second classification result of the corresponding second image is not included in the calculation of the classification loss. Since the category-independent label of the second class data is not an accurate labeling result, its classification label does not help the training of the model to be trained, and the corresponding category-independent label is not used to calculate the loss, that is, the loss is ignored.

[0068] Category-independent labels can be assigned a preset value during construction. This preset value is different from all the classification categories of the classification task, so that the loss calculated based on the category-independent label cannot participate in the composition of the classification loss. For example, the preset value is -1, which distinguishes category-related labels and prevents them from affecting the classification loss when calculating the classification loss.

[0069] Optionally, before determining the target bounding box loss based on the first target bounding box result and first target bounding box annotation of the first image, and the second target bounding box result and second target bounding box annotation of the second image, the method further includes:

[0070] When dynamically assigning positive and negative samples to the first image, the first target bounding box and the corresponding category-related annotation are used to calculate the first matching score;

[0071] When dynamically assigning positive and negative samples to the second image, the second target box is used for annotation, and the second matching score is calculated.

[0072] Based on the first matching score and the second matching score, the first image and the second image are divided into positive and negative samples to obtain the positive and negative sample division results. The positive and negative sample division results are used to indicate the calculation of the target box loss and the classification loss.

[0073] In the process of calculating the loss, the model needs to assign positive and negative samples to the data used for training. In this application, whether it is a proposal-based model, a pixel-based model, or a query-based model, its corresponding static allocation strategy only depends on the geometric and positional relationship between the anchor point and the target box annotation. In algorithms such as SimOTA, TOOD, and Hungarian matching, the one-to-one matching score between the anchor point and the target box annotation is calculated. The matching score is composed of the target box localization loss and the target box classification loss. Due to the existence of the second type of data in this application, the classification loss cannot accurately reflect the matching score. Therefore, in this application, the matching score is composed of the target box loss, thereby accurately realizing the positive and negative sample allocation.

[0074] Step S204: Adjust the parameters of the model to be trained based on the bounding box loss and classification loss, return to the step of inputting the first image into the model to be trained, obtain the first bounding box result and the first classification result of the first image, until training is completed and a trained object detection model is obtained.

[0075] During training, the parameters of the object detection model are adjusted through mechanisms such as inverse adjustment using the bounding box loss and classification loss. Specifically, these parameters can be applied to the backbone of the network within the object detection model. In one implementation, the bounding box loss and classification loss can be weighted and summed to obtain the total training loss. This total training loss is then used to adjust the parameters, enabling iterative training of the model.

[0076] The object detection model described above can either stop iterating when the preset number of iterations is reached, thus completing the training, or it can stop iterating when the total training loss converges, thus completing the training.

[0077] Optionally, the first type of data also includes a first target evaluation corresponding to the first image, and the second type of data also includes a second target evaluation corresponding to the second image;

[0078] Object detection training methods also include:

[0079] The first image is input into the model to be trained for object detection, and the first object confidence score corresponding to the first image is obtained. The second image is input into the model to be trained for object detection, and the second object confidence score corresponding to the second image is obtained.

[0080] The target recognition loss is determined based on the first target evaluation and first target confidence of the first image, and the second target evaluation and second target confidence of the second image;

[0081] Based on the bounding box loss and classification loss, adjust the parameters of the model to be trained, including:

[0082] The parameters of the model to be trained are adjusted based on the target recognition loss, the bounding box loss, and the classification loss.

[0083] The first type of data includes a first target evaluation of the first image, and the second type of data includes a second target evaluation of the second image. The target evaluation is used to characterize the confidence level of the presence or absence of a target, corresponding to the confidence level of the target output by the model in processing the image.

[0084] For example, the data format can be [xmin, ymin, xmax, ymax, cls, confidence], where [xmin, ymin, xmax, ymax] represents the target bounding box, [cls] represents the category, and [confidence] represents the confidence level (i.e., target evaluation).

[0085] Based on the above data format, the target recognition loss can be obtained by comparing target evaluation and target confidence. This loss characterizes the model's ability to identify the presence or absence of a target. Accordingly, when adjusting the model parameters, the target recognition loss, bounding box loss, and classification loss are required. In one embodiment, the target recognition loss, bounding box loss, and classification loss are weighted and summed to obtain a total loss, which is then used to adjust the model parameters.

[0086] In one embodiment, Varifocal Loss can be used to calculate the category loss between the classification result and the classification label, GIoU Loss can be used to calculate the target box loss between the target box result and the target box label, and the target recognition loss between the target evaluation and the target confidence can be calculated. This embodiment utilizes second-class data (which does not belong to the category corresponding to the classification task of the model to be trained, such as inaccurate or non-specific classification labels) and first-class data with accurate classification to train the model. During training, the loss calculation methods for the first-class and second-class data can be automatically distinguished, thereby accurately and effectively training the model. Compared to using only the first-class data, the trained target detection model has a more accurate target detection effect, thus providing a good foundation for the classification of detected targets and improving the accuracy of target classification.

[0087] See Figure 3 This is a flowchart illustrating a target detection training method provided in Embodiment 3 of this application. Figure 3 As shown, after obtaining the first type of data and the second type of data in step S201 above, the object detection training method may include the following steps:

[0088] Step S301: Perform data augmentation on the first type of data to obtain augmented first type of data.

[0089] The augmented first type of data includes N first images, the first bounding box annotations corresponding to the first images, and the category-related annotations corresponding to the first bounding box annotations. The category-related annotations belong to the category corresponding to the classification task of the model to be trained, and N is an integer greater than 1.

[0090] Step S302: Perform data augmentation on the second type of data to obtain augmented second type of data.

[0091] The second type of data includes M second images, the corresponding second bounding box annotations for the second images, and the category-independent annotations for the corresponding second bounding box annotations. The category-independent annotations do not belong to the category corresponding to the classification task, and M is an integer greater than 1.

[0092] The aforementioned augmentation can employ methods such as random flipping, cropping, and color transformation to obtain more image data, thereby improving the accuracy of the trained model and reducing the difficulty of constructing the training set.

[0093] Optionally, after augmenting the second type of data to obtain the augmented second type of data, the process may also include:

[0094] The augmented first-class data and the augmented second-class data are concatenated to obtain the augmented training set;

[0095] Each image in the augmented training set is sequentially input into the model to be trained.

[0096] The augmented first-class data and the augmented second-class data are concatenated to form a training set. This training set can be directly fed into the model to be trained, improving the automation of model training and avoiding the need for manual differentiation between the first-class and second-class data.

[0097] See Figure 4 This is a flowchart illustrating a target detection method provided in Embodiment 4 of this application. The target detection method is applied to... Figure 1 The server-side in [the context]. For example... Figure 4 As shown, the target detection method may include the following steps:

[0098] Step S401: Obtain the image to be processed.

[0099] Step S402: Input the image to be processed into the trained object detection model obtained after the above object detection training method is executed, and output the object detection result corresponding to the image to be processed. The object detection result includes the object box result and / or the classification category of the corresponding object box result.

[0100] The above-mentioned object detection method can be switched according to the user's needs. If the user wants to generate data with only object bounding boxes, the corresponding image can be input, and only the object bounding box result corresponding to the image can be obtained to achieve object detection. If the user wants to generate data with classification results, the corresponding image can be input, and the object bounding box result and the classification result of the corresponding object bounding box result can be obtained. If only the classification result is needed, the classification result can be obtained from the above content.

[0101] For example, if the target detection model is a human body detection model, and the purpose of this human body detection model is to detect "male" in image data, before detecting "male", the human body detection model needs to detect the target boxes of human bodies in the image data. Therefore, the human body detection model performs human body recognition on the image to be processed, which can obtain the target boxes of human bodies in the image (i.e., the boxes surrounding the human body), and can also classify the gender category of the target boxes. That is, if the target box surrounds the human body as "male", then the target box can be classified and labeled as "male".

[0102] For example, if the target detection model is a human pose detection model, this human pose detection model is used to detect the "standing" pose in image data. Before detecting the "standing" pose, the human pose detection model needs to detect the target boxes of the human body in the image data. Therefore, the human pose detection model performs human recognition on the image to be processed, which can obtain the target boxes of the human body in the image, and also classify the pose category of the target boxes. That is, if the target box surrounds the human body in a "standing" pose, the target box can be classified and labeled as "standing".

[0103] This application's method differs from existing methods that can only train on class-aware or class-agnostic data individually. It can train on both types of data simultaneously, enhancing both detection and classification capabilities. Compared to training only class-agnostic data, this application's method is more functional; compared to training only class-aware data, its detection capability is stronger. Furthermore, this application's method can arbitrarily switch between class-aware and class-agnostic modes according to user needs. When the superclass of the class-agnostic data and the subclass of the class-aware data have a partial or no inclusion relationship, and the output is class-agnostic, it can be directly used for open-set object detection, broadening its application scope. Moreover, this application's method can be seamlessly embedded into existing object detection frameworks, demonstrating strong scalability.

[0104] In this embodiment, first and second types of data are acquired. The classification and labeling accuracy of the first and second types of data differs. The first image is input into the model to be trained for object detection, resulting in a first bounding box and a first classification result for the corresponding first image. The second image is input into the model to be trained for object detection, resulting in a second bounding box and a second classification result for the corresponding second image. Based on the first bounding box and its labeling in the first image, and the second bounding box and its labeling in the second image, the bounding box loss is determined. Based on the first classification result and category-related labeling in the first image, the classification loss is determined. Category-independent labeling is used to indicate that the second classification result for the corresponding second image does not participate in the calculation of the classification loss. Based on the bounding box loss and the classification loss, the parameters of the model to be trained are adjusted. The process is repeated, inputting the first image into the model to be trained to obtain the first bounding box and the first classification result for the corresponding first image, until training is complete, resulting in a trained object detection model.

[0105] By using second-class data (which may have inaccurate or no classification labels, or data that does not belong to the classification task of the model to be trained) and first-class data (which have accurate classification labels), the model to be trained is trained. During the training process, the loss calculation methods for first-class and second-class data can be automatically distinguished, thereby accurately and effectively training the model. Compared with using only first-class data, the trained object detection model has a more accurate object detection effect, thus providing a good foundation for the classification of detected objects and improving the accuracy of object classification.

[0106] Corresponding to the object detection training method in the above embodiments, Figure 5 This diagram illustrates the structural block diagram of the target detection training device provided in Embodiment 5 of this application. The target detection training device can be used for the above-mentioned... Figure 1 The server-side component is shown. For ease of explanation, only the parts relevant to the embodiments of this application are shown.

[0107] See Figure 5 The target detection training device includes:

[0108] The first acquisition module 51 is used to acquire a first type of data and a second type of data. The first type of data includes a first image, a first target box annotation corresponding to the first image, and a category-related annotation corresponding to the first target box annotation. The category-related annotation belongs to the category corresponding to the classification task of the model to be trained. The second type of data includes a second image, a second target box annotation corresponding to the second image, and a category-independent annotation corresponding to the second target box annotation. The category-independent annotation does not belong to the category corresponding to the classification task.

[0109] The first object detection module 52 is used to input the first image into the model to be trained for object detection, and obtain the first object box result and the first classification result corresponding to the first image; and to input the second image into the model to be trained for object detection, and obtain the second object box result and the second classification result corresponding to the second image.

[0110] The first loss calculation module 53 is used to determine the target box loss based on the first target box result and the first target box annotation of the first image, the second target box result and the second target box annotation of the second image, and to determine the classification loss based on the first classification result and the category-related annotation of the first image. The category-independent annotation is used to indicate that the second classification result of the corresponding second image does not participate in the calculation of the classification loss.

[0111] The first iteration training module 54 is used to adjust the parameters of the model to be trained based on the target box loss and classification loss, and return to execute the input of the first image into the model to be trained to obtain the first target box result and the first classification result of the corresponding first image, until the training is completed and a trained object detection model is obtained.

[0112] Optionally, the first type of data also includes a first target evaluation corresponding to the first image, and the second type of data also includes a second target evaluation corresponding to the second image;

[0113] The target detection training device also includes:

[0114] The second target detection module is used to input the first image into the model to be trained for target detection, and obtain the first target confidence score corresponding to the first image; and to input the second image into the model to be trained for target detection, and obtain the second target confidence score corresponding to the second image.

[0115] The second loss calculation module is used to determine the target recognition loss based on the first target evaluation and first target confidence of the first image, and the second target evaluation and second target confidence of the second image.

[0116] The first iteration training module 54 includes:

[0117] The first iteration training unit is used to adjust the parameters of the model to be trained based on the target recognition loss, the bounding box loss, and the classification loss.

[0118] Optionally, the first acquisition module 51 includes:

[0119] An image acquisition unit is used to acquire a second image and a corresponding second bounding box annotation for the second image;

[0120] The category labeling unit is used to label the second target box annotation with a preset value to obtain the category-independent annotation of the corresponding second target box annotation;

[0121] The second type of data construction unit is used to form the second type of data by combining the second image, the corresponding second target box annotation of the second image, and the category-independent annotation of the corresponding second target box annotation.

[0122] Optionally, the object detection training device also includes:

[0123] The first augmentation module is used to augment the first type of data after acquiring the first type of data and the second type of data, so as to obtain the augmented first type of data. The augmented first type of data includes N first images, the first target box annotation corresponding to the first image, and the category-related annotation corresponding to the first target box annotation. The category-related annotation belongs to the category corresponding to the classification task of the model to be trained, and N is an integer greater than 1.

[0124] The second augmentation module is used to augment the second type of data to obtain augmented second type of data. The second type of data includes M second images, the second target box annotations corresponding to the second images, and the category-independent annotations corresponding to the second target box annotations. The category-independent annotations do not belong to the category corresponding to the classification task, and M is an integer greater than 1.

[0125] Optionally, the object detection training device also includes:

[0126] The first matching calculation module is used to calculate the first matching score by using the first target box annotation and the corresponding category-related annotation when dynamically allocating positive and negative samples in the first image before determining the target box loss based on the first target box result and the first target box annotation of the first image, the second target box result and the second target box annotation of the second image.

[0127] The second matching calculation module is used to calculate the second matching score by using the second target box annotation when dynamically assigning positive and negative samples to the second image;

[0128] The positive and negative sample partitioning module is used to partition the first image and the second image into positive and negative samples based on the first matching score and the second matching score, obtaining the positive and negative sample partitioning results. The positive and negative sample partitioning results are used to indicate the calculation of the target box loss and the classification loss. It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiment of this application. Their specific functions and the resulting technical effects can be found in the method embodiment section, and will not be repeated here.

[0129] Corresponding to the target detection method in the above embodiments, Figure 6 This diagram illustrates the structural block diagram of the target detection device provided in Embodiment Six of this application. The target detection device can be used in the above-described... Figure 1 The server-side component is shown. For ease of explanation, only the parts relevant to the embodiments of this application are shown.

[0130] See Figure 6 The target detection device includes:

[0131] The second acquisition module 61 is used to acquire the image to be processed;

[0132] The object detection module 62 is used to input the image to be processed into the trained object detection model obtained after the above object detection training method is executed, and output the object detection result corresponding to the image to be processed. The object detection result includes the object box result and / or the classification category of the corresponding object box result.

[0133] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0134] Figure 7 This is a schematic diagram of the structure of a computer device provided in Embodiment Seven of this application. Figure 7 As shown, the computer device of this embodiment includes: at least one processor ( Figure 7 (Only one is shown in the diagram), a memory, and a computer program stored in the memory and executable on a processor, wherein at least one processor executes the computer program to implement the steps of any of the above-described object detection training methods or object detection method embodiments.

[0135] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 7 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0136] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0137] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of the computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0138] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0139] The implementation of all or part of the processes in the methods of the above embodiments can also be accomplished by a computer program product. When the computer program product is run on a computer device, it enables the computer device to execute the steps in the above method embodiments.

[0140] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0141] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0142] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0143] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0144] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A target detection training method, characterized in that, The target detection training method comprises: obtaining first data and second data, wherein the first data comprises a first image, a first target frame label corresponding to the first image, and a class-related label corresponding to the first target frame label, the class-related label belongs to a class corresponding to a classification task of a to-be-trained model, the second data comprises a second image, a second target frame label corresponding to the second image, and a class-unrelated label corresponding to the second target frame label, the class-unrelated label does not belong to the class corresponding to the classification task; inputting the first image into the to-be-trained model for target detection to obtain a first target frame result and a first classification result corresponding to the first image, and inputting the second image into the to-be-trained model for target detection to obtain a second target frame result and a second classification result corresponding to the second image; determining a target frame loss according to the first target frame result and the first target frame label of the first image, and the second target frame result and the second target frame label of the second image, and determining a classification loss according to the first classification result and the class-related label of the first image, wherein the class-unrelated label is used to indicate that the second classification result corresponding to the second image does not participate in calculating the classification loss; adjusting parameters of the to-be-trained model according to the target frame loss and the classification loss, and returning to execute the inputting the first image into the to-be-trained model for target detection to obtain the first target frame result and the first classification result corresponding to the first image until training is completed, thereby obtaining a trained target detection model; before the determining the target frame loss according to the first target frame result and the first target frame label of the first image, and the second target frame result and the second target frame label of the second image, the method further comprises: when performing dynamic allocation of positive and negative samples on the first image, using the first target frame label and the corresponding class-related label to calculate a first matching score; when performing dynamic allocation of positive and negative samples on the second image, using the second target frame label to calculate a second matching score; performing positive and negative sample division on the first image and the second image according to the first matching score and the second matching score to obtain a positive and negative sample division result, wherein the positive and negative sample division result is used to indicate calculation of the target frame loss and the classification loss.

2. The object detection training method of claim 1, wherein, The first data further comprises a first target evaluation corresponding to the first image, and the second data further comprises a second target evaluation corresponding to the second image; The target detection training method further comprises: inputting the first image into the to-be-trained model for target detection to obtain a first target confidence corresponding to the first image, and inputting the second image into the to-be-trained model for target detection to obtain a second target confidence corresponding to the second image; determining a target recognition loss according to the first target evaluation and the first target confidence of the first image, and the second target evaluation and the second target confidence of the second image; The adjusting the parameters of the to-be-trained model according to the target frame loss and the classification loss comprises: Adjust parameters of the to-be-trained model according to the target recognition loss, the target frame loss and the classification loss.

3. The object detection training method of claim 1, wherein, The obtaining the second type of data comprises: obtaining a second image and a second target frame label corresponding to the second image; using a preset value to perform category labeling on the second target frame label to obtain category-independent labeling corresponding to the second target frame label; forming second type of data from the second image, the second target frame label corresponding to the second image and the category-independent labeling corresponding to the second target frame label.

4. The object detection training method of claim 1, wherein, After the first type of data and the second type of data are obtained, the method further comprises: performing data augmentation on the first type of data to obtain augmented first type of data, the augmented first type of data comprising N first images, first target frame labels corresponding to the first images and category-dependent labeling corresponding to the first target frame labels, the category-dependent labeling belonging to categories corresponding to a classification task of a to-be-trained model, N being an integer greater than 1; performing data augmentation on the second type of data to obtain augmented second type of data, the augmented second type of data comprising M second images, second target frame labels corresponding to the second images and category-independent labeling corresponding to the second target frame labels, the category-independent labeling not belonging to the categories corresponding to the classification task, M being an integer greater than 1.

5. A target detection method characterized by, The target detection method comprises: obtaining a to-be-processed image; inputting the to-be-processed image into a trained target detection model obtained by executing the target detection training method according to any one of claims 1 to 4 to output a target detection result corresponding to the to-be-processed image, the target detection result comprising a target frame result and / or a classification category corresponding to the target frame result.

6. An object detection training apparatus, characterized by, The target detection training device comprises: a first obtaining module configured to obtain first type of data and second type of data, wherein the first type of data comprises first images, first target frame labels corresponding to the first images and category-dependent labeling corresponding to the first target frame labels, the category-dependent labeling belonging to categories corresponding to a classification task of a to-be-trained model, and the second type of data comprises second images, second target frame labels corresponding to the second images and category-independent labeling corresponding to the second target frame labels, the category-independent labeling not belonging to the categories corresponding to the classification task; a first target detection module configured to input the first images into the to-be-trained model to perform target detection to obtain first target frame results and first classification results corresponding to the first images, and input the second images into the to-be-trained model to perform target detection to obtain second target frame results and second classification results corresponding to the second images; a first loss calculation module configured to determine a target frame loss according to the first target frame results and the first target frame labels of the first images and the second target frame results and the second target frame labels of the second images, determine a classification loss according to the first classification results of the first images and the category-dependent labeling, and use the category-independent labeling to indicate that the second classification results corresponding to the second images do not participate in calculating the classification loss. The first iterative training module is configured to adjust parameters of the to-be-trained model according to the target frame loss and the classification loss, and return to execute the inputting of the first image into the to-be-trained model for target detection to obtain a first target frame result and a first classification result corresponding to the first image until training is completed, and a trained target detection model is obtained. Further comprising: The first matching calculation module is configured to, before determining the target frame loss according to the first target frame result and the first target frame label of the first image and the second target frame result and the second target frame label of the second image, calculate a first matching score using the first target frame label and a corresponding class-related label when performing dynamic allocation of positive and negative samples on the first image. The second matching calculation module is configured to calculate a second matching score using the second target frame label when performing dynamic allocation of positive and negative samples on the second image. The positive and negative sample division module is configured to perform positive and negative sample division on the first image and the second image according to the first matching score and the second matching score to obtain a positive and negative sample division result, and the positive and negative sample division result is used to indicate calculation of the target frame loss and the classification loss.

7. A target detection apparatus characterized by comprising: The target detection device comprises: The second acquisition module is configured to acquire a to-be-processed image. The target detection module is configured to input the to-be-processed image into a trained target detection model obtained by executing the target detection training method according to any one of claims 1 to 4, and output a target detection result corresponding to the to-be-processed image, wherein the target detection result comprises a target frame result and / or a classification class corresponding to the target frame result.

8. A computer device, comprising: The computer device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the target detection training method according to any one of claims 1 to 4 or the target detection method according to claim 5 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the target detection training method according to any one of claims 1 to 4 or the target detection method according to claim 5.

Images