Target detection training method and device, target detection method and device, equipment and medium

Abstract

The invention is suitable for the technical field of image processing, and particularly relates to a target detection training method, a target detection method, devices, equipment and a medium. According to the method provided by the invention, the to-be-trained model is trained by utilizing the second type of data which does not belong to the type corresponding to the classification task of the to-be-trained model, such as inaccurate classification annotation or no specific classification annotation, and the first type of data which is accurately classified; in the training process, the loss calculation modes of the first type of data and the second type of data can be automatically distinguished, so that the training of the to-be-trained model is accurately and effectively realized, compared with only using the first type of data, the trained target detection model has a more accurate target detection effect, and the training efficiency is improved. Therefore, a good basis can be provided for classification of detected targets, and the accuracy of target classification is improved.

Description

Technical Field

[0001] The present application is applicable to the field of image processing technology, and in particular relates to a target detection training method, a target detection method, a device, a equipment and a medium. Background Art

[0002] At present, object detection based on deep learning has gradually become the core technology in the fields of autonomous driving, video surveillance, mechanical processing, intelligent robots, etc., and data and data volume are the most important in deep learning. In some application scenarios, due to labeling reasons, part of the massive data is labeled with specific categories (i.e. class-aware), and part of the data is only labeled with foreground categories, that is, all categories are divided into one foreground (i.e. class-agnostic).

[0003] The class-aware data is of practical significance for training the classification ability of the target detection model, while the class-agnostic data cannot provide any help for the classification of the target detection model. The class-agnostic data contains the target box but does not have a specific classification of the target box. It can be used to a certain extent to train the model to locate the target box.

[0004] The existing models are trained with class-aware data and have certain capabilities for detecting and classifying targets. However, they are mainly trained to improve classification accuracy. As the detection target is the basis of classification, the effect of using class-aware data on improving the accuracy of the detection target is limited.

[0005] Therefore, how to rationally utilize the characteristics of known data to optimize the accuracy of the model in detecting the target, and how to combine classification detection to improve the model's classification ability while improving its detection ability has become an urgent problem to be solved. Summary of the invention

[0006] In view of this, the embodiments of the present application provide a target detection training method, a target detection method, an apparatus, a device and a medium to solve the problem of how to reasonably utilize the characteristics of known data to optimize the accuracy of the model in detecting the target, so as to combine classification detection to improve the classification ability of the model while improving the detection ability.

[0007] In a first aspect, an embodiment of the present application provides a target detection training method, the target detection training method comprising: Acquire first-category data and second-category data, wherein the first-category data includes a first image, a first target frame annotation corresponding to the first image, and a category-related annotation corresponding to the first target frame annotation, wherein the category-related annotation belongs to a category corresponding to a classification task of a model to be trained, and the second-category data includes a second image, a second target frame annotation corresponding to the second image, and a category-independent annotation corresponding to the second target frame annotation, wherein the category-independent annotation does not belong to a category corresponding to the classification task; Input the first image into the model to be trained for target detection, and obtain a first target frame result and a first classification result corresponding to the first image; input the second image into the model to be trained for target detection, and obtain a second target frame result and a second classification result corresponding to the second image; Determine a target box loss according to a first target box result and a first target box annotation of the first image, a second target box result and a second target box annotation of the second image, and determine a classification loss according to a first classification result and a category-related annotation of the first image, wherein the category-independent annotation is used to indicate that the second classification result corresponding to the second image does not participate in calculating the classification loss; According to the target box loss and the classification loss, the parameters of the model to be trained are adjusted, and the step of inputting the first image into the model to be trained is returned to obtain a first target box result and a first classification result corresponding to the first image, until the training is completed to obtain a trained target detection model.

[0008] In a second aspect, an embodiment of the present application provides a target detection method, the target detection method comprising: Get the image to be processed; The image to be processed is input into the trained target detection model obtained after executing the target detection training method described in the first aspect, and the target detection result corresponding to the image to be processed is output, and the target detection result includes the target frame result and / or the classification category of the corresponding target frame result.

[0009] In a third aspect, an embodiment of the present application provides a target detection training device, the target detection training device comprising: A first acquisition module is used to acquire first-category data and second-category data, wherein the first-category data includes a first image, a first target frame annotation corresponding to the first image, and a category-related annotation corresponding to the first target frame annotation, wherein the category-related annotation belongs to a category corresponding to the classification task of the model to be trained, and the second-category data includes a second image, a second target frame annotation corresponding to the second image, and a category-independent annotation corresponding to the second target frame annotation, wherein the category-independent annotation does not belong to a category corresponding to the classification task; A first target detection module is used to input the first image into the model to be trained for target detection, obtain a first target frame result and a first classification result corresponding to the first image, and input the second image into the model to be trained for target detection, obtain a second target frame result and a second classification result corresponding to the second image; a first loss calculation module, configured to determine a target box loss according to a first target box result and a first target box annotation of the first image, a second target box result and a second target box annotation of the second image, and to determine a classification loss according to a first classification result and a category-related annotation of the first image, wherein the category-irrelevant annotation is used to indicate that a second classification result corresponding to the second image does not participate in calculating the classification loss; The first iterative training module is used to adjust the parameters of the model to be trained according to the target box loss and the classification loss, return to execute the step of inputting the first image into the model to be trained, obtain a first target box result and a first classification result corresponding to the first image, until the training is completed to obtain a trained target detection model.

[0010] In a fourth aspect, an embodiment of the present application provides a target detection device, the target detection device comprising: A second acquisition module is used to acquire the image to be processed; The target detection module is used to input the image to be processed into a trained target detection model obtained by executing the target detection training method described in the first aspect, and output a target detection result corresponding to the image to be processed, wherein the target detection result includes a target frame result and / or a classification category corresponding to the target frame result.

[0011] In a fifth aspect, an embodiment of the present application provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the target detection training method as described in the first aspect or the target detection method as described in the second aspect is implemented.

[0012] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the target detection training method as described in the first aspect, or the target detection method as described in the second aspect is implemented.

[0013] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the present application obtains first-category data and second-category data, and the classification and annotation accuracies of the first-category data and the second-category data are different; the first image is input into the model to be trained for target detection, and a first target frame result and a first classification result corresponding to the first image are obtained; the second image is input into the model to be trained for target detection, and a second target frame result and a second classification result corresponding to the second image are obtained; the target frame loss is determined according to the first target frame result and the first target frame annotation of the first image, and the second target frame result and the second target frame annotation of the second image; the classification loss is determined according to the first classification result of the first image and the category-related annotation; the category-irrelevant annotation is used to indicate that the second classification result corresponding to the second image does not participate in the calculation of the classification loss; the parameters of the model to be trained are adjusted according to the target frame loss and the classification loss; the execution is returned to input the first image into the model to be trained, and the first target frame result and the first classification result corresponding to the first image are obtained, until the training is completed, and a trained target detection model is obtained.

[0014] The model to be trained is trained using the second category data that does not belong to the category corresponding to the classification task of the model to be trained and has inaccurate classification labels or no specific classification labels, and the first category data with accurate classification. During the training process, the loss calculation method of the first category data and the second category data can be automatically distinguished, so as to accurately and effectively realize the training of the model to be trained. Compared with using only the first category data, the trained target detection model has a more accurate target detection effect, which can provide a good basis for the classification of the detected targets, so as to improve the accuracy of target classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0016] Figure 1 It is a schematic diagram of an application environment of a target detection training method or a target detection method provided in Example 1 of the present application; Figure 2 It is a flowchart of a target detection training method provided in Example 2 of the present application; Figure 3 It is a flowchart of a target detection training method provided in Example 3 of the present application; Figure 4 It is a flowchart of a target detection method provided in Example 4 of the present application; Figure 5It is a structural schematic diagram of a target detection training device provided in Example 5 of the present application; Figure 6 is a structural schematic diagram of a target detection device provided in Example 6 of the present application; Figure 7 It is a structural diagram of a computer device provided in Example 7 of the present application. DETAILED DESCRIPTION

[0017] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0018] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0019] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0020] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0021] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0022] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0023] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0024] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0025] It should be understood that the size of the serial numbers of the steps in the following embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0026] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0027] The target detection training method and target detection method provided in the first embodiment of the present application can be applied to Figure 1 In an application environment, the target detection training method and the target detection method can both be applied to the server, and the server is used to deploy and implement the target detection training method and the model corresponding to the target detection method. The model corresponding to the target detection training method is an untrained model, which is trained by obtaining image data from a client or a database, and the model corresponding to the target detection method is a model trained using the target detection training method, which is used to obtain image data from a client or a database to perform target detection, target classification, etc.

[0028] The client includes, but is not limited to, a PDA, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a server computer device, a personal digital assistant (PDA), and other computer devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers.

[0029] See also Figure 2 , is a flow chart of a target detection training method provided in Example 2 of the present application. The target detection training method is applied to Figure 1 The server in . Figure 2 As shown, the target detection training method may include the following steps: Step S201, obtaining first category data and second category data.

[0030] Among them, the first type of data (i.e., class-aware data) includes the first image, the first target box annotation corresponding to the first image, and the category-related annotation corresponding to the first target box annotation, and the category-related annotation belongs to the category corresponding to the classification task of the model to be trained; the second type of data (i.e., class-agnostic data) includes the second image, the second target box annotation corresponding to the second image, and the category-independent annotation corresponding to the second target box annotation, and the category-independent annotation does not belong to the category corresponding to the classification task.

[0031] The difference between the first category of data and the second category of data is that the first category of data can be directly used for classification training of the model, which includes the target box and the classification annotation of each target box, and the classification annotation corresponds to the classification task of the model. Since the amount of the first category of data is small, a lot of manpower and material resources are required for classification annotation. Therefore, the second category of data does not have classification annotation, but has information such as the target box, which can play a certain positive role in the model processing the image to obtain the target box.

[0032] In order to meet the use of the model to be trained, the second type of data contains classification annotations corresponding to the target box (i.e., category-independent annotations). The category-independent annotations do not belong to the category of the classification task. The category of the category-independent annotations belongs to the foreground category, which corresponds to a superclass, and the category corresponding to the category-related annotations corresponds to a subclass. The superclass and subclass can be fully inclusive, partially inclusive, or completely exclusive. For example, the superclass is the human body, and the subclass can be a specific human body such as a man, a woman, an old man, a child, or a human body in different postures such as a sitting person, a standing person, or a lying person.

[0033] For example, the target detection model is a human body detection model, which is used to process image data such as pictures and video streams to identify the target human body. If the purpose is to identify a "man" in an image, the first type of data in the training of the human body detection model is image data containing a target frame of "male", and the second type of data is image data containing a target frame of "human body". Finally, the human body detection model can be used to track image data of "male", wherein the first type of data includes images, target frames and classifications of target frames (i.e., male), and the second type of data includes images, target frames and classifications of target frames (i.e., human body). The classification task of the human body detection model is a gender task. Therefore, the first type of data corresponds to the classification task, and the second type of data does not correspond to the classification task.

[0034] For another example, the target detection model is a human posture detection model, which is also used to process image data such as pictures and video streams to identify the target posture of the human body. If the purpose is to identify the "standing" posture in the image, the first type of data in the training of the human posture detection model is image data containing a target frame of "standing" posture, and the second type of data is image data containing a target frame of "human body". Finally, the human posture detection model can be used to identify image data of "standing" posture, wherein the first type of data includes images, target frames and classification of target frames (i.e. standing), and the second type of data includes images, target frames and classification of target frames (i.e. human body). The classification task of the human detection model is a posture task. Therefore, the first type of data corresponds to the classification task, and the second type of data does not correspond to the classification task. The image can be acquired by the corresponding image acquisition device. There is a target area formed by targets such as faces in the image, so that target detection can be performed on the image to capture the target in the target area. For example, when capturing a target, target detection can be performed on the image acquired in real time to capture the corresponding target, which can be used in scenarios such as target recognition and target tracking.

[0035] Optionally, obtain the second type of data, including: Acquire a second image and a second target box annotation corresponding to the second image; Using the preset value, the second target frame annotation is subjected to category annotation to obtain a category-independent annotation corresponding to the second target frame annotation; The second image, the second target box annotation corresponding to the second image, and the category-independent annotation corresponding to the second target box annotation form a second category of data.

[0036] Among them, when constructing the second category of data, the second image and the second target box annotation of the corresponding image can be obtained, that is, this part of the data only contains the image and target box annotation when it is collected. Therefore, for each target box, a preset value is used as its category annotation to form a category-independent annotation, so that the second category of data has the same structure as the first category of data to meet the use of the model.

[0037] The index of the category corresponding to any image in the constructed first category data is one of all categories in the classification task, and the index of the category corresponding to any image in the constructed second category data is the above-mentioned preset value, which can be 0, -1, etc.

[0038] Step S202: Input the first image into the model to be trained for target detection, obtain a first target frame result and a first classification result corresponding to the first image, and input the second image into the model to be trained for target detection, obtain a second target frame result and a second classification result corresponding to the second image.

[0039] The model to be trained is a target detection and classification model for images. After the image is input into the model to be trained, the model to be trained can output the target detection result and classification result, that is, the target box result and classification result corresponding to each image. The model to be trained includes but is not limited to the proposal-based two-stage convolutional neural network (CNN) target detection framework (i.e., the FasterRCNN series of networks), the pixel-based single-stage CNN target detection framework (i.e., the YOLO series of networks), and the query-based Transformer target detection framework (i.e., the DETR series of networks).

[0040] For example, according to the coordinate point and the spatial feature value [x1, y1, x2, y2], the spatial position of the target box in the image is obtained [xmin, ymin, xmax, ymax], where (xmin, ymin) represents the coordinates of the upper left vertex of the target box, and (xmax, ymax) represents the coordinates of the lower right vertex of the target box.

[0041] For another example, the classification result can be calculated by similarity matching or the like to obtain the category, and the target category can be outputted as the sequence number of the category corresponding to the maximum similarity value.

[0042] In the target detection process of the model to be trained, the first image and the second image are processed in the same way, and both can output target box results and classification results. Therefore, there is no need to set up a separate training process for the second image, which can effectively reduce the workload of training design and improve training efficiency.

[0043] Step S203, determining the target box loss according to the first target box result and the first target box annotation of the first image, the second target box result and the second target box annotation of the second image, and determining the classification loss according to the first classification result and the category-related annotation of the first image.

[0044] The category-independent annotation is used to indicate that the second classification result corresponding to the second image does not participate in the calculation of the classification loss. Since the category-independent annotation of the second category of data is not an accurate annotation result, its classification annotation is not helpful for the classification training of the training model, and the corresponding category-independent annotation is not used to calculate the loss, that is, the calculation loss is ignored.

[0045] The category-independent annotation can be assigned a preset value during construction. The preset value is different from all classification categories of the classification task, so that the classification loss calculated based on the category-independent annotation cannot participate in the composition of the classification loss. For example, the preset value is -1, so as to distinguish the category-related annotation and cannot affect the classification loss when calculating the classification loss.

[0046] Optionally, before determining the target frame loss according to the first target frame result and the first target frame annotation of the first image, and the second target frame result and the second target frame annotation of the second image, the method further includes: When dynamically allocating positive and negative samples for the first image, a first matching score is calculated using the first target box annotation and the corresponding category-related annotation; When dynamically allocating positive and negative samples for the second image, the second target box annotation is used to calculate a second matching score; According to the first matching score and the second matching score, the first image and the second image are divided into positive and negative samples to obtain positive and negative sample division results, and the positive and negative sample division results are used to indicate the calculation of target box loss and classification loss.

[0047] Among them, in the process of calculating the loss, the model is required to distribute positive and negative samples to the data used for training. In this application, whether it is a proposal-based model, a pixel-based model or a query-based model, its corresponding static allocation strategy only depends on the geometry and position relationship between the anchor point and the target box annotation. In algorithms such as SimOTA, TOOD, and Hungarian matching, a one-to-one matching score between the anchor point and the target box annotation is calculated, and the matching score is composed of the positioning loss of the target box and the classification loss of the target box. Due to the existence of the second type of data in this application, the classification loss cannot accurately reflect the matching score. Therefore, the matching score in this application is composed of the target box loss, thereby accurately realizing the positive and negative sample allocation.

[0048] Step S204, adjust the parameters of the model to be trained according to the target box loss and the classification loss, return to execute and input the first image into the model to be trained, obtain the first target box result and the first classification result corresponding to the first image, until the training is completed, and obtain the trained target detection model.

[0049] Among them, during training, the parameter conditions of the target detection model are realized by relying on the target box loss and the classification loss through reverse adjustment and other mechanisms, which can specifically be the parameters of the network backbone in the target detection model. In one embodiment, the target box loss and the classification loss can be calculated by weighted summation to obtain the total training loss, and the total training loss is used to adjust the parameters to achieve iterative training of the model.

[0050] The target detection model may stop iterating and complete training when the number of iterations reaches a preset number, or may stop iterating and complete training when the total training loss converges.

[0051] Optionally, the first type of data further includes a first target evaluation corresponding to the first image, and the second type of data further includes a second target evaluation corresponding to the second image; The target detection training method also includes: Inputting the first image into the model to be trained for target detection to obtain a first target confidence corresponding to the first image, and inputting the second image into the model to be trained for target detection to obtain a second target confidence corresponding to the second image; Determining a target recognition loss based on a first target evaluation and a first target confidence of the first image, and a second target evaluation and a second target confidence of the second image; According to the target box loss and classification loss, adjust the parameters of the model to be trained, including: Adjust the parameters of the model to be trained based on the target recognition loss, target box loss, and classification loss.

[0052] Among them, the first category of data also includes a first target evaluation of the first image, and the second category of data also includes a second target evaluation of the second image. The target evaluation is used to characterize the credibility of the presence or absence of a target, which corresponds to the confidence of the output target of the model processing image.

[0053] For example, the format of the data can be [xmin, ymin, xmax, ymax, cls, confidence], where [xmin, ymin, xmax, ymax] represents the target box, [cls] represents the category, and [confidence] represents the confidence (i.e., target evaluation).

[0054] Based on the above data format, the target recognition loss can be obtained by comparing the target evaluation and the target confidence, which is used to characterize the recognition loss of the model for the presence or absence of the target. Accordingly, when adjusting the parameters of the model, the target recognition loss, the target box loss and the classification loss are needed. In one embodiment, the total loss of the weighted sum of the target recognition loss, the target box loss and the classification loss is used to adjust the model parameters.

[0055] In one embodiment, Varifocal Loss can be used to calculate the category loss between the classification result and the classification label, GIoU Loss can be used to calculate the target frame loss between the target frame result and the target frame label, and the target recognition loss between the target evaluation and the target confidence can be calculated. The embodiment of the present application uses the second category data of the category that does not belong to the classification task of the model to be trained, such as inaccurate classification labels or no specific classification labels, and the first category data with accurate classification to train the model to be trained. During the training process, the loss calculation method of the first category data and the second category data can be automatically distinguished, thereby accurately and effectively realizing the training of the model to be trained. Compared with using only the first category data, the trained target detection model has a more accurate target detection effect, thereby providing a good basis for the classification of the detected target, so as to improve the accuracy of target classification.

[0056] See also Figure 3 , is a flow chart of a target detection training method provided in Example 3 of the present application. Figure 3 As shown, after obtaining the first type of data and the second type of data in the above step S201, the target detection training method may include the following steps: Step S301, performing data augmentation on the first category of data to obtain augmented first category of data.

[0057] Among them, the augmented first category data includes N first images, first target box annotations corresponding to the first images, and category-related annotations corresponding to the first target box annotations, the category-related annotations belong to the category corresponding to the classification task of the model to be trained, and N is an integer greater than 1.

[0058] Step S302: perform data augmentation on the second category data to obtain augmented second category data.

[0059] Among them, the second category of data includes M second images, second target box annotations corresponding to the second images, and category-independent annotations corresponding to the second target box annotations, the category-independent annotations do not belong to the category corresponding to the classification task, and M is an integer greater than 1.

[0060] The above-mentioned augmentation can be performed by random flipping, cropping, color conversion, etc. to obtain more image data to improve the accuracy of the trained model and reduce the difficulty of constructing the training set.

[0061] Optionally, after performing data augmentation on the second category of data to obtain augmented second category of data, the method further includes: The augmented first-category data and the augmented second-category data are concatenated to obtain an augmented training set; Input each image in the augmented training set into the model to be trained in turn.

[0062] Among them, the augmented first-category data and the augmented second-category data are spliced ​​to form a training set, which can be directly fed to the model to be trained, thereby improving the automation level of model training and avoiding manual distinction between the first-category data and the second-category data.

[0063] See also Figure 4 , is a flow chart of a target detection method provided in Example 4 of the present application, and the target detection method is applied to Figure 1 The server in . Figure 4 As shown, the target detection method may include the following steps: Step S401, obtaining an image to be processed.

[0064] Step S402, input the image to be processed into the trained target detection model obtained after executing the above-mentioned target detection training method, and output the target detection result corresponding to the image to be processed, where the target detection result includes the target frame result and / or the classification category of the corresponding target frame result.

[0065] Among them, the above-mentioned target detection method can be switched according to user needs. If the user wants to generate data with only a target box, the corresponding image can be input and only the target box result corresponding to the image can be obtained to realize target detection; if the user wants to generate data with classification results, the corresponding image can be input to obtain the target box result and the classification result of the corresponding target box result. If only the classification result is needed, the classification result can be obtained from the above content.

[0066] For example, if the target detection model is a human body detection model, the purpose of the human body detection model is to detect "male" in image data. Before the human body detection model detects "male", it is necessary to detect the target frame of the human body in the image data. Therefore, the human body detection model performs human body recognition on the image to be processed, that is, it can obtain the target frame of the human body in the image (that is, the frame surrounding the human body), and can also classify the gender category of the target frame, that is, if the target frame surrounds the human body as a "male", then the target frame can be classified and marked as "male".

[0067] For another example, if the target detection model is a human posture detection model, the human posture detection model is used to detect the "standing" posture in the image data. Before the human posture detection model detects the "standing" posture, it is necessary to detect the target frame of the human body in the image data. Therefore, the human posture detection model performs human body recognition on the image to be processed, that is, it can obtain the target frame of the human body in the image, and can also classify the posture category of the target frame, that is, if the target frame surrounds the human body in a "standing" posture, then the target frame can be classified and marked as "standing". The method of the present application is different from the existing methods that can only train class-aware or class-agnostic data separately. It can combine the two types of data for simultaneous training, empowering detection capabilities as well as classification capabilities. Compared with only training class-agnostic, the method of the present application is more functional, and compared with only training class-agnostic, the method of the present application has stronger detection capabilities. In addition, the method of the present application can switch between class-aware and class-agnostic at will according to user needs. When the superclass of the class-agnostic data and the subclass of the class-agnostic data are in a partially contained or fully non-contained relationship, and the output is class-agnostic, it can be directly used for open-set target detection, with a wider range of applications. In addition, the method of the present application can be painlessly embedded in an existing target detection framework, and has strong scalability.

[0068] In an embodiment of the present application, first and second data are obtained, and the classification and annotation accuracies of the first and second data are different. The first image is input into the model to be trained for target detection, and a first target box result and a first classification result corresponding to the first image are obtained. The second image is input into the model to be trained for target detection, and a second target box result and a second classification result corresponding to the second image are obtained. The target box loss is determined according to the first target box result and the first target box annotation of the first image, and the second target box result and the second target box annotation of the second image. The classification loss is determined according to the first classification result of the first image and the category-related annotation. The category-irrelevant annotation is used to indicate that the second classification result corresponding to the second image does not participate in the calculation of the classification loss. According to the target box loss and the classification loss, the parameters of the model to be trained are adjusted, and the execution is returned to input the first image into the model to be trained to obtain the first target box result and the first classification result corresponding to the first image, until the training is completed to obtain a trained target detection model.

[0069] The model to be trained is trained using the second category data that does not belong to the category corresponding to the classification task of the model to be trained and has inaccurate classification labels or no specific classification labels, and the first category data with accurate classification. During the training process, the loss calculation method of the first category data and the second category data can be automatically distinguished, so as to accurately and effectively realize the training of the model to be trained. Compared with using only the first category data, the trained target detection model has a more accurate target detection effect, which can provide a good basis for the classification of the detected targets, so as to improve the accuracy of target classification.

[0070] Corresponding to the target detection training method in the above embodiment, Figure 5 The structure block diagram of the target detection training device provided in the fifth embodiment of the present application is shown. The target detection training device can be used for the above Figure 1 For ease of description, only the parts related to the embodiment of the present application are shown.

[0071] See also Figure 5 , the target detection training device comprises: A first acquisition module 51 is used to acquire first-category data and second-category data, wherein the first-category data includes a first image, a first target frame annotation corresponding to the first image, and a category-related annotation corresponding to the first target frame annotation, the category-related annotation belongs to a category corresponding to the classification task of the model to be trained, and the second-category data includes a second image, a second target frame annotation corresponding to the second image, and a category-independent annotation corresponding to the second target frame annotation, the category-independent annotation does not belong to a category corresponding to the classification task; A first target detection module 52 is used to input the first image into the model to be trained for target detection, obtain a first target frame result and a first classification result corresponding to the first image, and input the second image into the model to be trained for target detection, obtain a second target frame result and a second classification result corresponding to the second image; A first loss calculation module 53 is used to determine the target box loss according to the first target box result and the first target box annotation of the first image, the second target box result and the second target box annotation of the second image, and to determine the classification loss according to the first classification result and the category-related annotation of the first image, and the category-irrelevant annotation is used to indicate that the second classification result corresponding to the second image does not participate in the calculation of the classification loss; The first iterative training module 54 is used to adjust the parameters of the model to be trained according to the target box loss and the classification loss, return to execute the first image input to the model to be trained, obtain the first target box result and the first classification result corresponding to the first image, until the training is completed, and obtain the trained target detection model.

[0072] Optionally, the first type of data further includes a first target evaluation corresponding to the first image, and the second type of data further includes a second target evaluation corresponding to the second image; The target detection training device also includes: A second target detection module is used to input the first image into the model to be trained for target detection, and obtain a first target confidence corresponding to the first image, and input the second image into the model to be trained for target detection, and obtain a second target confidence corresponding to the second image; A second loss calculation module, used to determine the target recognition loss according to the first target evaluation and the first target confidence of the first image, the second target evaluation and the second target confidence of the second image; The first iterative training module 54 comprises: The first iterative training unit is used to adjust the parameters of the model to be trained according to the target recognition loss, the target box loss and the classification loss.

[0073] Optionally, the first acquisition module 51 includes: An image acquisition unit, used to acquire a second image and a second target frame annotation corresponding to the second image; A category labeling unit, used to label the second target frame labeling with a category using a preset value to obtain a category-independent labeling corresponding to the second target frame labeling; The second category data construction unit is used to form the second category data by combining the second image, the second target frame annotation corresponding to the second image, and the category-independent annotation corresponding to the second target frame annotation.

[0074] Optionally, the target detection training device further includes: A first augmentation module is used to perform data augmentation on the first category data after acquiring the first category data and the second category data to obtain augmented first category data, wherein the augmented first category data includes N first images, first target frame annotations corresponding to the first images, and category-related annotations corresponding to the first target frame annotations, wherein the category-related annotations belong to categories corresponding to the classification tasks of the model to be trained, and N is an integer greater than 1; The second augmentation module is used to perform data augmentation on the second category data to obtain augmented second category data, where the second category data includes M second images, second target box annotations corresponding to the second images, and category-independent annotations corresponding to the second target box annotations, where the category-independent annotations do not belong to the category corresponding to the classification task, and M is an integer greater than 1.

[0075] Optionally, the target detection training device further includes: A first matching calculation module is used to calculate a first matching score using the first target box annotation and the corresponding category-related annotation when dynamically allocating positive and negative samples for the first image before determining the target box loss according to the first target box result and the first target box annotation of the first image and the second target box result and the second target box annotation of the second image; A second matching calculation module, used to use the second target box annotation to calculate a second matching score when dynamically allocating positive and negative samples for the second image; The positive and negative sample division module is used to divide the first image and the second image into positive and negative samples according to the first matching score and the second matching score, and obtain the positive and negative sample division results, and the positive and negative sample division results are used to indicate the calculation of the target box loss and the classification loss. It should be noted that the information interaction, execution process and other contents between the above modules are based on the same concept as the embodiment of the method of the present application, and their specific functions and technical effects can be specifically referred to in the method embodiment part, which will not be repeated here.

[0076] Corresponding to the target detection method in the above embodiment, Figure 6 The structure block diagram of the target detection device provided in the sixth embodiment of the present application is shown. The target detection device can be used in the above Figure 1 For ease of description, only the parts related to the embodiment of the present application are shown.

[0077] See also Figure 6 , the target detection device comprises: A second acquisition module 61, used to acquire an image to be processed; The target detection module 62 is used to input the image to be processed into the trained target detection model obtained by executing the above-mentioned target detection training method, and output the target detection result corresponding to the image to be processed. The target detection result includes the target frame result and / or the classification category of the corresponding target frame result.

[0078] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0079] Figure 7 This is a schematic diagram of the structure of a computer device provided in Example 7 of this application. Figure 7 As shown, the computer device of this embodiment includes: at least one processor ( Figure 7 Only one is shown), a memory, and a computer program stored in the memory and executable on a processor, and when at least one processor executes the computer program, steps in any of the above-mentioned target detection training methods or target detection method embodiments are implemented.

[0080] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 7This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0081] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0082] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0083] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0084] The present application implements all or part of the processes in the above-mentioned embodiment method, and may also be completed through a computer program product. When the computer program product runs on a computer device, the steps in the above-mentioned method embodiment can be implemented when the computer device executes it.

[0085] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0086] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0087] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0088] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0089] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A target detection training method, characterized in that: The target detection training method comprises: Acquire first-category data and second-category data, wherein the first-category data includes a first image, a first target frame annotation corresponding to the first image, and a category-related annotation corresponding to the first target frame annotation, wherein the category-related annotation belongs to a category corresponding to a classification task of a model to be trained, and the second-category data includes a second image, a second target frame annotation corresponding to the second image, and a category-independent annotation corresponding to the second target frame annotation, wherein the category-independent annotation does not belong to a category corresponding to the classification task; Input the first image into the model to be trained for target detection, and obtain a first target frame result and a first classification result corresponding to the first image; input the second image into the model to be trained for target detection, and obtain a second target frame result and a second classification result corresponding to the second image; Determine a target box loss according to a first target box result and a first target box annotation of the first image, a second target box result and a second target box annotation of the second image, and determine a classification loss according to a first classification result and a category-related annotation of the first image, wherein the category-independent annotation is used to indicate that the second classification result corresponding to the second image does not participate in calculating the classification loss; According to the target box loss and the classification loss, the parameters of the model to be trained are adjusted, and the step of inputting the first image into the model to be trained is returned to obtain a first target box result and a first classification result corresponding to the first image, until the training is completed to obtain a trained target detection model.

2. The target detection training method according to claim 1, characterized in that: The first type of data also includes a first target evaluation corresponding to the first image, and the second type of data also includes a second target evaluation corresponding to the second image; The target detection training method further includes: Inputting the first image into the model to be trained for target detection to obtain a first target confidence level corresponding to the first image, and inputting the second image into the model to be trained for target detection to obtain a second target confidence level corresponding to the second image; Determining a target recognition loss based on a first target evaluation and a first target confidence of the first image, and a second target evaluation and a second target confidence of the second image; The adjusting the parameters of the model to be trained according to the target box loss and the classification loss includes: According to the target recognition loss, the target box loss and the classification loss, the parameters of the model to be trained are adjusted.

3. The target detection training method according to claim 1, characterized in that: The obtaining of the second type of data includes: Acquire a second image and a second target box annotation corresponding to the second image; Using a preset value, performing category labeling on the second target frame labeling to obtain a category-independent labeling corresponding to the second target frame labeling; The second image, a second target box annotation corresponding to the second image, and a category-independent annotation corresponding to the second target box annotation form a second category of data.

4. The target detection training method according to claim 1, characterized in that: After obtaining the first category data and the second category data, the method further includes: Performing data augmentation on the first category of data to obtain augmented first category data, wherein the augmented first category data includes N first images, first target box annotations corresponding to the first images, and category-related annotations corresponding to the first target box annotations, wherein the category-related annotations belong to categories corresponding to classification tasks of the model to be trained, and N is an integer greater than 1; Data augmentation is performed on the second category data to obtain augmented second category data, where the second category data includes M second images, second target box annotations corresponding to the second images, and category-independent annotations corresponding to the second target box annotations, where the category-independent annotations do not belong to the category corresponding to the classification task, and M is an integer greater than 1.

5. The target detection training method according to any one of claims 1 to 4, characterized in that: Before determining the target frame loss according to the first target frame result and the first target frame annotation of the first image, the second target frame result and the second target frame annotation of the second image, the method further includes: When dynamically allocating positive and negative samples for the first image, using the first target box annotation and the corresponding category-related annotation, a first matching score is calculated; When dynamically allocating positive and negative samples for the second image, the second target box annotation is used to calculate a second matching score; According to the first matching score and the second matching score, the first image and the second image are divided into positive and negative samples to obtain positive and negative sample division results, and the positive and negative sample division results are used to indicate the calculation of target box loss and classification loss.

6. A target detection method, characterized in that: The target detection method comprises: Get the image to be processed; The image to be processed is input into a trained target detection model obtained by executing the target detection training method as described in any one of claims 1 to 5, and a target detection result corresponding to the image to be processed is output, wherein the target detection result includes a target frame result and / or a classification category corresponding to the target frame result.

7. A target detection training device, characterized in that: The target detection training device comprises: A first acquisition module is used to acquire first-category data and second-category data, wherein the first-category data includes a first image, a first target frame annotation corresponding to the first image, and a category-related annotation corresponding to the first target frame annotation, wherein the category-related annotation belongs to a category corresponding to the classification task of the model to be trained, and the second-category data includes a second image, a second target frame annotation corresponding to the second image, and a category-independent annotation corresponding to the second target frame annotation, wherein the category-independent annotation does not belong to a category corresponding to the classification task; A first target detection module is used to input the first image into the model to be trained for target detection, obtain a first target frame result and a first classification result corresponding to the first image, and input the second image into the model to be trained for target detection, obtain a second target frame result and a second classification result corresponding to the second image; a first loss calculation module, configured to determine a target box loss according to a first target box result and a first target box annotation of the first image, a second target box result and a second target box annotation of the second image, and to determine a classification loss according to a first classification result and a category-related annotation of the first image, wherein the category-irrelevant annotation is used to indicate that a second classification result corresponding to the second image does not participate in calculating the classification loss; The first iterative training module is used to adjust the parameters of the model to be trained according to the target box loss and the classification loss, return to execute the step of inputting the first image into the model to be trained, obtain a first target box result and a first classification result corresponding to the first image, until the training is completed to obtain a trained target detection model.

8. A target detection device, characterized in that: The target detection device comprises: A second acquisition module, used for acquiring an image to be processed; A target detection module is used to input the image to be processed into a trained target detection model obtained after executing the target detection training method as described in any one of claims 1 to 5, and output a target detection result corresponding to the image to be processed, wherein the target detection result includes a target frame result and / or a classification category corresponding to the target frame result.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the target detection training method according to any one of claims 1 to 5 or the target detection method according to claim 6 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the target detection training method according to any one of claims 1 to 5 or the target detection method according to claim 6 is implemented.

Images