A semi-supervised object detection method and system

Abstract

A semi-supervised object detection method is provided for detecting defect bounding boxes and defect categories in test sample images. The method includes: acquiring sample images, including defect sample images and test sample images; pre-training a Teacher model and an object detection model based on the labeled defect sample images; and obtaining the bounding boxes and classification results of the Teacher model and the Student model for each defect sample image, and calculating the distance L between the classification results. diff Determine the distance L diff With loss L s The algorithm checks if the sum L converges. If it does, it indicates that the Teacher and Student models have completed training; otherwise, it continues iteratively. The test sample image is input into the Teacher and Student models, and the defect detection bounding boxes and defect categories of the test sample image are output. This invention jointly trains labeled and unlabeled sample data, improving the accuracy of the object detection model.

Description

Technical Field

[0001] This invention belongs to the field of power equipment condition monitoring, and more specifically, relates to a semi-supervised target detection method and system. Background Technology

[0002] In power grid operation scenarios, using target detection technology for defect detection can accelerate defect detection time, reduce manual inspection costs, improve detection rates, and enhance the safe and reliable operation of the power grid.

[0003] Current target detection technologies often require a large number of labeled defect samples. However, labeling detection boxes on image samples is a very time-consuming and labor-intensive process, especially in the power grid field, where annotators need sufficient professional knowledge to label accurate defect detection boxes. Therefore, a large number of defect samples require a lengthy labeling process to obtain relatively accurate annotations. The annotation includes the defect detection box and the defect category to which the detection box belongs. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to quickly incorporate defect samples captured on-site into model training, reduce manual annotation costs, and improve the detection rate and accuracy of the model. Therefore, a semi-supervised target detection method is proposed.

[0005] The present invention adopts the following technical solution.

[0006] A semi-supervised object detection method for detecting defect detection boxes and defect categories in a sample image includes the following steps:

[0007] Step S1: Obtain sample images, including: defect sample images and sample images to be detected, wherein the defect sample images include labeled defect sample images and unlabeled defect sample images; wherein the labels include: defect detection boxes and defect categories;

[0008] Step S2: Based on the labeled defect sample images, complete the pre-training of the Teacher model and the object detection model;

[0009] Step S3: Based on the target detection model and the Student model, obtain the detection bounding box and classification result z of the Teacher model corresponding to each defect sample image. t The detection bounding box and classification result z of the Student model s ; where z t Includes: Teacher's model detection bounding box results Compared with classification results z s Includes: Detection box results of the Student model Compared with classification results

[0010] Step S4, based on the detection bounding box and classification result z of the Teacher model t The detection bounding box and classification result z of the Student model s Calculate the distance L between the classification results. diff ;

[0011] Step S5, determine the distance L diff With loss L s Check if the sum L converges. If it does, it means that the Teacher model and Student model have been trained. Otherwise, return to step S1 to continue.

[0012] Step S6: Input the sample image to be tested into the Teacher model and the Student model, and output the defect detection box and defect category of the sample image to be tested.

[0013] Furthermore, step S2 specifically includes steps S21 to S24;

[0014] Step S21: Using the Teacher model, extract the features of the defect sample image as the first image features;

[0015] Step S22: Select the first image features of the labeled defect sample image, and predict the defect detection box and defect category according to the target detection model;

[0016] Step S23: Calculate the loss L of the labeled data based on the predicted defect detection box, defect category, and label. s Loss L s This includes: bounding box regression loss and classification loss;

[0017] Step S24, based on the loss L s This involves correcting the internal parameters of the Teacher model and the object detection model, thereby completing the pre-training of the Teacher model and the object detection model.

[0018] Furthermore, step S3 specifically includes steps S31 to S33;

[0019] Step S31: Using the Student model, extract the features of the defect sample image as the second image features;

[0020] Step S32: Map the second image features according to the preset mapping function f;

[0021] Step S33: Input the first image features and the mapped second image features into the object detection model, and output the detection bounding box and classification result z of the Teacher model respectively.t The detection bounding box and classification result z of the Student model s ; where z t Includes: Teacher's model detection bounding box results Compared with classification results z s Includes: Detection box results of the Student model Compared with classification results

[0022] Furthermore, the mapping function is used to prevent the parameters in the Scaled-YOLO model from being corrupted.

[0023] Furthermore, the target detection model is the Scaled-YOLO model.

[0024] Furthermore, the distance L between the classification results in step S5 diff for:

[0025]

[0026] Where d represents the Euclidean distance.

[0027] Furthermore, in step S6, the distance L is determined. diff With loss L s Whether the sum L converges is specifically determined by:

[0028] If the absolute value of the difference between the previous iteration's L and the current iteration's L is less than a preset fixed value, then the iteration is considered convergent; otherwise, it is considered non-convergent.

[0029] A semi-supervised object detection system includes: a sample acquisition module, a pre-training module, and a semi-supervised object detection module;

[0030] The sample acquisition module is used to acquire defect sample images;

[0031] The pre-training module is used to pre-train the Teacher model and the object detection model.

[0032] The semi-supervised object detection module is used to extract the second image features of defect sample images, and output the detection bounding boxes and classification results z of the Teacher model. t The detection bounding box and classification result z of the Student model s Calculate the distance L between the classification results. diff And determine the distance L diff With loss L s Does the sum L converge?

[0033] Furthermore, the pre-training module includes: a feature extraction network, a neural learning network, and a logic computation module;

[0034] The feature extraction network is used to extract the first image features from defective sample images.

[0035] Neural learning networks are used to predict defect detection boxes and defect categories;

[0036] The logic calculation module is used to calculate the loss L of labeled data. s And to correct the internal parameters of the Teacher model and the object detection model.

[0037] Furthermore, the semi-supervised object detection module includes: a feature extraction network, an object detection module, and a logical judgment calculation module;

[0038] The feature extraction network is used to extract second image features from defect sample images;

[0039] The object detection module is used to output the detection bounding boxes and classification results z of the Teacher model. t The detection bounding box and classification result z of the Student model s ;

[0040] The logical judgment calculation module is used to calculate the distance L between classification results. diff And determine the distance L diff With loss L s Does the sum L converge?

[0041] The beneficial effects of the present invention are as follows: Compared with the prior art, the present invention has the following advantages:

[0042] (1) A semi-supervised object detection model based on the Teacher-Student model is proposed. This model is jointly trained with labeled and unlabeled sample data to improve the accuracy of the object detection model.

[0043] (2) Use the same Scaled-YOLO model to measure the distance between the Student model and the Teacher model, so that the Student model can learn the knowledge of the Teacher model. Attached Figure Description

[0044] Figure 1 This is an architecture diagram of a semi-supervised object detection method based on Teacher-Student.

[0045] Figure 2 This is a structural diagram of the pre-trained module.

[0046] Figure 3 This is a flowchart of a semi-supervised target detection method. Detailed Implementation

[0047] The present application will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention, and should not be construed as limiting the scope of protection of the present application.

[0048] A semi-supervised object detection method is provided, wherein the object to be detected is an unlabeled sample image, and the final output of the method is the defect detection box and defect category corresponding to the sample image. Figure 1 As shown, it includes the following steps:

[0049] Step S1, acquire sample images, including: defect sample images and sample images to be detected, wherein the defect sample images include labeled defect sample images (i.e., Figure 1 Labeled images (in the image) and unlabeled defect sample images (i.e.) Figure 1 (The unlabeled image in the image); where the labels include: defect detection bounding box and defect category;

[0050] Step S2: Based on the labeled defect sample images, complete the pre-training of the Teacher model and the object detection model;

[0051] Step S21: Extract the features of the defect sample image as the first image feature through the Teacher model; the Teacher model contains several layers of neural network, and the first image feature is a 2048-dimensional vector.

[0052] Step S22: Select the first image features of the labeled defect sample image, and predict the defect detection box and defect category through the target detection model;

[0053] The first image features are input into the Scaled-YOLO model to predict the detection boxes and categories. The prediction process can be seen as several neural networks performing transformations. Finally, a specified number of detection boxes are output. Each detection box has a four-dimensional location coordinate (coordinates in the image) and a classification score vector (the classification result of the detection box corresponding to the category with the highest score in the vector).

[0054] Furthermore, the object detection model can be a Scaled-YOLO model, such as YOLOv1, YOLOv2, etc. The following text will use the Scaled-YOLO model for illustration.

[0055] Step S23: Calculate the loss L of the labeled data based on the predicted defect detection box, defect category, and label. s Loss L s This includes: bounding box regression loss and classification loss;

[0056] Understandably, the labels contain actual (manually labeled) defect detection boxes and defect categories. Therefore, the detection box regression loss is calculated using the actual defect detection boxes and the predicted defect detection boxes; the classification loss is calculated using the predicted defect categories and the actual defect categories.

[0057] Loss L s This is a general formula used in Scaled-YOLO, widely applied in deep learning, and needs no further explanation. It's understandable that regression loss calculates the distance between the model's predicted bounding boxes and the human-annotated bounding boxes, while classification loss calculates the difference between the model's predicted classification and the actual human-annotated categories. There are also some other details involved in calculating the losses.

[0058] Step S24, based on the loss L s The internal parameters of the Teacher model and the object detection model are corrected, thereby completing the pre-training of the Teacher model and the Scaled-YOLO model.

[0059] It should be noted that the so-called "pre-trained Teacher model and Scaled-YOLO model" means "optimizing the parameters of the Teacher model and Scaled-YOLO model".

[0060] Understandably, steps S2 to S5 complete the pre-training process of the entire algorithm.

[0061] Step S3: Based on the target detection model and the Student model, obtain the detection bounding box and classification result z of the Teacher model corresponding to each defect sample image. t The detection bounding box and classification result z of the Student model s ;

[0062] Step S31: Using the Student model, extract the features of the defect sample image as the second image features;

[0063] Step S32: Map the features of the second image using function f; function f is any small model containing several layers of neural networks.

[0064] It should be noted that the second image features are not directly input into the Scaled-YOLO model, but rather need to be mapped to prevent the parameters of the Scaled-YOLO model from being corrupted. Understandably, the function f here can be any non-linear mapping function, its purpose being to prevent the data in the original "second image features" from corrupting the parameters of the Scaled-YOLO model. In some embodiments, the function f(x) can be as follows:

[0065]

[0066] Where N is a natural number, depending on the complexity of the Scaled-YOLO model, and N can usually be 1.

[0067] Step S33: Input the first image features and the mapped second image features into the Scaled-YOLO model to output (or predict) the detection boxes and classification results z of the Teacher model, respectively. t The detection bounding box and classification result z of the Student model s ; where z t Includes: Teacher's model detection bounding box results Compared with classification results z s Includes: Detection box results of the Student model Compared with classification results

[0068] Understandably, the prediction process described above can be viewed as several neural networks in a Scaled-YOLO model performing transformations, ultimately outputting a specified number of detection boxes. Each detection box has four-dimensional location coordinates (coordinates in the image) as the detection result (e.g.: and The classification result is a classification score vector (the classification result of the bounding box corresponding to the category with the highest score in the vector) and a classification score vector (e.g., the classification result of the bounding box corresponding to the category with the highest score in the vector). and It should be noted that, Both are vectors. For example It can be [21.2, 34.3, 66.3, 22.5], representing the coordinates of the center point of the detection box and the width and height of the detection box, respectively. It can be [0.1, 0.3, 0.6], which represents the probability of being classified into each of the three categories.

[0069] Step S4, based on the detection bounding box and classification result z of the Teacher model t The detection bounding box and classification result z of the Student model s Calculate the distance L between the classification results. diff This guides the Student model to learn the structured information extracted by the Teacher model.

[0070] Step S5, determine the distance L diff With loss L sIf the sum L converges, it indicates that the Teacher-Student model (i.e., the Teacher model and the Student model) has been trained and the iteration stops; otherwise, return to step S1 to continue execution (iteration) and let C = C + 1.

[0071] Furthermore, "determine distance L" aiff With loss L s The specific process of "whether the sum L converges" is as follows:

[0072] If the absolute value of the difference between L in the previous iteration (i.e., the C-1th iteration) and L in the current iteration is less than a preset fixed value, then it is considered convergent; otherwise, it is considered non-convergent.

[0073] Step S6: Input the sample image to be tested into the Teacher model and the Student model, and output the defect detection box b of the sample image to be tested. z With defect category c z Specifically, the category corresponding to the maximum value among the defect categories of the output test sample image is taken as the defect category of the test sample image.

[0074] In summary, this invention cleverly uses distance L diff The Student and Teacher models are learned (or trained) using structured information (i.e., the parameters of the Student and Teacher models are optimized) to ultimately obtain a Teacher-Student model that can correctly classify defective sample images.

[0075] Accordingly, the present invention also discloses a semi-supervised target detection system, which includes the following modules:

[0076] 1. Sample Acquisition Module

[0077] Used to obtain defect sample images.

[0078] 2. Pre-training module

[0079] The architecture diagram of the pre-training module is as follows: Figure 2 As shown, this patent first uses labeled defect sample images to pre-train a Teacher model. Specifically, the Teacher model is used to extract the first image features (i.e., Figure 1 x in t Then, the Scaled-YOLO model is used to predict the defect detection box and category, and the loss L is calculated by combining the image label, i.e., the ground truth. s The loss includes the bounding box regression loss and the classification loss.

[0080] Furthermore, the pre-training module includes: a feature extraction network, a neural learning network, and a logic computation module;

[0081] The feature extraction network is used to extract the first image features from defective sample images.

[0082] Neural learning networks are used to predict defect detection boxes and defect categories;

[0083] The logic calculation module is used to calculate the loss L of labeled data. s And to correct the internal parameters of the Teacher model and the object detection model.

[0084] 3. Semi-supervised target detection module

[0085] The architecture diagram of the semi-supervised object detection module is as follows: Figure 1 As shown, the Student model and the Teacher model share the parameters of a pre-trained Scaled-YOLO model. It's important to note that the parameters of the Scaled-YOLO model refer to the parameters of the pre-trained model, which contains many network layers, each with numerous parameters. Because traditional Teacher-Student models do not focus on learning the structured information of the pre-trained Teacher model when training the Student model, their generalization ability is poor. To address this issue, this patent utilizes the same object detection model (i.e., the Scaled-YOLO model) to measure the difference between the Teacher and Student models, guiding the Student model to learn the structured information extracted by the Teacher model.

[0086] This invention trains the model using both labeled and unlabeled defect image samples. For labeled data, the Student model is first used to learn the classification loss and bounding box regression loss L mentioned above. s Then, in order to learn the structured information of the first image features from the Teacher model, this patent uses a function f to process the second image features extracted by the Student model (i.e., Figure 1 x in s After mapping (i.e., becoming) Figure 1 x′ in s The features extracted by the teacher are fed into a pre-trained Scaled-YOLO model, which outputs the corresponding detection boxes and classification results, respectively. s and z tHere, function f is a small model containing several layers of neural networks. "Mapping" refers to taking the second image features as input, processing them through the Student model, and outputting a value with a different dimension as input to the Scaled-YOLO model, rather than directly using the second image features as input to the Scaled-YOLO model.

[0087] Then, the Euclidean distance is used to measure z. s The detection box and z t The distance to the detection box, and z s Classification results and z t The distance between the classification results is L, and the sum of the two distances is L. diff As shown in the following formula:

[0088]

[0089] Where d represents the Euclidean distance. They represent z respectively s Detection results in the middle frame, z t Detection results in the middle frame, z s Classification results and z t The classification results are as follows. The difference between unlabeled and labeled data is that unlabeled data lacks ground truth, therefore L cannot be calculated. s Loss. The model is iteratively trained using randomly selected labeled and unlabeled sample data. During testing, defective sample images are directly input into the Student model to extract second image features, and then the corresponding Scaled-YOLO is used to extract detection boxes and classify them.

[0090] Typically, defect sample images are unlabeled. This invention makes reasonable use of the massive amount of unlabeled defect sample images, combined with a small number of labeled defect sample images, and creatively employs the Scaled-YOLO model to calculate L. diff Complete the construction of the Teacher-Student model.

[0091] Furthermore, the semi-supervised object detection module includes: a feature extraction network, an object detection module, and a logical judgment calculation module;

[0092] The feature extraction network is used to extract second image features from defect sample images;

[0093] The object detection module is used to output the detection bounding boxes and classification results z of the Teacher model. t The detection bounding box and classification result z of the Student model s ;

[0094] The logical judgment calculation module is used to calculate the distance L between classification results. diff And determine the distance L diff With loss L s Does the sum L converge?

[0095] It should be noted that in this invention, "detection box" and "defect detection box" have the same meaning. The detection box refers to marking a defect in an image with a long box, that is, finding the location of the defect in the image. This is also the ultimate goal of this patent. Similarly, in this invention, "annotation" and "label" have the same meaning. "Annotation" should be understood as marking with a "label." Because "annotation" is more of a verb and "label" is more of a noun, they are used interchangeably due to grammatical conventions.

[0096] In summary, this patent utilizes the popular Teacher-Student model in the semi-supervised learning field. This model consists of a Student and a Teacher, with the Teacher guiding the Student to learn "knowledge" from data. In real-world scenarios, capturing sufficient sample images for each category and expending significant resources to label them with bounding boxes and categories is extremely difficult, and the number of samples for different categories can vary greatly. Therefore, learning the structured information of image features from samples is beneficial for the model to maintain a certain level of detection accuracy and classification accuracy across datasets with varying distributions. This patent trains the model using a large number of defective image samples without labeled bounding boxes and categories, thereby improving the model's detection accuracy. Furthermore, this patent utilizes the Scaled-YOLO object detection model to complete the object detection task.

[0097] The applicant of this invention has provided a detailed description of the embodiments of the invention in conjunction with the accompanying drawings. However, those skilled in the art should understand that the above embodiments are merely preferred embodiments of the invention. The detailed description is only intended to help readers better understand the spirit of the invention and is not intended to limit the scope of protection of the invention. On the contrary, any improvements or modifications made based on the inventive spirit of the invention should fall within the scope of protection of the invention.

Claims

1. A semi-supervised target detection method for detecting defect detection boxes and defect categories in a sample image, characterized in that, Includes the following steps: Step S1: Obtain sample images, including: defect sample images and sample images to be detected, wherein the defect sample images include labeled defect sample images and unlabeled defect sample images; wherein the labels include: defect detection boxes and defect categories; Step S2: Based on the labeled defect sample images, complete the pre-training of the Teacher model and the object detection model; Step S2 specifically includes steps S21 to S24. Step S21: Using the Teacher model, extract the features of the defect sample image as the first image features; Step S22: Select the first image features of the labeled defect sample image, and predict the defect detection box and defect category according to the target detection model; Step S23: Calculate the loss of labeled data based on the predicted defect detection box, defect category, and label. ; Step S24, based on the loss The internal parameters of the Teacher model and the object detection model are corrected, thereby completing the pre-training of the Teacher model and the object detection model; Step S3: Based on the target detection model and the Student model, obtain the detection bounding box and classification result of the Teacher model corresponding to each defect sample image. The detection bounding boxes and classification results of the Student model Step S3 specifically includes steps S31 to S33. Step S31: Using the Student model, extract the features of the defect sample image as the second image features; Step S32: Map the second image features according to the preset mapping function f; Step S33: Input the first image features and the mapped second image features into the object detection model, and output the detection bounding box and classification result of the Teacher model respectively. The detection bounding boxes and classification results of the Student model ;in, Includes: Teacher's model detection bounding box results Compared with classification results , Includes: Detection box results of the Student model Compared with classification results ; Step S4: Based on the detection bounding box and classification results of the Teacher model The detection bounding boxes and classification results of the Student model Calculate the distance between classification results ; Step S5, determine the distance With loss sum Whether it converges, the loss This includes: bounding box regression loss and classification loss. If convergence is achieved, it indicates that the Teacher model and Student model have been trained successfully; otherwise, return to step S1 to continue. Step S6: Input the sample image to be tested into the Teacher model and the Student model, and output the defect detection box and defect category of the sample image to be tested.

2. The semi-supervised target detection method according to claim 1, characterized in that, Mapping functions are used to prevent parameters in Scaled-YOLO models from being corrupted.

3. The semi-supervised target detection method according to claim 1, characterized in that, The target detection model is the Scaled-YOLO model.

4. The semi-supervised target detection method according to claim 1, characterized in that, Distance between classification results in step S5 for: in, This represents the Euclidean distance.

5. The semi-supervised target detection method according to claim 1, characterized in that, In step S6, the distance is determined. With loss sum Whether it has converged depends on the specific circumstances: Get the previous iteration If it is related to the current iteration If the absolute value of the difference is less than a preset fixed value, it is considered convergent; otherwise, it is considered non-convergent.

6. A semi-supervised target detection system, used to perform the method according to any one of claims 1-5, characterized in that, The system includes: a sample acquisition module, a pre-training module, and a semi-supervised object detection module; The sample acquisition module is used to acquire defect sample images; The pre-training module is used to pre-train the Teacher model and the object detection model; the pre-training module includes: a feature extraction network, a neural learning network, and a logic computation module; The feature extraction network is used to extract the first image features from defective sample images; Neural learning networks are used to predict defect detection boxes and defect categories; The logic calculation module is used to calculate the loss of labeled data. And correct the internal parameters of the Teacher model and the object detection model; The semi-supervised object detection module is used to extract the second image features of defect sample images, and output the detection boxes and classification results of the Teacher model. The detection bounding boxes and classification results of the Student model Calculate the distance between classification results and determine the distance With loss sum Whether it converges; loss This includes: bounding box regression loss and classification loss; the semi-supervised object detection module includes: a feature extraction network, an object detection module, and a logical judgment calculation module; The feature extraction network is used to extract second image features from defect sample images; The object detection module is used to output the detection bounding boxes and classification results of the Teacher model. The detection bounding boxes and classification results of the Student model ; The logical judgment calculation module is used to calculate the distance between classification results. and determine the distance With loss sum Whether it converges.

Images