A semi-supervised open-set object detection method based on label fusion

Abstract

The present invention discloses a semi-supervised open-set target detection method based on label fusion. The method first intercepts the foreground part and the corresponding annotation in the labeled data image, constructs a foreground information library and dynamically enhances it, and fuses it with the unlabeled data to obtain synthetic data. Secondly, the labeled data is simultaneously sent to two teacher networks for training, namely the original teacher network and the detector. The two teacher networks are then used to predict the unlabeled data and the newly generated synthetic data, and the predicted labels are fused to obtain the final pseudo-labels. Finally, the labeled data, the unlabeled data with pseudo-labels, and the synthetic data are sent to the semi-supervised framework, the target detection results are output, and training is performed. The present invention is a semi-supervised target detection method that processes open set problems and alleviates category imbalance problems based on the construction of a foreground information library, thereby alleviating common category imbalance problems.

Description

Technical Field

[0001] The present invention belongs to the field of computer-aided image processing, and in particular relates to a semi-supervised open-set target detection method based on label fusion. Background Art

[0002] Semi-supervised techniques are widely used in the field of medical imaging because they can effectively utilize unlabeled data for training. However, real-world data is often imbalanced and contains out-of-distribution (OOD) classes. This phenomenon is particularly prevalent in medical image datasets, where the number of common and rare diseases often differs significantly, and the presence of unknown diseases is also common. In the field of natural images, little research has been devoted to addressing the unknown class problem under semi-supervised conditions, and in the field of medical imaging, no such research has even begun. Existing methods primarily assume that unlabeled and test data do not contain OOD classes.

[0003] Existing open-set semi-supervised object detection methods applied to natural images have two shortcomings: first, they do not consider the class imbalance situation; second, OOD instances are identified during pseudo-labeling and simply discarded. Summary of the Invention

[0004] In response to the shortcomings of the existing technology and the demand for unknown category detection in the field of medical images, this paper proposes a semi-supervised open-set object detection method based on label fusion. Consider the open-set semi-supervised object detection problem, which utilizes unlabeled data containing OOD classes to improve object detection in medical images. The research of this paper combines two key innovations: Category Controlled Embedding (CCE) and Out-of-Distribution Detection Fusion Classifier (OODFC). CCE aims to address the dataset imbalance problem by building a foreground information library, while OODFC solves the open-set problem by integrating "unknown" information into the basic pseudo-labels.

[0005] To achieve the above object, the present invention comprises the following steps:

[0006] S1. According to the annotation information, the foreground part and the corresponding annotation in the labeled data image are cut out to construct a foreground information library, and dynamic enhancement is performed.

[0007] S2. Fusing the content in the dynamically enhanced foreground information library with the unlabeled data to obtain newly generated synthetic data.

[0008] S3. The labeled data is fed into two teacher networks for training at the same time, where one teacher network is the original teacher network and the other is an additional detector OpenDET for detecting unknown categories.

[0009] S4. Use the two trained teacher networks to predict the unlabeled data and the newly generated synthetic data, generating two pseudo-labels. The pseudo-labels generated by the original teacher network are fused with the unknown category information in the pseudo-labels generated by OpenDET to complete the label fusion and obtain the final pseudo-labels.

[0010] S5: Feed the labeled data, pseudo-labeled unlabeled data, and synthetic data into the semi-supervised framework, output object detection results, and conduct training. Training is terminated when the model reaches the set number of iterations. The optimal evaluation metric on the validation set is used as the result of that round of training. Repeat the training for five rounds and take the average value to eliminate random errors.

[0011] The original teacher network is a fully supervised FCOS network with a ResNet50 backbone, and the unknown category detection teacher network is a fully supervised OpenDET network. The entire semi-supervised framework is based on a pseudo-labeling method and is continuously updated via EMA.

[0012] Furthermore, in the above-mentioned semi-supervised target detection method based on label fusion for processing the open set problem and alleviating the category imbalance problem based on building a foreground information database, step S1 specifically includes the following steps:

[0013] S11. The foreground portion of the image with the labeled data portion is cut out together with its corresponding foreground information label.

[0014] S12. In a data set with imbalanced categories, the amount of foreground information of different categories varies greatly. During the enhancement process, the amount of foreground information of each category is deliberately controlled so that the number of foreground information of each category in the final enhanced information database is consistent.

[0015] Furthermore, in the above-mentioned semi-supervised target detection method based on label fusion for processing the open set problem and alleviating the category imbalance problem by building a foreground information database, step S2 specifically includes the following steps:

[0016] S21. Randomly select a foreground information image and corresponding label information from the foreground information library that has completed dynamic enhancement, and fuse them with the unlabeled data.

[0017] S22. During fusion, foreground information is added to random areas of the unlabeled data, and the two are fused at a ratio between 0 and 1 to avoid information loss of the covered image caused by direct coverage.

[0018] Furthermore, in the above-mentioned semi-supervised target detection method based on label fusion for processing the open set problem and alleviating the category imbalance problem by building a foreground information database, step S3 specifically includes the following steps:

[0019] S31. The labeled part of the original data is sent to two teacher networks for training at the same time to produce pseudo labels for unlabeled data.

[0020] S32. The original teacher network can only generate labels containing known categories, while some of the targets in the labels generated by OpenDET will be classified as unknown categories.

[0021] Furthermore, in the above-mentioned semi-supervised target detection method based on label fusion for processing the open set problem and alleviating the category imbalance problem based on building a foreground information database, step S4 specifically includes the following steps:

[0022] S41, merge all the contents in label 1 generated by the original teacher network and the 'unknown' label in label 2 generated by opendet.

[0023] S42. When two models make conflicting predictions, filter and choose whether to retain the 'unknown' category label information.

[0024] Furthermore, in the above-mentioned semi-supervised target detection method based on label fusion for processing the open set problem and alleviating the class imbalance problem by building a foreground information database, step S5 specifically includes the following steps:

[0025] S51. The data fed into the main network includes labeled data, unlabeled data, and synthetic data. An IoU loss function is calculated between the network output and the labels. The loss function used by the network is Focal Loss.

[0026] S52. When the model training reaches the set number of iterations, the training is stopped. During this period, the model parameters are saved based on the optimal evaluation index results of the validation set, where the optimal evaluation index of the validation set is defined as the mAP for the prediction of the disease type.

[0027] Beneficial effects of the present invention:

[0028] The present invention's semi-supervised target detection method, which uses label fusion to address open set problems and builds a foreground information library to alleviate class imbalance, is the first semi-supervised target detection method applied to the field of medical imaging to address open set problems. It also alleviates the common class imbalance problem in the field. First, the imbalanced foreground information in labeled data is innovatively selectively enhanced to construct a balanced foreground information library for fusion with unlabeled data, resulting in synthetic data containing relatively balanced foreground information. Second, an additional detector specifically designed to identify unknown categories is introduced to produce new pseudo-labels containing unknown category information. The pseudo-labels generated by the two teacher networks are fused, so that the pseudo-labels ultimately used for training contain some unknown category information, which serves as a hint to the model to alleviate the impact of the presence of unknown categories on model judgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is the overall flow chart of the present invention;

[0030] Figure 2 This is a schematic diagram of the structure of the module CCE applied to the present invention;

[0031] Figure 3 This is a structural diagram of the OODFC module applied to the present invention. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings:

[0033] like Figure 1 As shown, the overall network architecture consists of a teacher network and a student network. Unlike traditional teacher-student networks, the network here is updated using the exponential moving average (EMA) method. The OODFC module works in conjunction with the original teacher network to generate additional pseudo-labels containing unknown category information in addition to the original pseudo-labels. These two pseudo-labels are fused to form a new hybrid label. Foreground information from the labeled data is extracted and processed in the CCE module to form a foreground information database. This dynamically processed foreground information is fused with unlabeled data to generate new synthetic data. Unlabeled data, labeled data, and synthetic data generated by the CCE module are all fed into the network for training. During training, the student network is continuously optimized using EMA to serve as the teacher network. The new teacher network makes new predictions on the unlabeled data, which serve as new, more accurate pseudo-labels for subsequent training. The model output is used to calculate supervised and unsupervised losses. The supervised loss is calculated by comparing the model output with the label information, while the unsupervised loss is calculated by comparing the model output with the pseudo-label information.

[0034] The embodiment of the present invention, as Figure 1 As shown, the following steps are included:

[0035] S1. Preprocess the image data. Since the network's input image size must be fixed, the original images of varying sizes are cropped and scaled to a uniform size of 1000 × 480. The image is then patch-shuffled to randomly rearrange local regions of the image. Shuffling methods include flip and flop.

[0036] S2. Input the image into the FCOS network and perform feature extraction first.

[0037] S2.1. The data is fed into an RLA_ResNet backbone network with the number of residual blocks being [3, 4, 6, 3]. This part extracts features from the image and outputs a series of feature maps with different resolutions and semantic levels for subsequent processing.

[0038] S2.2. The extracted features from the previous step are then fed into a neck network of the FPN type to further process the features extracted by the backbone network and generate a multi-scale feature map pyramid. The input channel number list of this neck is [256, 512, 1024, 2048], the output channel number is 256, and the final output number of feature maps is 5.

[0039] S3. After obtaining the feature map, we need to predict the position and category of objects in the image based on the feature map, and we will input it into a detection head FCOSHead.

[0040] The number of input channels for this header is set to 256, and the loss function weight parameters loss_weight, soft_weight, and soft_warm_up are defined as 2.0, 1.0, and 0, respectively. This concludes the process of feeding labeled images into the original teacher network for pre-trained fully supervised training. After training, the FCOS network predicts the unlabeled data and generates the first pseudo-labels.

[0041] S4, such as Figure 2 As shown, the CCE module is implemented.

[0042] S4.1. Extract the part of the marked foreground frame in the marked image together with its annotation information to form a new set foreground information library.

[0043] S4.2. Count the frequency of each category f c, in the defined balanced foreground information base, the target frequency of each category is set to the average of the frequencies of each category:

[0044]

[0045] When dynamically enhancing the information database, C is the number of categories in the data set, and the number of each category will be controlled based on this goal so that the number of foreground images in each category in the final foreground information database is the same.

[0046] S4.3. The foreground information in the database that has completed dynamic enhancement and foreground information equalization is added to the unlabeled data. The foreground information fragments are added to random locations of the unlabeled image. The overlapping areas of the two images are fused at a fusion rate of β, where β is a number between 0 and 1 that adds foreground information while preserving the information of the added image:

[0047]

[0048] x i is the unlabeled image with foreground information added, b′ j It is the foreground information fragment after enhancement. It is a new picture generated by fusing two pictures; i is the annotation of the unlabeled image, y j It is the annotation of foreground information fragments. It is the annotation of the newly generated image after fusion.

[0049] After the addition is completed, a new synthetic data will be generated, corresponding to the empty label of the unlabeled data. The annotation information of this synthetic data contains the added foreground information.

[0050] S5. Implement the OODFC module

[0051] S5.1, such as Figure 3 As shown in Figure 2, similar to the way data is fed into the FCOS network for training in S2, OODFC introduces a new teacher network, Opendet. Labeled data is also fed into the Opendet network for training.

[0052] S5.2. After training, OpenDET will also make predictions for unlabeled data and give a second set of pseudo-labels. This pseudo-label will use the 'unknown' category that is not in the first set of labels to mark targets that the model believes to be of unknown categories.

[0053] S5.3. Fuse the information from the two labels by fusing the known category information from the first label with the 'unknown' information from the second label. Some areas have different predictions from the first and second labels, so filtering is required. The remaining unknown labels after filtering are added directly to the first label. When two areas in the label with iou > 0.7 give different results, the following formula is used for filtering:

[0054]

[0055] Where T i is the threshold used to determine whether to retain the 'unknown' category, i is the category, and γ is a hyperparameter used to control the growth of the function. It is set to 1.5 in the experiment. This threshold is affected by the initial teacher network's prediction accuracy of each category target AP i When the judgment accuracy of the initial teacher network is very high, it means that the prediction result of the first label is more likely to be correct. At this time, the threshold T i It will become higher, and the 'unknown' target box information in the second label will be more difficult to be preserved. The exponential function is used because AP i It is easy to approach 1. The characteristics of the exponential function and concave function can be seen in AP i When it is close to 1, the discrimination is better.

[0056] Example:

[0057] 1.1 Dataset Division:

[0058] Slit Lamp Ophthalmology Dataset: This private slit lamp ophthalmology dataset contains 11 known classes and three unknown classes: "conjunctival hyperemia," "keratoconjunctival tumor," and "keratitis." These three classes are completely excluded from the training process and appear only in the validation set. From the labeled data for the 11 known classes, 10% of each class was selected as labeled training data. This data, along with the unlabeled training data, forms the semi-supervised training set. The remaining 90% of the labeled data and data from all three unknown classes serve as the test set.

[0059] This private dataset includes 136 labeled training images, 1224 unlabeled training images, and 1782 validation images, using the classic training set split of 10% labeled data and 90% unlabeled data.

[0060] Parasite Dataset: Similarly, the public parasite dataset contains six known classes and two unknown parasite classes: tapeworms and whipworms. These two unknown classes are not included in the training process and are used only for performance evaluation. From the six known classes, 10% of each class is used as labeled training data. This data, along with the unlabeled data, forms the semi-supervised training set. The original validation set of this dataset, along with all data from the two unknown classes, serves as the test set.

[0061] The public parasite dataset consists of 152 labeled training images, 1,332 unlabeled training images, and 411 validation images. The labeled and unlabeled data in the training set also account for 10% and 90% respectively.

[0062] 1.2 Evaluation method:

[0063] The average precision (AP) for each category and the mean average precision (mAP) for all categories are reported, following the convention of object detection. AP (average precision) measures the accuracy of the model's predictions in object detection by considering both precision and recall. It is calculated as the area under the precision-recall curve. mAP (mean average precision) is the average of the AP scores for all categories in the dataset, providing a single metric for evaluating the overall model performance across multiple categories. In our experiments, AP50 is used as the evaluation metric, which means measuring the average precision with an intersection-over-union (IoU) threshold of 50%. This indicates that if the predicted bounding box overlaps with the true bounding box by at least 50%, it is considered a positive sample.

[0064] 2. Experimental results:

[0065] The experimental results are shown in Table 1 and Table 2. It can be clearly seen that the present invention has obvious effects. Due to other comparative experiments, the semi-supervised target detection method of the present invention alleviates the category imbalance problem, and at the same time, it also alleviates the category imbalance problem commonly found in the image field.

[0066] Table 1 Slit lamp ophthalmology dataset

[0067] method Average mAP of five repeated experiments FCOS (2019) 0.4609 Opendet (CVPR, 2022) 0.4751 DSL (CVPR, 2022) 0.5029 Consistent-Teacher (CVPR, 2023) 0.4814 The present invention 0.5307

[0068] Table 2 Parasite dataset

[0069] method Average mAP of five repeated experiments FCOS (2019) 0.4725 Opendet (CVPR, 2022) 0.4813 DSL (CVPR, 2022) 0.5265 Consistent-Teacher (CVPR, 2023) 0.5310 The present invention 0.5712

[0070] The embodiments of the present invention described above do not limit the scope of protection of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the scope of protection of the claims of the present invention.

Claims

1. A semi-supervised open-set target detection method based on label fusion, characterized in that: The following steps are involved: S1. Based on the annotation information, the foreground part and the corresponding annotation in the labeled data image are extracted to construct a foreground information library, and dynamic enhancement is performed; S2, fusing the content in the dynamically enhanced foreground information library with the unlabeled data to obtain newly generated synthetic data; S3. Feed the labeled data into two teacher networks for training at the same time. One teacher network is the original teacher network, and the other is the detector OpenDET for detecting unknown categories. The specific process is as follows: S31, the labeled part of the original data is fed into two teacher networks for training at the same time to produce pseudo labels for the unlabeled data; S32. The original teacher network generates labels with known categories, while some of the targets in the labels generated by OpenDET are classified as unknown categories; S4. Use the two trained teacher networks to predict unlabeled data and synthetic data, fuse the pseudo labels generated by the original teacher network with the unknown category information in the pseudo labels generated by OpenDET, complete label fusion, and obtain the final pseudo labels; The specific process of label fusion is as follows: S41, merge all the content in the labels generated by the original teacher network and the unknown labels in the labels generated by opendet; S42: When two models make contradictory predictions, filter them and choose whether to retain the unknown category label information; S5. Feed the labeled data, unlabeled data with pseudo labels, and synthetic data into the semi-supervised framework, output the target detection results, and perform training.

2. The semi-supervised open-set target detection method based on label fusion according to claim 1, characterized in that During the dynamic enhancement process, the amount of foreground information of each category is controlled so that the amount of foreground information of each category in the enhanced information database is consistent.

3. The semi-supervised open-set target detection method based on label fusion according to claim 1, characterized in that The specific process of obtaining the newly generated synthetic data is as follows: S21, randomly selecting a foreground information image and corresponding label information from the dynamically enhanced foreground information library, and fusing it with the unlabeled data; S22. During fusion, foreground information is added to random areas of the unlabeled data, and the two are fused at a ratio between 0 and 1.

4. The semi-supervised open-set target detection method based on label fusion according to claim 1, characterized in that The original teacher network is a fully supervised FCOS network with ResNet50 as the backbone, and the unknown category detection teacher network is a fully supervised OpenDET network; The semi-supervised framework is a network based on pseudo labels and continuously updated through EMA.

Images