Class imbalance workshop tool detection method based on semi-supervised learning

Abstract

The invention discloses a class unbalance workshop tool detection method based on semi-supervised learning, and aims to solve the problems of difficult acquisition of annotated images and class unbalance of the annotated images in the process of a workshop tool detection task. In a model architecture design process, a teacher model and a student model which are identical in structure are selected, so that an integral combined learning framework is formed. In a training cycle, in order to utilize labeled and unlabeled images to a greater extent, part of image samples are randomly selected from the labeled and unlabeled images according to a specific proportion and are combined into a training small batch for processing, so that the diversification of the images is ensured, and a model in training is promoted to be in wide contact with image examples with various different characteristics.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and image processing technology, and in particular to a method for detecting imbalanced shop floor tools based on semi-supervised learning. Background Technology

[0002] In the field of image analysis, object detection is one of the most widely researched key areas. Object detection is the process of locating and identifying targets of interest in an image using advanced machine vision technology. With the rapid development of deep learning technology, the widespread use of object detection technology in intelligent manufacturing and automated production systems is having an increasingly profound impact on the overall management and operational efficiency of production systems. In some large manufacturing workshops, the types and quantities of tools required are enormous. Warehouse tool managers organize and sort tools according to workshop tool needs, and tool delivery personnel distribute tools to designated locations in the workshop in an orderly manner according to requirements. However, problems such as frequent tool borrowing and exchange, random placement, and serious losses often occur during work, affecting the cleanliness of the workshop working environment and making it impossible to achieve refined tool management. To prevent resource waste and safety hazards caused by lost tools, companies currently often use manual counting to monitor and verify the measurement work. However, this manual counting management model not only increases labor management costs but also suffers from problems such as ineffective monitoring and low accuracy, resulting in lower workshop production efficiency. By leveraging artificial intelligence technology to promptly detect and record the entry and exit status of various tools carried by workers, precise management and supervision of tool usage can be achieved. This not only effectively prevents tools from being left behind but also improves tool utilization, helping to reduce the time workers spend managing and preparing tools, thereby enhancing overall work efficiency. Furthermore, precise detection and management of workshop tools can reduce potential safety risks caused by tools being forgotten in the workshop. Therefore, using intelligent technology for workshop tool detection and management has significant application value.

[0003] In practical research, it was found that model training within supervised learning systems heavily relies on a large number of accurately labeled images. Manual labeling is not only tedious and time-consuming but also requires a high level of specialized skill, resulting in consistently high costs for obtaining these labeled images. Secondly, in the image set of workshop tools, commonly used tools dominate due to their relatively frequent use, while specialized tools requiring high precision have extremely limited sample numbers due to their infrequent use. This class imbalance causes the model to focus more on learning the features of the majority class tools during training, while failing to deeply learn the features of the minority class tools. This leads to missed detections or misclassifications in subsequent detection tasks, significantly impacting overall detection accuracy and limiting the model's effectiveness in real-world industrial environments. Therefore, this paper proposes a semi-supervised learning-based method for class-imbalanced workshop tool detection to achieve accurate localization and identification of workshop tools in images. Summary of the Invention

[0004] Based on the technical problems existing in the background art, the present invention proposes a class-unbalanced shop tool detection method based on semi-supervised learning.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A semi-supervised learning-based method for detecting class-imbalanced shop floor tools includes the following steps:

[0007] S1. For labeled images, apply label resampling, followed by filtering.

[0008] S2. Perform weak and strong enhancement processing on the input unlabeled image and obtain the result image;

[0009] S3. The teacher model provides pseudo-label information for weakly enhanced images through feature extraction and prediction. Then, it uses confidence threshold to filter and remove pseudo-labels with low thresholds to improve the quality of pseudo-label information.

[0010] S4. The image sequence containing pseudo-label information obtained in step S3 is sent to the CMM (CutMix-Mosaic Mixup) module. The mixed images are merged by the CutMix operation, and then the merged images are stitched together by the Mosaic operation.

[0011] S5. Input the stitched image obtained in step S4 into the student model, and optimize the loss function L to update the parameters of the student model.

[0012] S6. Input the labeled image obtained in step S1, the enhanced image obtained in step S2, and the stitched image obtained in step S4 into the student model, and adjust the teacher model parameters using the Exponential Moving Average (EMA) mechanism.

[0013] S7. Set the number of iterations for the training process from steps S1 to S6 to obtain a stable detection model. This model can detect the position information of the tools contained in the input test image and determine their category.

[0014] Furthermore, step S1 further comprises:

[0015] First, oversampling is performed on tail categories in the labeled images to improve the detection accuracy of these categories. Then, a detector is used to extract sufficient pseudo-labels for head categories from the unlabeled images to compensate for the undersampling of head categories in the labeled images. The specific process is as follows:

[0016] (1) Calculation of category proportion: For each category τ, calculate its occurrence proportion f(τ) in the labeled image.

[0017] (2) Category-level repetition factor: Based on the above proportions, calculate the repetition factor r(τ) for each category.

[0018]

[0019] Here, power is a hyperparameter with a value range of [0,1], used to adjust the degree of oversampling.

[0020] (3) Image-level repetition factor: For each image I, an image-level repetition factor is defined based on all categories contained in the image:

[0021] r(I) = max c∈I r(τ)

[0022] If an image contains a tail category, its repetition factor will be increased accordingly, thus receiving more attention during training.

[0023] The labeled images are filtered, including random scaling, random flipping, color enhancement, and target filtering.

[0024] Furthermore, step S2 further comprises:

[0025] S21. Perform weak enhancement processing on the input unlabeled image, including random scaling and random flipping;

[0026] S22. Perform strong enhancement processing on the input unlabeled image, including: random scaling, random flipping, color transformation, geometric transformation, random sequential application of enhancement, target filtering, and random erasure.

[0027] Furthermore, step S3 further comprises:

[0028] S31. The teacher model makes predictions for the weakly enhanced image input, including random scaling and random flipping.

[0029] S32. The teacher model uses non-maximum suppression (NMS) to filter images containing false annotation information.

[0030] Furthermore, step S4 further comprises:

[0031] S41. For the image sequence of pseudo-labeled information fed into the CMM (CutMix-Mosaic Mixup) module, the mixed images are merged through the CutMix operation. The CutMix operation process is as follows: First, two different images are randomly selected. Then, a rectangular region is randomly selected from each of the two images for cropping. The two cropped rectangular regions are then swapped to generate a newly stitched image. The label calculation of the new synthesized image is based on the original image label and is processed by weighted averaging according to the area of ​​each cropped region. During the training iteration: First, four pseudo-labeled images and their corresponding augmented images are randomly sampled from the pseudo-label cache of the previous iteration as supplementary images for the mixing operation. Then, the four sampled images are region-cropped and mixed through the CutMix operation to generate four new mixed images. Next, the pseudo-labeled images of the current batch are merged with the above four CutMix mixed images, resulting in a total of eight images.

[0032] S42. For the 8 images obtained through the CutMix operation, 4 images are randomly selected and stitched together using a Mosaic enhancement strategy to generate a hybrid image containing multi-scale targets. The stitching process is as follows: The module first downsamples the 4 images with pseudo-annotations to reduce the image resolution. Then, these 4 downsampled images are stitched together in a certain way to form a new Mosaic hybrid image, which contains the pseudo-annotation information of all the original images.

[0033] S43. Replace the original four unlabeled images with one Mosaic blended image, four CutMix blended images, and one labeled image, and use them as the input images for the student model.

[0034] Furthermore, step S5 further includes:

[0035] S51. In the student model, the loss function is optimized to update the parameters of the student model. The expression of the loss function is:

[0036] L = L s +w u L u

[0037] Where: L s The loss from the supervised learning portion is called supervised loss. u The loss is the loss from the unsupervised learning part, also known as semi-supervised loss. u These are the weights used in unsupervised learning. Optimization is performed on them in the student model:

[0038] S52, Supervised Loss L s It includes classification loss and regression loss, and its expression is:

[0039] L s =α cls L cls +β reg L reg

[0040] Where, α cls and β reg These are the weighting coefficients, and the classification loss L. cls and regression loss L reg These are cross-entropy loss and L1 loss, respectively, with the following expressions:

[0041]

[0042]

[0043] Among them, for labeled images (x) i y i ), where x i It is the input sample, y i These are the corresponding labels, N represents the number of labeled samples, and C represents the total number of different categories involved in the entire classification task. ij p represents the true label of sample i belonging to category j. ij It is the probability that the model predicts sample i as class j. These are the model's predicted values.

[0044] S53, Unsupervised Loss L u It is the pseudo-label loss L p CutMix loss L c and Mosaic loss Lm The summation is expressed as follows:

[0045] L u =L p +γL c +δL m

[0046] Where γ and δ are weighting coefficients, and the pseudo-label loss L p CutMix loss L c and Mosaic loss L m It consists of its corresponding classification loss and regression loss, respectively, and the expressions are as follows:

[0047]

[0048] Furthermore, step S6 further includes:

[0049] Given a set of weight parameters, the EMA mechanism uses a decay rate, typically denoted as β, to adjust the weighted average of the current parameters at each update. Assume the parameters at the current time step are θ. t The EMA update is as follows:

[0050] EMA t =β·EMA t-1 +(1-β)·θ t

[0051] Here EMA t This represents the EMA parameter value at time t. t-1 θ represents the EMA parameter value at the previous moment. t This represents the parameter value at the current moment.

[0052] Furthermore, step S7 further includes:

[0053] Input the test image into the trained workshop tool detection model, and the model will provide detailed detection results, including the minimum bounding box based on the location of the detected tool and marking its detection confidence, as well as determining the type of the tool.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] This invention aims to address the difficulties in acquiring labeled images and the imbalance of labeled image categories encountered in workshop tool inspection tasks. In the model architecture design, a teacher model and a student model with identical structures were selected to form an overall joint learning framework. During the training loop, to maximize the use of both labeled and unlabeled images, a portion of image samples are randomly selected from both in a specific ratio and combined into a training mini-batch for processing. This ensures image diversity and allows the model to be exposed to a wide range of image examples with various features during training.

[0056] 1. Construct a semi-supervised detection framework based on dynamic hybrid enhancement technology.

[0057] 2. Build a label resampling mechanism based on repetition factor to balance the class distribution of labeled and unlabeled images and improve the balance of model training.

[0058] 3. A CMM hybrid enhancement module was designed. This module combines the local feature reconstruction characteristics of CutMix with the global scene fusion advantages of Mosaic. It not only significantly enhances the cross-scale object detection performance, but also overcomes the limitations of single-iteration images through cross-batch hybrid operations, greatly enriching the diversity of training samples.

[0059] 4. Compared with existing algorithms, this method outperforms other semi-supervised detection methods in key metrics such as mAP50:95 and mAP75 under different annotation ratios, proving that it can efficiently utilize pseudo-annotated images and effectively alleviate the problem of long-tailed image distribution. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the overall network framework structure in this application;

[0061] Figure 2 This is a schematic diagram of the teacher-student model framework in this application;

[0062] Figure 3 This is a schematic diagram of the CutMix module in this application;

[0063] Figure 4 This is a schematic diagram of the Mosaic module in this application. Figure 5 This is a typical example of some test results. Detailed Implementation

[0064] The following is in conjunction with the appendix Figure 1-5 Further explanation of the present invention:

[0065] Based on the problems mentioned in the background art, this application addresses the class imbalance problem in workshop tool target detection by designing a semi-supervised learning-based method. This method oversamples the tail category while compensating for the head category using unlabeled images. A two-stage teacher-student model architecture is constructed to effectively achieve high-quality detection under limited labeled image conditions. The teacher model generates high-quality pseudo-labels using weakly enhanced unlabeled images. After confidence filtering, the results are input into the CMM module for hybrid enhancement processing.

[0066] The technical solution of this invention is as follows:

[0067] Step 1: Apply label resampling to labeled images, followed by filtering.

[0068] Step 2: Perform weak and strong enhancement processing on the input unlabeled image;

[0069] Step 3: The teacher model provides pseudo-label information for the weakly enhanced result image through feature extraction and prediction. Then, it uses a confidence threshold to filter and remove pseudo-labels with low thresholds to improve the quality of pseudo-label information.

[0070] Step 4: Send the image sequence containing pseudo-label information into the CMM module, merge the mixed images using the CutMix operation, and stitch the merged images using the Mosaic operation;

[0071] Step 5: Input the stitched image into the student model and use the loss function L to optimize and update the student model parameters;

[0072] Step 6: Input the labeled image, the enhanced image, and the stitched image into the student model, and adjust the teacher model parameters using the EMA mechanism.

[0073] Step 7: Input the test image to detect the location information of the tools contained in the image and determine their category.

[0074] The specific details of each of the above steps are as follows:

[0075] The overall network structure is as follows Figure 1 As shown, teacher and student models with identical structures were selected to form an overall joint learning framework. During each training cycle, to maximize the use of labeled and unlabeled images, a portion of image samples were randomly selected from both in a specific ratio and combined into a training mini-batch for processing. This random selection method ensures image diversity, allowing the trained model to be exposed to a wide range of image examples with various features.

[0076] In this application, the tail category in the labeled image is first oversampled to improve the detection accuracy of these categories. Then, a detector is used to mine sufficient false annotations for the head category in the unlabeled image to compensate for the undersampling of the head category in the labeled image. The specific process is as follows:

[0077] (1) Calculation of category proportion: For each category τ, calculate its occurrence proportion f(τ) in the labeled image.

[0078] (2) Category-level repetition factor: Based on the above proportions, calculate the repetition factor r(T) for each category.

[0079]

[0080] Here, power is a hyperparameter with a value range of [v, 1], used to adjust the degree of oversampling.

[0081] (3) Image-level repetition factor: For each image I, an image-level repetition factor is defined based on all categories contained in the image:

[0082] r(I) = max c∈I r(τ)

[0083] If an image contains a tail category, its repetition factor will be increased accordingly, thus receiving more attention during training.

[0084] The above methods can not only effectively increase the number of samples in the tail category, but also balance the distribution of the head category in labeled and unlabeled images, thereby enhancing the robustness and generalization ability of the model across different categories.

[0085] Secondly, unlabeled images undergo both weak and strong enhancement processing. Weak enhancement employs gentler methods such as fine-tuning brightness and minor rotation, aiming to enrich image types while preserving original characteristics as much as possible. The teacher model uses this as a basis for predicting these weakly enhanced images, providing pseudo-labels based on its feature extraction. Given that the generated pseudo-labels may contain noise, a confidence threshold is set to improve the overall quality of the pseudo-labels. High-quality pseudo-labeled images are then fed into the CMM module. In the CMM module, supervised and unsupervised losses are optimized to update student model parameters, ensuring the stability and consistency of the teacher model. The EMA mechanism of the student model is used to adjust the teacher model parameters. The EMA mechanism selectively updates the teacher model parameters based on changes in student model parameters during training, allowing the teacher model to maintain its independent properties while also accepting changes brought about by the student model's training.

[0086] This kind of image processing allows student models to be exposed to a richer and more diverse collection of images for training tasks. This complex sample set covers different permutations and combinations of tools of various scales, enabling student models to fully learn the characteristics of tools in various situations, thereby effectively addressing the problem of tool category imbalance in the workshop.

[0087] In this application, the teacher-student model framework, such as... Figure 2 As shown, its workflow revolves around unlabeled and labeled images. Through close and efficient collaboration between teacher and student models, it achieves in-depth image mining and continuous optimization of model performance.

[0088] Unlabeled images undergo weak enhancement before being fed into the teacher model. The teacher model performs in-depth analysis and prediction on these images, generating high-quality pseudo-labeled images. These pseudo-labeled images reflect key information such as the category and location of targets in the unlabeled images to a certain extent, serving as additional supervisory signals and providing information support for subsequent model training. The generated pseudo-labeled images, along with the strongly enhanced unlabeled images and the original labeled images, are fed into the student model for training. During this process, the student model continuously optimizes its parameters to better adapt to the feature distribution of the input images, thereby improving detection accuracy and stability.

[0089] The teacher model updates its parameters from the student model using the EMA mechanism. This teacher-student collaboration mechanism uncovers the latent information contained in unlabeled images, effectively alleviating the model training dilemma caused by insufficient labeled images. Through two key steps—the teacher model generating pseudo-labeled images to provide additional supervision to the student model, and the teacher model updating its parameters from the student model—the model can fully utilize the information carried by a large number of unlabeled images on a limited basis, thereby improving its generalization ability and enabling it to accurately detect targets when faced with new and unseen images. Simultaneously, it enhances the model's robustness, maintaining stable and reliable detection performance in complex and changing environments.

[0090] In this application, the proposed hybrid enhancement module mainly consists of the CutMix module and the Mosaic module.

[0091] CutMix module, such as Figure 3As shown, image synthesis provides sample materials for model training. The process involves: first, arbitrarily selecting two different images; then, randomly selecting a rectangular region from each image for cropping; and finally, swapping the positions of these two cropped rectangular regions to generate a newly stitched image. This operation breaks down the local correlations between different parts of the original image, allowing the model to learn not only from local features but also from more local features and cross-regional information. The label calculation for the synthesized image is based on the original image labels and is weighted and averaged according to the area of ​​each cropped region. The synthesized images generated by the CutMix module can simulate complex real-world situations and more efficiently grasp the relevant methods for handling the above problems. From the perspective of model training, because the synthesized image contains information from different images, its decision boundary is usually smoother, making classification decisions less susceptible to the influence of small noises. This not only effectively alleviates overfitting but also improves the model's ability to recognize edges and small targets.

[0092] The Mosaic module is another important component of the CMM module, such as Figure 4 As shown, the specific operation process is as follows: First, downsampling is performed on four images with pseudo-labels to reduce the image resolution. Then, these four downsampled images are stitched together in a certain way to form a new image, which contains all the pseudo-label information from the original images. This processing method increases the number of pseudo-labels in a single image by four times, allowing the model to access more target information during a single training process, thus improving training efficiency. Furthermore, the downsampling operation generates more small and medium-sized targets, resulting in a more balanced distribution of target sizes. By balancing the target size distribution, the Mosaic module enables the model to better learn the features of targets of different sizes, improving the model's ability to detect various targets.

[0093] Furthermore, the Mosaic module enriches the scene complexity and target diversity in the training samples by fusing contextual information from different sources in an image, enabling the model to better learn the interrelationships between targets and global background features, thereby improving the accuracy and reliability of target detection.

[0094] In CMM, the CutMix and Mosaic modules each play unique roles and work together to improve the model's robustness and generalization performance. Through this image enhancement method, the model can better adapt to complex and ever-changing real-world scenarios, providing a more effective solution for target detection tasks such as shop floor tool inspection.

[0095] In this application, the total loss function consists of supervised loss and unsupervised loss, and the calculation formula is as follows:

[0096] L = L s +w u L u

[0097] Among them, L s There is a supervisory loss, L u It is unsupervised loss, w u These are the weights in unsupervised learning, used to balance the relative importance of supervised and unsupervised losses.

[0098] The loss in supervised learning includes classification loss and regression loss, and its calculation formula is as follows:

[0099] L s =α cls L cls +β reg L reg

[0100] Where, α cls and β reg It is the weighting coefficient, L cls and L reg These are cross-entropy loss and L1 loss, respectively, for a labeled image (x i ,y i ), where x i It is the input sample, y i The corresponding labels are used, and the formulas for calculating classification loss and regression loss are as follows:

[0101]

[0102]

[0103] Where N represents the number of labeled samples, reflecting the image size used for supervised learning, and C represents the number of classes, i.e., the total number of different classes involved in the entire classification task. ij This indicates the true label of sample i belonging to category j; it is a clear category indicator value used for comparison with the model's prediction results. ij It is the probability that the model predicts sample i as class j. These are the model's predicted values.

[0104] The loss of the unsupervised part is the pseudo-label loss L. p CutMix loss L c and Mosaic loss L m The sum of is calculated using the following formula:

[0105] L u =L p +γL c +δLm

[0106] Where γ and δ are weighting coefficients, the pseudo-label loss, CutMix loss, and Mosaic loss are composed of their corresponding classification loss and regression loss, and the calculation formulas are as follows:

[0107]

[0108] To better illustrate the solution and practical effects of this application, specific embodiments are provided to explain the implementation of the invention. Both training and testing are performed on a public image set (from roboflow) to ensure the reliability and fairness of the experimental results. The image set covers five different categories of tools, containing a total of 9302 images. The image set is divided into training, testing, and evaluation sets in a 7:2:1 ratio. 10%, 20%, and 30% of the images in the training set are randomly selected as labeled images, and the remainder are used as unlabeled images. To fully utilize the limited labeled information while simultaneously mining potential features in the unlabeled images, during training, each GPU uses a configuration of 1 labeled image and 4 unlabeled images to achieve an effective balance between supervised and unsupervised images.

[0109] In this application, a series of ablation experiments were conducted to evaluate the impact of each module on the semi-supervised shop floor tool detection model. The experimental results are shown in Table 1. As can be seen from the images in the table, different image enhancement strategies improved performance across multiple evaluation metrics, validating the effectiveness of each module.

[0110] Table 1 Ablation Experiment Results

[0111]

[0112] Label resampling, as a means to address class imbalance, demonstrated its effectiveness in the initial stages of the model. Experimental images show that the baseline model achieves higher mAP. 50:95 The initial result was 54.7%. After introducing oversampling, it improved to 55.4%. This shows that oversampling, through tail-class oversampling, alleviates the imbalance in class distribution to a certain extent and improves the performance of detecting a minority of classes.

[0113] The introduction of the CutMix module further optimizes model performance, increasing the model's mAP. 50:95The accuracy was improved to 56.7%, a 1.3% improvement compared to applying standard sampling alone. CutMix's method of randomly cropping and swapping local image regions enhances its ability to learn local features, allowing the model to focus more on the interaction information between different parts, and making it easier to handle the effects of occlusion and deformation in complex backgrounds. By smoothing the decision boundary, CutMix effectively reduces the risk of overfitting and improves the model's generalization ability.

[0114] After adding the Mosaic module, the diversity of training samples was enriched, and the target size distribution appeared more uniform, improving the model's mAP. 50:95 The performance reached 59.1%. Mosaic generated training samples containing small and medium-sized targets by stitching together regions from different images. This not only increased the number of targets but also improved the size distribution of the targets, enabling the model to better learn diverse target features.

[0115] From the experimental results images in Table 1, mA P50 It increased from 81.1% to 85.1%, while mAP 75 The increase from 61.5% to 67.4% indicates that the model has significantly improved target detection accuracy at both low and high IoU thresholds.

[0116] The proposed oversampling strategy effectively alleviates the class imbalance problem in shop floor tool inspection tasks. The core idea of ​​oversampling is to adjust the class distribution by oversampling the tail classes in labeled images, thereby improving the model's ability to detect rare classes.

[0117] Table 2. Impact of different power levels on performance

[0118] Method Power <![CDATA[mAP 50:95 ]]> Drill Hammer pliers Screwdriver wrench Supervised 0 38.7 57.9 28.3 48.5 32.4 26.6 Ours 0 58.9 73.8 52.3 64.8 51.2 52.2 Ours 0.25 58.5 74.5 49.9 66.8 49.5 51.8 Ours 0.5 58.9 72.6 51.9 66.1 51.3 52.7 Ours 1.0 59.1 73.8 50.2 66.4 51.9 53.4

[0119] In this application, different Power values ​​were tested to evaluate their role in overall mAP. 50:95 The performance in terms of category specificity is shown in Figure 2. The results show that when the Power value is 1, mAP... 50:95 The accuracy was increased to 59.1%, which verifies that a higher Power value can more effectively adjust the weight of tail categories, thereby improving the detection accuracy of rare categories.

[0120] The proposed method was evaluated using labeled and unlabeled images with different proportions. The fully supervised model (Supervised) employed Faster R-CNN, while the others were semi-supervised models. Both the student and teacher networks used Faster R-CNN. Tables 3, 4, and 5 show the performance comparison of the fully supervised method and various semi-supervised object detection methods under different labeled image proportions (10%, 20%, and 30%). The results in the tables demonstrate that the proposed method achieves good performance under various image proportions.

[0121] Table 3. Comparison Experiments with Label Images at 10% Scale

[0122] Method <![CDATA[mAP 50:95 ]]> <![CDATA[mAP 50 ]]> <![CDATA[mAP 75 ]]> <![CDATA[mAP m ]]> <![CDATA[mAP l ]]> Supervised 38.7 71.6 36.7 4.5 40 STAC 48.1 80.2 51.4 5.5 50 Label Match 55.3 83.7 62.1 <![CDATA[ 6.4 ]]> 57.4 Soft Teacher 54 82.9 62.7 4.9 56.1 Unbiased Teacher <![CDATA[ 55.6 ]]> <![CDATA[ 84.2 ]]> <![CDATA[ 62.8 ]]> 4.6 <![CDATA[ 57.8 ]]> Ours 59.1 85.1 67.4 8.8 61.2

[0123] In a 10% scale labeled image scenario, the method proposed in this application achieves high mAP. 50:95 mAP 50 mAP 75 It significantly outperformed other methods in all three metrics, including mAP. 50:95 Increased to 59.1%, the mAP (meta-analysis accuracy) for medium-sized targets m The accuracy rate reached 8.8%, demonstrating its effectiveness on low-annotation images.

[0124] Table 4. Comparison Experiments with Label Images at 20% Scale

[0125] Method <![CDATA[mAP 50:95 ]]> <![CDATA[mAP 50 ]]> <![CDATA[mAP 75 ]]> <![CDATA[mAP m ]]> <![CDATA[mAP l ]]> Supervised 48.6 81.1 52.2 7.9 50.1 STAC 55.2 85.2 63.7 7.4 57.2 Label Match 59.6 88.3 <![CDATA[ 68.9 ]]> 12.1 61.6 Soft Teacher 59.8 87.4 68 8.1 61.9 Unbiased Teacher <![CDATA[ 60 ]]> 87.7 <![CDATA[ 68.9 ]]> <![CDATA[ 12.7 ]]> <![CDATA[ 62 ]]> Ours 62.8 <![CDATA[ 87.8 ]]> 71.7 15.1 64.4

[0126] In scenarios with 20% labeled images, the method proposed in this application maintains a leading position in all metrics, including mAP. 50:95 It reached 62.8%, in mAP m In terms of metrics, it reached 15.1%, indicating its effectiveness in medium-sized target detection tasks.

[0127] Table 5. Comparison Experiments with Label Images at 30% Scale

[0128] Method <![CDATA[mAP 50:95 ]]> <![CDATA[mAP 50 ]]> <![CDATA[mAP 75 ]]> <![CDATA[mAP m ]]> <![CDATA[mAP l ]]> Supervised 53 84.1 58.8 12 54.5 STAC 58.6 <![CDATA[ 87.2 ]]> 67.9 9.1 60.6 Label Match 60.9 <![CDATA[ 88.5 ]]> 69.4 9.8 63 Soft Teacher <![CDATA[ 62.4 ]]> 88.9 <![CDATA[ 70.9 ]]> 10.9 <![CDATA[ 64.4 ]]> Unbiased Teacher 62.2 88.4 70.2 15.1 64.2 Ours 63.2 88.1 72.8 <![CDATA[ 12.8 ]]> 65

[0129] In a scenario with 30% labeled images, the mAP of the method proposed in this application is [missing information]. 50:95 Reaching 63.2%, mAP 75 The accuracy rate increased to 72.8%, demonstrating its advantage in high-precision target detection. Figure 5 This is a typical example of some test results.

[0130] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely illustrative descriptions of the application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Thus, if such modifications and modifications fall within the scope of the claims and their equivalents, this application is also intended to include such modifications and modifications.

[0131] The above descriptions are all preferred embodiments of the present invention. For those skilled in the art, any modifications to the present invention in various equivalent forms without departing from the principle of the present invention shall fall within the protection scope of the appended claims.

Claims

1. A method for detecting class-imbalanced shop floor tools based on semi-supervised learning, characterized in that: Includes the following steps: S1. Apply labels to the input labeled image, resample and filter it; S2. Perform weak and strong enhancement processing on the input unlabeled image and obtain the result image; S3. The teacher model provides pseudo-label information for weakly enhanced images through feature extraction and prediction. Then, it uses confidence threshold to filter and remove pseudo-labels with low thresholds to improve the quality of pseudo-label information. S4. The image sequence containing pseudo-label information obtained in step S3 is sent to the CutMix-Mosaic Mixup module. The mixed images are merged by the CutMix operation, and then the merged images are stitched together by the Mosaic operation. S5. Input the stitched image obtained in step S4 into the student model, and optimize the loss function L to update the parameters of the student model. S6. Input the labeled image obtained in step S1, the strongly enhanced image obtained in step S2, and the stitched image obtained in step S4 into the student model, and adjust the teacher model parameters using the Exponential Moving Average mechanism. S7. Set the number of iterations for the training process from steps S1 to S6 to obtain a stable detection model. This model can detect the position information of the tools contained in the input test image and determine their category.

2. The method for class-imbalanced shop floor tool detection based on semi-supervised learning according to claim 1, characterized in that: The specific process of S1 is as follows: First, the tail category in the labeled image is oversampled. Then, a detector is used to mine sufficient false annotations of the head category in the unlabeled image to compensate for the undersampling of the head category in the labeled image. The specific process is as follows: (1) Calculation of category proportion: For each category τ, calculate its occurrence proportion f(τ) in the labeled image; (2) Category-level repetition factor: Based on the above proportion f(τ), calculate the repetition factor r(τ) for each category. Here, power is a hyperparameter with a value range of [0,1], which is used to adjust the degree of oversampling; (3) Image-level repetition factor: For each image I, an image-level repetition factor is defined based on all categories contained in the image: r(I)=max c∈I r(τ); If an image contains a tail category, its repetition factor will be increased accordingly, thus receiving more attention during training; The labeled images are filtered, including random scaling, random flipping, color enhancement, and target filtering.

3. The method for class-imbalanced shop floor tool detection based on semi-supervised learning according to claim 1, characterized in that: In S2, the weak enhancement processing for the input unlabeled image includes random scaling and random flipping, and the strong enhancement processing for the input unlabeled image includes random scaling, random flipping, color transformation, geometric transformation, random sequential application of enhancement, target filtering, and random erasure.

4. The method for class-imbalanced shop floor tool detection based on semi-supervised learning according to claim 1, characterized in that: The S3 step is as follows: S31. The teacher model makes predictions for the weakly enhanced image input, including random scaling and random flipping. S32. The teacher model uses the Non-Maximum Suppression operation to filter images containing false annotation information.

5. The method for class-imbalanced shop floor tool detection based on semi-supervised learning according to claim 1, characterized in that: The S4 step is as follows: S41. For the image sequence of pseudo-label information fed into the CutMix-Mosaic Mixup module, the mixed images are merged through the CutMix operation. The CutMix operation process is as follows: First, two different images are randomly selected. Then, a rectangular region is randomly selected from each of the two images for cropping. Then, the positions of the two cropped rectangular parts are swapped to generate a newly stitched image. The label calculation of the new synthesized image is based on the original image label and is processed by weighted averaging according to the area of ​​each cropped region. During the training iteration: First, four pseudo-label images and their corresponding augmented images are randomly sampled from the pseudo-label cache of the previous iteration as supplementary images for the mixing operation. Then, the four sampled images are cropped and mixed through the CutMix operation to generate four new mixed images. Then, the pseudo-label images of the current batch are merged with the above four CutMix mixed images, for a total of eight images. S42. For the 8 images obtained through the CutMix operation, 4 images are randomly selected and stitched together using the Mosaic enhancement strategy to generate a hybrid image containing multi-scale targets. The stitching process is as follows: the module first downsamples the 4 images with pseudo-labels to reduce the resolution of the images; then, the 4 downsampled images are stitched together in a certain way to form a new Mosaic hybrid image, which contains the pseudo-label information of all the original images. S43. Replace the original four unlabeled images with one Mosaic blended image, four CutMix blended images, and one labeled image, and use them as the input images for the student model.

6. The method for class-imbalanced shop floor tool detection based on semi-supervised learning according to claim 1, characterized in that: The S5 step is as follows: S51. In the student model, the loss function is optimized to update the parameters of the student model. The expression of the loss function is: L = L s +w u L u Where: L s The loss is the portion learned through supervised learning, also known as Supervised Loss, L. u The loss is the loss from the unsupervised learning part, also known as semi-supervised loss. u These are the weights for unsupervised learning, which are optimized in the student model; S52, Supervised Loss L s It includes classification loss and regression loss, and its expression is: L s =a cls L cls +b reg L reg Where, α cls and β reg These are the weighting coefficients, and the classification loss L. cls and regression loss L reg These are cross-entropy loss and L1 loss, respectively, with the following expressions: Among them, for labeled images (x) i ,y i ), where x i It is the input sample, y i These are the corresponding labels, N represents the number of labeled samples, C represents the total number of different categories involved in the entire classification task, and y ij p represents the true label of sample i belonging to category j. ij It is the probability that the model predicts sample i as class j. These are the model's predicted values. S53, Unsupervised Loss L u It is the pseudo-label loss L p CutMix loss L c and Mosaic loss L m The summation is expressed as follows: L u =L p +γL c +δL m ; Where γ and δ are weighting coefficients, and the pseudo-label loss L p CutMix loss L c and Mosaic loss L m It consists of its corresponding classification loss and regression loss, respectively, and the expressions are as follows:

7. The method for class-imbalanced shop floor tool detection based on semi-supervised learning according to claim 1, characterized in that: The S6 step is as follows: Given a set of weight parameters, the EMA mechanism uses a decay rate, usually denoted as β, to adjust the weighted average of the current parameters at each update; assuming the parameter at the current time is θ. t The EMA update is as follows: EMA t =β·EMA t-1 +(1-β)·θ t Here EMA t This represents the EMA parameter value at time t. t-1 θ represents the EMA parameter value at the previous moment. t This represents the parameter value at the current moment.

8. The method for class-imbalanced shop floor tool detection based on semi-supervised learning according to claim 1, characterized in that: The S7 step is as follows: Input the test image into the trained workshop tool detection model. The model will give detailed detection results, including the minimum bounding box based on the position of the detected tool and marking its detection confidence, and at the same time determine the type of the tool.

Images