A pseudo-label-based semi-supervised small sample target detection method

Abstract

The present application belongs to the field of computer vision, and particularly relates to a kind of semi-supervised small sample target detection method based on pseudo label, aims at solving the problem that new class sample lacks intra-class variation under the condition that the labeled sample of new class target object is limited, thereby affecting the performance of small sample target detection.This method includes: obtaining the image of the scene to be detected by a vision sensor as a first image;The first image is sent to the student network in the trained semi-supervised small sample target detection network based on pseudo label, and the result of target detection corresponding to the image of the scene to be detected is obtained;The semi-supervised small sample target detection network based on pseudo label includes TFA network, teacher network and student network;The network structures of TFA network, teacher network and student network are the same.The present application can effectively improve the adaptability of small sample target detection network to new class objects, and provide technical support for small sample target detection of robots in the field of home service, office and the like.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a semi-supervised few-sample target detection method, system, and device based on pseudo-labels. Background Technology

[0002] With the rapid development of artificial intelligence technology, the representation learning capabilities of deep neural networks are increasingly enhanced, and object detection networks trained on large-scale labeled datasets have achieved excellent detection performance. However, building large-scale labeled datasets is not only time-consuming and labor-intensive, but also difficult to construct a sample set that meets training requirements when there are many object categories and data is not readily available. For example, in real-world environments such as homes and offices, service robots face a wide variety of objects with diverse appearances, making it difficult to collect a sufficient number of training samples with rich intra-class variations for all items. Therefore, how to achieve the detection of new object categories using a small amount of labeled data (i.e., small-sample object detection) has attracted much attention from researchers.

[0003] For small-sample object detection, researchers both domestically and internationally have conducted in-depth studies, combining meta-learning or transfer learning strategies to learn representative features of categories from a small number of samples. Among these, transfer learning-based small-sample object detection methods are widely used due to their simplicity and effectiveness, with TFA (Two-stage Fine-tuning Approach) being a representative example. TFA is based on the Faster R-CNN object detection network and mainly consists of a feature extractor, a Region Proposal Network (RPN), RoI (Region of Interest) Pooling, a classifier, and a regressor. TFA divides the training process into two stages: the first stage trains on a base class dataset containing abundant labeled data to obtain a general object detector; the second stage freezes the weight parameters of the feature extractor, RPN, and RoI Pooling, and fine-tunes the classifier and regressor using a small amount of labeled data from new object classes to achieve the detection of new object classes. Because the labeled data for new object classes is limited and lacks intra-class variation, the fine-tuned network struggles to capture class-discriminating features from the limited samples of new object classes, leading to poor network adaptability to new classes. In reality, the base class dataset contains a large number of images containing new class objects. These new class objects are unlabeled and therefore treated as background during the first stage of training. If these new class objects could be automatically labeled and learned, it would improve network performance. Semi-supervised learning offers an effective solution in this regard. Semi-supervised learning typically employs a teacher network and a student network for collaborative training. The teacher network generates pseudo-labels for the unlabeled new class objects, which are then used to further train the student network, thus alleviating the problem of scarce labeled data for new classes.

[0004] Therefore, it is necessary for those skilled in the art to combine semi-supervised learning and few-shot object detection methods to fully explore the large number of unlabeled new class objects in the base class dataset, in order to solve the problem that the lack of intra-class variation of new class samples when the labeled samples of new class objects are limited, thus affecting the performance of few-shot object detection. Summary of the Invention

[0005] To address the aforementioned problems in existing technologies, specifically the lack of intra-class variation in new target object samples when labeled samples of new target objects are limited, thus affecting the performance of small-sample target detection, this invention provides a semi-supervised small-sample target detection method based on pseudo-labels, comprising:

[0006] Step S10: Acquire an image of the scene to be detected using a visual sensor, and use it as the first image;

[0007] Step S20: The first image is fed into the student network of the trained pseudo-label-based semi-supervised few-shot object detection network to obtain the object detection result corresponding to the scene image to be detected.

[0008] The pseudo-label-based semi-supervised few-shot target detection network includes a pre-trained TFA network, a teacher network, and a student network; the network structures of the pre-trained TFA network, the teacher network, and the student network are the same.

[0009] In some preferred embodiments, the training method for the student network is as follows:

[0010] Step A10: Obtain training data, which includes unlabeled training images and a small number of new class labeled samples; input the unlabeled training images into a pre-trained TFA network to obtain the prediction results of the new class target objects in the unlabeled training images, which are used as the first prediction results; the unlabeled training images are images of new class target objects in the base class dataset of the COCO dataset.

[0011] Step A20: Remove the first object candidate boxes with confidence scores lower than the set background threshold from the first prediction result, and calculate the values ​​of the remaining first object candidate boxes of the same type at the set quantiles as the primary category adaptive thresholds for each new class, and use the first object candidate boxes with confidence scores higher than the primary category adaptive thresholds of the corresponding class as the first pseudo-labels; the first object candidate boxes are the object candidate boxes corresponding to the first prediction result.

[0012] Step A30: Perform weak data augmentation and strong data augmentation on the unlabeled training images to obtain weakly augmented images and strongly augmented images; input the weakly augmented images into the teacher network to obtain the prediction results of the new class of target objects in the weakly augmented images, which are used as the second prediction results;

[0013] Step A40: Remove the second object candidate boxes with confidence scores lower than the set background threshold from the second prediction result, and in the remaining second object candidate boxes, use the second object candidate boxes with confidence scores higher than the advanced category adaptive threshold of the corresponding category as the second pseudo-label; the second object candidate box is the object candidate box corresponding to the second prediction result;

[0014] Step A50: Fuse the first pseudo-label and the second pseudo-label to obtain a third pseudo-label; then combine the third pseudo-label with the strongly augmented image. A new class of unlabeled samples is formed. The new class of unlabeled samples and a small number of new class labeled samples are input into the student network, and the loss value is calculated. The weight parameters of the student network are updated, and the weight parameters of the teacher network are updated using the updated weight parameters of the student network.

[0015] Step A60, repeat steps A10-A50 until a trained student network is obtained.

[0016] In some preferred embodiments, the values ​​of the remaining first object candidate boxes of the same type at a set quantile are calculated as follows:

[0017] The quantile function in the pandas library, a third-party data analysis tool library in Python, is used to calculate the value of the first object candidate box of the remaining class at a set quantile.

[0018] In some preferred embodiments, among the remaining second object candidate boxes, the second object candidate boxes with confidence scores higher than the corresponding category's high-level category adaptive threshold are used as second pseudo-labels. The method is as follows:

[0019] The advanced category adaptive threshold is initialized with the primary category adaptive threshold and is dynamically adjusted and updated during training; the dynamic adjustment and update process is as follows:

[0020] Let the number of second pseudo-labels of the j-th new class generated by accumulating n mini-batches be . n is the cumulative number of mini-batches, when Calculate the adaptive quantile for each category in, To accumulate the number of candidate pseudo-labels for the j-th new class obtained from n mini-batches, N max This represents the maximum number of pseudo-tags.

[0021] when Calculate the adaptive quantile for each category Where, N min This minimizes the number of pseudo-labels; in other cases, the category-adaptive quantile remains unchanged.

[0022] Then, the quantile function from the pandas library, a third-party data analysis tool library in Python, was used to calculate the confidence score of those belonging to the same new class that is not lower than the background threshold τ. b The value of the second object candidate box at the category adaptive quantile is used to update the advanced category adaptive threshold;

[0023] The second object candidate box in the second prediction result whose confidence score is higher than the high-level category adaptive threshold of the corresponding category is used as the second pseudo label.

[0024] In some preferred embodiments, the first pseudo-tag and the second pseudo-tag are merged to obtain a third pseudo-tag, and the method is as follows:

[0025] The first pseudo-label and the second pseudo-label are merged, and the merged result is used as a candidate set of pseudo-labels.

[0026] In the pseudo-label candidate set, non-maximum suppression (NMS) is used to suppress pseudo-labels with low confidence scores belonging to the same object, thereby obtaining the label fusion result, i.e., the third pseudo-label; wherein, the criterion for judging that two pseudo-labels in the pseudo-label candidate set belong to the same object is that they are of the same category and their Intersection over Union (IoU) is not lower than a set IoU threshold.

[0027] In some preferred embodiments, the student network has the following loss function during training:

[0028]

[0029]

[0030]

[0031] in, The total loss of the student network during training. To address the supervised loss for newly labeled samples, To address the unsupervised loss for novel unlabeled samples, λ u N is the loss weight hyperparameter. l and N u These represent the number of newly labeled samples and the number of unlabeled samples of the new class in a mini-batch, respectively. This represents the k-th newly labeled sample. This represents the k-th unlabeled sample of the new class. These represent the RPN classification loss, RPN regression loss, RoI head classification loss, and RoI head regression loss, respectively.

[0032] In some preferred embodiments, the weight parameters of the student network are updated, and the updated weight parameters of the teacher network are then used to update the weight parameters of the teacher network. The method is as follows:

[0033]

[0034] θ t ←αθ t +(1-α)θ st

[0035] Where γ is the learning rate hyperparameter, α is the smoothing hyperparameter, and θ st Let θ be the weight parameter of the student network. t This represents the weight parameters of the teacher network.

[0036] In a second aspect, the present invention proposes a semi-supervised few-shot target detection system based on pseudo-labels, comprising: an image acquisition module and a few-shot target detection module;

[0037] The image acquisition module is configured to acquire an image of the scene to be detected through a visual sensor, which is used as the first image.

[0038] The few-shot target detection module is configured to send the first image into the student network of a pre-trained pseudo-label-based semi-supervised few-shot target detection network to obtain the target detection result corresponding to the scene image to be detected.

[0039] The pseudo-label-based semi-supervised few-shot target detection network includes a pre-trained TFA network, a teacher network, and a student network; the network structures of the pre-trained TFA network, the teacher network, and the student network are the same.

[0040] In a third aspect, the present invention provides a storage device storing a plurality of programs adapted to be loaded and executed by a processor to implement the above-described pseudo-label-based semi-supervised few-sample target detection method.

[0041] In a fourth aspect, the present invention provides a processing apparatus, comprising a processor and a storage device; the processor is adapted to execute various programs; the storage device is adapted to store multiple programs; the programs are adapted to be loaded and executed by the processor to implement the above-described pseudo-label-based semi-supervised few-sample target detection method.

[0042] The beneficial effects of this invention are:

[0043] This invention can effectively improve the adaptability of small sample target detection networks to new types of objects, and provide technical support for small sample target detection in robots in fields such as home service and office. Attached Figure Description

[0044] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0045] Figure 1 This is a flowchart illustrating a semi-supervised few-sample target detection method based on pseudo-labels according to the present invention.

[0046] Figure 2This is a schematic diagram of the framework of a semi-supervised small sample target detection system based on pseudo-labels according to the present invention. Detailed Implementation

[0047] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0048] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0049] This invention provides a semi-supervised few-sample target detection method based on pseudo-labels, such as... Figure 1 As shown, the method includes the following steps:

[0050] Step S10: Acquire an image of the scene to be detected using a visual sensor, and use it as the first image;

[0051] Step S20: The first image is fed into the student network of the trained pseudo-label-based semi-supervised few-shot object detection network to obtain the object detection result corresponding to the scene image to be detected.

[0052] The pseudo-label-based semi-supervised few-shot target detection network includes a pre-trained TFA network, a teacher network, and a student network; the network structures of the pre-trained TFA network, the teacher network, and the student network are the same.

[0053] To more clearly explain the pseudo-label-based semi-supervised few-sample target detection method of this invention, the following will be combined with... Figure 1 The steps in the embodiments of the method of the present invention are described in detail below.

[0054] In the following embodiments, the construction and training process of the pseudo-label-based semi-supervised few-shot object detection network will be described in detail first, and then the inference process of few-shot object detection will be described.

[0055] 1. Construction and training process of a semi-supervised few-shot object detection network based on pseudo-labels

[0056] This embodiment represents a preferred implementation. A semi-supervised few-shot object detection network is pre-constructed, comprising a pre-trained TFA network, a teacher network, a student network, a primary category adaptive threshold filtering module, a higher category adaptive threshold filtering module, and a label fusion module. The training process of the pre-trained TFA network is detailed in X. Wang, TE Huang, T. Darrell, JEGonzalez, and F. Yu. Frustratingly simple few-shot object detection. International Conference on Machine Learning, 2020: 9919-9928. The teacher and student networks have the same network structure as the pre-trained TFA network, and their weight parameters are initialized using the pre-trained TFA network's weight parameters. The feature extractor weight parameters of both the teacher and student networks are frozen during training. Following a semi-supervised learning paradigm, the teacher and pre-trained TFA networks generate pseudo-labels for a large number of unlabeled new class objects in the base class dataset. These pseudo-labels are then used to further train the student network, thereby improving its adaptability to new classes. The primary category adaptive thresholding module selects pseudo-labels generated by the pre-trained TFA network as the first pseudo-label, while the advanced category adaptive thresholding module selects pseudo-labels generated by the teacher network as the second pseudo-label. The label fusion module merges the first and second pseudo-labels into a third pseudo-label, providing better supervision for the training of the student network. Guided by a loss function (the specific settings of which will be explained in detail below), the student network updates its weights using stochastic gradient descent (SGD). The teacher network, on the other hand, combines the weight parameters of the student network and directly updates its own weight parameters to generate better pseudo-labels. The training process of the pseudo-label-based semi-supervised few-shot object detection network is as follows:

[0057] Step A10: Obtain training data, which includes unlabeled training images and a small number of new class labeled samples; input the unlabeled training images into a pre-trained TFA network to obtain the prediction results of the new class target objects in the unlabeled training images, which are used as the first prediction results; the unlabeled training images are images of new class target objects in the base class dataset of the COCO dataset.

[0058] In this embodiment, we first consider the image x of the new class target object in the base class dataset of the COCO dataset. u x is generated using a pre-trained TFA network. u Prediction results of the new type of target object y 0That is, the first prediction result (including the position, category and corresponding confidence score of the object candidate box of the new type of target object).

[0059] Step A20: Remove the first object candidate boxes with confidence scores lower than the set background threshold from the first prediction result, and calculate the values ​​of the remaining first object candidate boxes of the same type at the set quantiles as the primary category adaptive thresholds for each new class, and use the first object candidate boxes with confidence scores higher than the primary category adaptive thresholds of the corresponding class as the first pseudo-labels; the first object candidate boxes are the object candidate boxes corresponding to the first prediction result.

[0060] In this embodiment, the object candidate boxes corresponding to the first prediction result (i.e., the first object candidate boxes) are statistically analyzed by category to determine their confidence scores, and those with confidence scores lower than τ are removed. b The first object candidate bounding box, where τ b The background threshold constant is preferably set to 0.1 in this invention; the remaining candidate boxes of the first object belonging to the same new class are combined and denoted as... in The confidence score of the i-th first object candidate box of the j-th new class, j = 1, 2, ..., C New C New This represents the number of categories in the new class. It is calculated using the quantile function `pandas.DataFrame.quantile` from the Python third-party data analysis library pandas (see https: / / pandas.pydata.org / docs / reference / api / pandas.DataFrame.quantile.html for details). The value at the q-quantile is used as the primary category adaptive threshold for the j-th new class. In this invention, the q-quantile is preferably set to 0.85. The calculated primary class adaptive thresholds for each new class are used as the filtering thresholds for the primary class adaptive threshold filtering module, and the prediction results y from the pre-trained TFA network are used... 0 The first object candidate box with a confidence score higher than the corresponding category screening threshold is used as the first pseudo-label.

[0061] Step A30: Perform weak data augmentation and strong data augmentation on the unlabeled training images to obtain weakly augmented images and strongly augmented images; input the weakly augmented images into the teacher network to obtain the prediction results of the new class of target objects in the weakly augmented images, which are used as the second prediction results;

[0062] In this embodiment, the base class dataset of the COCO dataset contains images x of new class target objects. u Weak data augmentation and strong data augmentation were used respectively (for details on data augmentation methods, see YCLiu, CYMa, Z.He, CWKuo, K.Chen, P.Zhang, B.Wu, Z.Kira, and P.Vajda. Unbiased teacher for semi-supervised object detection. International Conference on Learning Representations, 2021) to obtain images after weak data augmentation. Images with strong data augmentation Teacher network targeting The prediction result y of generating the new class of target objects u The second prediction result (which includes the position, category, and corresponding confidence score of the object candidate box of the new type of target object) serves as the second prediction result.

[0063] Step A40: Remove the second object candidate boxes with confidence scores lower than the set background threshold from the second prediction result, and in the remaining second object candidate boxes, use the second object candidate boxes with confidence scores higher than the advanced category adaptive threshold of the corresponding category as the second pseudo-label; the second object candidate box is the object candidate box corresponding to the second prediction result;

[0064] In this embodiment, first, the second prediction result y u Remove objects with a confidence score lower than τ from the corresponding object candidate boxes (i.e., the second object candidate boxes). b The candidate bounding boxes of objects are then filtered by the advanced category adaptive threshold filtering module to generate a second pseudo-label. The advanced category adaptive thresholding module initializes its category adaptive quantiles and advanced category adaptive thresholds using the q-quantiles and thresholds of the primary category adaptive thresholding module, respectively. During training, the category adaptive quantiles and advanced category adaptive thresholds are dynamically adjusted and updated. The specific dynamic adjustment and update process is as follows:

[0065] Let the number of second pseudo-labels of the j-th new class generated by the advanced category adaptive threshold filtering module from n mini-batches be . n is the number of cumulative mini-batches, which is preferably set to 500 in this invention. Calculate the adaptive quantile for each category in This represents the number of candidate pseudo-labels for the j-th new class obtained from n mini-batches; N maxThe maximum number of pseudo-tags is preferably set to 250 in this invention; when Calculate the adaptive quantile for each category Where, N min To minimize the number of pseudo-labels, this invention preferentially sets it to 150; in other cases, the adaptive quantile for the category remains unchanged. Furthermore, the quantile function `pandas.DataFrame.quantile` from the Python third-party data analysis library pandas is used to calculate the confidence score of members belonging to the same new class that is not lower than τ. b The value of the second object candidate box at the category adaptive quantile is used to update the high-level category adaptive threshold. The second prediction result y is then used. u Second object candidate boxes with a confidence score higher than the corresponding class's advanced adaptive threshold are used as second pseudo-labels.

[0066] Step A50: Fuse the first pseudo-label and the second pseudo-label to obtain a third pseudo-label; then combine the third pseudo-label with the strongly augmented image. A new class of unlabeled samples is formed. The new class of unlabeled samples and a small number of new class labeled samples are input into the student network, and the loss value is calculated. The weight parameters of the student network are updated, and the weight parameters of the teacher network are updated using the updated weight parameters of the student network.

[0067] In this embodiment, the tag fusion module is used to integrate the first pseudo-tag. Second pseudo-label To integrate, specifically, will and The results are merged and used as a pseudo-label candidate set. Non-Maximum Suppression (NMS) is then applied to suppress pseudo-labels with low confidence scores belonging to the same object, thus obtaining the label fusion result. That is, the third pseudo-label, wherein the criterion for judging that two pseudo-labels in the pseudo-label candidate set belong to the same object is that they are of the same category and their Intersection over Union (IoU) is not lower than a set IoU threshold, wherein the IoU threshold is preferably 0.5 in this invention.

[0068] The third pseudo-label and images after strong data augmentation Forming new class of unlabeled samples Along with a small number of newly labeled samples (x) l ,y l The student network is input for training. The loss function of the student network is calculated as follows:

[0069]

[0070] in, The total loss of the student network during training. To address the supervised loss for newly labeled samples, To address the unsupervised loss for novel unlabeled samples, λ u As a loss weight hyperparameter, it is preferably set to 2 in this invention. (Supervised Loss) and unsupervised loss The specific forms are shown in formulas (2) and (3) respectively:

[0071]

[0072]

[0073] Where, N l and N u N represents the number of newly labeled samples and the number of newly unlabeled samples in a mini-batch, respectively. l and N u The preferred values ​​are all 8; This represents the k-th newly labeled sample. This represents the k-th unlabeled sample of the new class. These represent the RPN classification loss, RPN regression loss, RoI head classification loss, and RoI head regression loss, respectively. In this invention, the standard cross-entropy loss is preferably used for the RPN classification loss, the smooth-L1 loss is preferably used for the RPN regression loss and the RoI head regression loss, and the focal loss is preferably used for the RoI head classification loss. For details, please refer to TYLin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. IEEE International Conference on Computer Vision, 2017:2980-2988.

[0074] Total loss based on student network The weight parameters θ of the student network st The update is as follows:

[0075]

[0076] Wherein, γ is the learning rate hyperparameter, which is preferably set to 0.01 in this invention.

[0077] Further combining θ st The weight parameters θ of the teacher network are calculated. t Update:

[0078] θ t ←αθ t +(1-α)θ st (5)

[0079] Wherein, α is a smoothing hyperparameter, which is preferably set to 0.999 in this invention.

[0080] Step A60, repeat steps A10-A50 until a trained student network is obtained.

[0081] 2. Inference process for small sample target detection

[0082] Step S10: Acquire an image of the scene to be detected using a visual sensor, and use it as the first image;

[0083] In this embodiment, an image of the scene to be detected is acquired by the visual sensor ZED2 and used as the first image.

[0084] Step S20: The first image is fed into the student network of the pre-trained pseudo-label-based semi-supervised few-shot object detection network to obtain the object detection result corresponding to the scene image to be detected.

[0085] In this embodiment, the first image is fed into the student network of a pre-trained pseudo-label-based semi-supervised few-shot object detection network to obtain the bounding boxes of the new class of target objects contained in the first image and their corresponding categories, thereby achieving few-shot object detection.

[0086] This invention can effectively improve the adaptability of small sample target detection networks to new types of objects, and provide technical support for small sample target detection in service robots in fields such as home service and office.

[0087] A second embodiment of the present invention provides a semi-supervised small-sample target detection system based on pseudo-labels, such as... Figure 2 As shown, it includes: an image acquisition module 100 and a small sample target detection module 200;

[0088] The image acquisition module 100 is configured to acquire an image of the scene to be detected through a visual sensor, which is used as the first image.

[0089] The few-shot target detection module 200 is configured to send the first image into the student network of a pre-trained pseudo-label-based semi-supervised few-shot target detection network to obtain the target detection result corresponding to the scene image to be detected.

[0090] The pseudo-label-based semi-supervised few-shot target detection network includes a pre-trained TFA network, a teacher network, and a student network; the network structures of the pre-trained TFA network, the teacher network, and the student network are the same.

[0091] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the system described above can be found in the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0092] It should be noted that the pseudo-label-based semi-supervised small-sample target detection system provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the various modules or steps and are not considered as an improper limitation of the present invention.

[0093] A storage device according to a third embodiment of the present invention stores multiple programs, which are adapted to be loaded and executed by a processor to implement the above-described pseudo-label-based semi-supervised few-sample target detection method.

[0094] A processing apparatus according to a fourth embodiment of the present invention includes a processor and a storage device; the processor is adapted to execute various programs; the storage device is adapted to store multiple programs; the programs are adapted to be loaded and executed by the processor to implement the above-described pseudo-label-based semi-supervised small sample target detection method.

[0095] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the storage device and processing device described above can be referred to the corresponding process in the foregoing method examples, and will not be repeated here.

[0096] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. The programs corresponding to the software modules and method steps can be placed in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the invention.

[0097] The terms “first,” “second,” “third,” etc., are used to distinguish similar objects, not to describe or indicate a specific order or sequence.

[0098] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A semi-supervised few-sample target detection method based on pseudo-labels, characterized in that, The method includes the following steps: Step S10: Acquire an image of the scene to be detected using a visual sensor, and use it as the first image; Step S20: The first image is fed into the student network of the pre-trained pseudo-label-based semi-supervised few-shot object detection network to obtain the object detection result corresponding to the scene image to be detected. The pseudo-label-based semi-supervised few-shot target detection network includes a pre-trained TFA network, a teacher network, and a student network; the pre-trained TFA network, the teacher network, and the student network have the same network structure; the training method for the student network is as follows: Step A10: Obtain training data, which includes unlabeled training images and a small number of new class labeled samples; input the unlabeled training images into a pre-trained TFA network to obtain the prediction results of the new class target objects in the unlabeled training images, which are used as the first prediction results; the unlabeled training images are images of new class target objects in the base class dataset of the COCO dataset. Step A20: Remove the first object candidate boxes with confidence scores lower than the set background threshold from the first prediction result, and calculate the values ​​of the remaining first object candidate boxes of the same type at the set quantiles as the primary category adaptive thresholds for each new class, and use the first object candidate boxes with confidence scores higher than the primary category adaptive thresholds of the corresponding class as the first pseudo-labels; the first object candidate boxes are the object candidate boxes corresponding to the first prediction result. Step A30: Perform weak data augmentation and strong data augmentation on the unlabeled training images to obtain weakly augmented images and strongly augmented images; input the weakly augmented images into the teacher network to obtain the prediction results of the new class of target objects in the weakly augmented images, which are used as the second prediction results; Step A40: Remove the second object candidate boxes with confidence scores lower than the set background threshold from the second prediction result, and in the remaining second object candidate boxes, use the second object candidate boxes with confidence scores higher than the advanced category adaptive threshold of the corresponding category as the second pseudo-label; the second object candidate box is the object candidate box corresponding to the second prediction result; Step A50: Fuse the first pseudo-label and the second pseudo-label to obtain a third pseudo-label; then combine the third pseudo-label with the strongly augmented image. A new class of unlabeled samples is formed. The new class of unlabeled samples and a small number of new class labeled samples are input into the student network, and the loss value is calculated. The weight parameters of the student network are updated, and the weight parameters of the teacher network are updated using the updated weight parameters of the student network. Step A60, repeat steps A10-A50 until a trained student network is obtained.

2. A semi-supervised small-sample target detection method based on pseudo-labels as described in claim 1, characterized in that, The method for calculating the values ​​of the remaining first object candidate boxes of the same type at a set quantile is as follows: The quantile function in the pandas library, a third-party data analysis tool library in Python, is used to calculate the value of the first object candidate box of the remaining class at a set quantile.

3. A semi-supervised small-sample target detection method based on pseudo-labels as described in claim 1, characterized in that, In the remaining second object candidate boxes, the second object candidate boxes with confidence scores higher than the corresponding category's high-level category adaptive threshold are used as the second pseudo-labels. The method is as follows: The advanced category adaptive threshold is initialized with the primary category adaptive threshold and is dynamically adjusted and updated during training; the dynamic adjustment and update process is as follows: Recording Accumulation The first mini-batch generated the first The number of second pseudo-labels for each new class is , To accumulate the number of mini-batches, when Calculate adaptive quantiles for categories ,in, For accumulation The first mini-batch obtained The number of candidate pseudo-labels for each new class. This represents the maximum number of pseudo-tags. when Calculate adaptive quantiles for categories ,in, This minimizes the number of pseudo-labels; in other cases, the category-adaptive quantile remains unchanged. Then, the quantile function from the pandas library, a third-party data analysis tool library in Python, was used to calculate the confidence score of those belonging to the same new class, which is not lower than the background threshold. The value of the second object candidate box at the category adaptive quantile is used to update the advanced category adaptive threshold; The second object candidate box in the second prediction result whose confidence score is higher than the high-level category adaptive threshold of the corresponding category is used as the second pseudo label.

4. A semi-supervised small-sample target detection method based on pseudo-labels as described in claim 1, characterized in that, The first pseudo-tag and the second pseudo-tag are merged to obtain the third pseudo-tag. The method is as follows: The first pseudo-label and the second pseudo-label are merged, and the merged result is used as a candidate set of pseudo-labels. In the pseudo-label candidate set, non-maximum suppression (NMS) is used to suppress pseudo-labels with low confidence scores belonging to the same object, thereby obtaining the label fusion result, i.e., the third pseudo-label; wherein, the criterion for judging that two pseudo-labels in the pseudo-label candidate set belong to the same object is that they are of the same category and their Intersection over Union (IoU) is not lower than a set IoU threshold.

5. A semi-supervised small-sample target detection method based on pseudo-labels as described in claim 1, characterized in that, The loss function of the student network during training is: ； ； ； in, The total loss of the student network during training. To address the supervised loss for newly labeled samples, To address the unsupervised loss for new types of unlabeled samples, For loss weight hyperparameters, and These represent the number of newly labeled samples and the number of unlabeled samples of the new class in a mini-batch, respectively. Indicates the first A new class-labeled sample, Indicates the first A new class of unlabeled samples, These represent the RPN classification loss, RPN regression loss, RoI head classification loss, and RoI head regression loss, respectively.

6. A semi-supervised small-sample target detection method based on pseudo-labels as described in claim 5, characterized in that, The method for updating the weight parameters of the student network and then using the updated weight parameters of the student network to update the weight parameters of the teacher network is as follows: ； ； in, The learning rate hyperparameter, To smooth hyperparameters, For the weight parameters of the student network, This represents the weight parameters of the teacher network.

7. A semi-supervised small-sample target detection system based on pseudo-labels, characterized in that, include: Image acquisition module, small sample target detection module; The image acquisition module is configured to acquire an image of the scene to be detected through a visual sensor, which is used as the first image. The few-shot target detection module is configured to send the first image into the student network of a pre-trained pseudo-label-based semi-supervised few-shot target detection network to obtain the target detection result corresponding to the scene image to be detected. The pseudo-label-based semi-supervised few-shot target detection network includes a pre-trained TFA network, a teacher network, and a student network; the pre-trained TFA network, the teacher network, and the student network have the same network structure. The training method for the student network is as follows: Step A10: Obtain training data, which includes unlabeled training images and a small number of new class labeled samples; input the unlabeled training images into a pre-trained TFA network to obtain the prediction results of the new class target objects in the unlabeled training images, which are used as the first prediction results; the unlabeled training images are images of new class target objects in the base class dataset of the COCO dataset. Step A20: Remove the first object candidate boxes with confidence scores lower than the set background threshold from the first prediction result, and calculate the values ​​of the remaining first object candidate boxes of the same type at the set quantiles as the primary category adaptive thresholds for each new class, and use the first object candidate boxes with confidence scores higher than the primary category adaptive thresholds of the corresponding class as the first pseudo-labels; the first object candidate boxes are the object candidate boxes corresponding to the first prediction result. Step A30: Perform weak data augmentation and strong data augmentation on the unlabeled training images to obtain weakly augmented images and strongly augmented images; input the weakly augmented images into the teacher network to obtain the prediction results of the new class of target objects in the weakly augmented images, which are used as the second prediction results; Step A40: Remove the second object candidate boxes with confidence scores lower than the set background threshold from the second prediction result, and in the remaining second object candidate boxes, use the second object candidate boxes with confidence scores higher than the advanced category adaptive threshold of the corresponding category as the second pseudo-label; the second object candidate box is the object candidate box corresponding to the second prediction result; Step A50: Fuse the first pseudo-label and the second pseudo-label to obtain a third pseudo-label; then combine the third pseudo-label with the strongly augmented image. A new class of unlabeled samples is formed. The new class of unlabeled samples and a small number of new class labeled samples are input into the student network, and the loss value is calculated. The weight parameters of the student network are updated, and the weight parameters of the teacher network are updated using the updated weight parameters of the student network. Step A60, repeat steps A10-A50 until a trained student network is obtained.

8. A storage device storing a plurality of programs, characterized in that, The program is applicable to being loaded and executed by a processor to implement a pseudo-label-based semi-supervised few-sample target detection method as described in any one of claims 1-6.

9. A processing apparatus, comprising a processor and a storage device; the processor being adapted to execute various programs; the storage device being adapted to store multiple programs; characterized in that, The program is applicable to being loaded and executed by a processor to implement a pseudo-label-based semi-supervised few-sample target detection method as described in any one of claims 1-6.

Images