Small sample object detection method, device, storage medium and electronic device

By conducting migration structure training on the detection network of small sample objects, reconstructing query and support features, the problem of inaccurate recognition of small sample objects is solved, the accuracy of recognition and positioning is improved, and it is suitable for fields such as intelligent transportation.

CN118673174BActive Publication Date: 2024-11-26ZHEJIANG DAHUA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411159397.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-11-26
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

The identification of small sample objects in the prior art is inaccurate, especially when the training data is insufficient in deep learning, resulting in overfitting problems.

Method used

By inputting query images and support images into the initial network model, the migration structure is used to reconstruct query instance features and support instance features, narrowing the feature distance of similar parts and pushing away the feature distance of different parts, thereby improving the target detection network accuracy.

Benefits of technology

It improves the recognition accuracy and positioning accuracy of small sample objects, solves the problem of inaccurate recognition of small sample objects, and is suitable for fields such as intelligent transportation and license plate recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118673174B_ABST
    Figure CN118673174B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention provides a method, device, storage medium and electronic device for detecting small sample objects, wherein the method comprises: inputting a small sample image into a target detection network to determine the target object, the target category of the target object and the target candidate frame of the target object included in the small sample image; the target detection network is trained in the following manner: inputting a query image and a plurality of support images into a detection subnetwork model included in an initial network model to obtain a query instance feature of the query image and a support instance feature of each support image; inputting the query instance feature and the plurality of support instance features into a migration structure included in the initial network model for reconstruction to obtain a target reconstruction query feature and a target reconstruction support feature; determining the target loss value of the initial network model based on the reconstruction query feature and the reconstruction support feature. Through the present invention, the accuracy of identifying small sample objects is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing, and in particular, to a method, device, storage medium and electronic device for detecting small sample objects. Background Art

[0002] Deep learning relies on large-scale labeled data, and is prone to overfitting when the scale of training data is insufficient. Small sample learning studies how to achieve the effect of traditional deep learning with very little labeled data. In related technologies, small sample object detection methods mainly include two categories: meta-learning-based methods and fine-tuning-based methods. However, in related technologies, there is a problem of inaccurate recognition of small sample objects.

[0003] Currently, no effective solution has been proposed for the above-mentioned problems existing in the related technologies. Summary of the invention

[0004] Embodiments of the present invention provide a method, device, storage medium and electronic device for detecting small sample objects, so as to at least solve the problem of inaccurate recognition of small sample objects existing in the related art.

[0005] According to one embodiment of the present invention, a method for detecting small sample objects is provided, comprising: inputting a small sample image into a target detection network to determine a target object included in the small sample image, a target category of the target object, and a target candidate box of the target object; wherein the target detection network is trained in the following manner: inputting a query image and a plurality of support images into a detection subnetwork model included in an initial network model to obtain a query instance feature of the query image and a support instance feature of each of the support images, wherein the query image is an unlabeled image, the support image is a labeled image, the support image includes a support object and labeling information, and the labeling information includes the type of the support object and the support object The query instance feature is a feature of the region where the query object included in the query image is located predicted by the detection sub-network model, and the support instance feature is a feature of the region where the candidate frame is located determined by the detection sub-network model based on the annotation information; the query instance feature and a plurality of the support instance features are input into the migration structure included in the initial network model for reconstruction to obtain a target reconstruction query feature and a target reconstruction support feature; a target loss value of the initial network model is determined based on the reconstructed query feature and the reconstructed support feature; a network parameter of the initial network model is iteratively updated based on the target loss value to obtain an initial detection model; the initial detection model is trained using the acquired small sample annotated data to obtain the target detection network.

[0006] According to another embodiment of the present invention, a detection device for a small sample object is provided, comprising: a detection module, for inputting a small sample image into a target detection network to determine a target object included in the small sample image, a target category of the target object, and a target candidate box of the target object; wherein the target detection network is trained in the following manner: a query image and a plurality of support images are input into a detection subnetwork model included in an initial network model to obtain a query instance feature of the query image and a support instance feature of each of the support images, wherein the query image is an unlabeled image, the support image is a labeled image, the support image includes a support object and labeling information, and the labeling information includes the type of the support object and the A candidate box of a supporting object, the query instance feature is a feature of the area where the query object is located included in the query image predicted by the detection sub-network model, and the supporting instance feature is a feature of the area where the candidate box is located determined by the detection sub-network model based on the annotation information; the query instance feature and a plurality of the supporting instance features are input into the migration structure included in the initial network model for reconstruction to obtain a target reconstruction query feature and a target reconstruction support feature; a target loss value of the initial network model is determined based on the reconstructed query feature and the reconstructed support feature; a network parameter of the initial network model is determined to be iteratively updated based on the target loss value to obtain an initial detection model; the initial detection model is trained using the acquired small sample annotated data to obtain the target detection network.

[0007] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0008] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0009] According to yet another embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the method described in each embodiment of the present application are implemented.

[0010] Through the present invention, the target detection network for identifying small sample images is to train the initial network model through the query image and the support image to obtain the initial detection model, and then train the output detection model using the small sample annotated data to obtain the target detection network. Among them, the migration structure is used when training the output network model, and the query instance features of the query image and the support instance features of the support image are reconstructed through the migration results, shortening the feature distance of the similar parts between the support features and the query features, and at the same time pushing away the feature distance of the dissimilar parts between them, thereby improving the effect of feature alignment, and further improving the accuracy of locating the target object and determining the type of the target object. Therefore, the problem of inaccurate recognition of small sample objects existing in the related art can be solved, and the effect of improving the accuracy of identifying small sample objects can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a hardware structure block diagram of a mobile terminal of a small sample object detection method according to an embodiment of the present invention;

[0012] Figure 2 is a flow chart of a method for detecting a small sample object according to an embodiment of the present invention;

[0013] Figure 3 is a schematic diagram of an initial network model structure according to an embodiment of the present invention;

[0014] Figure 4 is a flow chart of a method for detecting a small sample object according to a specific embodiment of the present invention;

[0015] Figure 5 4 is a structural block diagram of a detection device for a small sample object according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0017] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0018] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal for a small sample object detection method according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0019] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the detection method of small sample objects in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0020] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0021] In this embodiment, a method for detecting small sample objects is provided. Figure 2 is a flow chart of a method for detecting a small sample object according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0022] Step S202, inputting the small sample image into the target detection network to determine the target object included in the small sample image, the target category of the target object, and the target candidate box of the target object;

[0023] The target detection network is trained in the following way:

[0024] Input a query image and a plurality of support images into a detection sub-network model included in an initial network model, and obtain a query instance feature of the query image and a support instance feature of each of the support images, wherein the query image is an unlabeled image, the support images are labeled images, the support images include support objects and labeling information, the labeling information includes the type of the support object and a candidate box of the support object, the query instance feature is a feature of an area where the query object included in the query image is located predicted by the detection sub-network model, and the support instance feature is a feature of an area where the candidate box is located determined by the detection sub-network model based on the labeling information;

[0025] The query instance feature and the plurality of supporting instance features are input into the migration structure included in the initial network model for reconstruction to obtain a target reconstruction query feature and a target reconstruction support feature; a target loss value of the initial network model is determined based on the reconstruction query feature and the reconstruction support feature; a network parameter of the initial network model is iteratively updated based on the target loss value to obtain an initial detection model; and the initial detection model is trained using the acquired small sample annotated data to obtain the target detection network.

[0026] In the above embodiment, the small sample image can be an image in the field of intelligent transportation. In the field of intelligent transportation, due to the small number of times a certain type of object appears on the road, there are not enough samples to learn the image of this type of object, resulting in inaccurate recognition of this type of object. For example, in urban roads, the probability of a horse-drawn carriage is low, so there will be inaccurate recognition when using a network model to identify a horse-drawn carriage. Therefore, a detection method for sampling a small sample object can be used to identify a small sample image to improve the accuracy of identifying the object and the accuracy of locating the object. In other words, the detection method for a small sample object can be applied in the field of intelligent transportation. Of course, the detection method for a small sample object can also be applied in the field of license plate recognition. Since it is impossible to obtain license plate images in remote areas during model training, the recognition of license plates in remote areas is inaccurate. Therefore, the detection method for a small sample object can be used to identify license plates to improve the accuracy of license plate recognition.

[0027] It should be noted that the small sample object detection method can also be applied to other fields, and the present invention does not limit this. The small sample object detection method is applied in different fields, and the target objects can be different, and the target objects can include people, cars, animals, etc.

[0028] In the above embodiment, the target detection network can be a framework with Faster RCNN as the target detection algorithm. Faster RCNN can be selected as the framework of the initial network model to build the initial network model. The initial network model may include a detection sub-network model for extracting image features. The initial network model may also include a migration structure for aligning query features and supporting features. The detection sub-network model may include a feature extraction network, an RPN (region proposal network) network, and an RoI Pooling (Region of interest pooling). The migration structure may include a reconstruction module and a domain adversarial module. The initial network model may also include a detection head for determining candidate boxes and classifications of objects.

[0029] In the above embodiment, the support image can be annotated training data, and the number of support images can be multiple, that is, a large number of support images can be used to train the initial network model, and the multiple support images include objects of basic categories. The input of the network of the initial network model can be a set of support images of n categories and a query image. The support images are some images with targets already annotated, which are used for auxiliary training, and the query images are some unknown images, which are images to be detected. In the first stage of training, that is, the stage of obtaining the initial detection network, only the basic categories are considered. The basic categories are some categories with a large amount of annotated data, and each basic category provides a large number of samples. In the second stage, that is, the stage of obtaining the target detection network, each small sample category provides K samples, and the small sample category is some categories with only a small amount of annotated data, where K is set to 1, 2, 3, 5 and 10, etc.

[0030] In the above embodiment, the first stage of training can be performed first. The first stage of training is performed on basic categories, which are categories with a large amount of labeled data, such as people, cars, dogs, etc. The purpose of this stage is to train the basic detection capabilities of the detector so that the network can accurately locate and classify targets of the basic categories, and learn some common feature representations to lay the foundation for subsequent learning. Then, the second stage of training is performed. The second stage of training is performed on small sample categories, which are categories with only a small amount of labeled data, such as cats, birds, flowers, etc. The purpose of this stage is to optimize and adjust the network so that the network can adapt to targets of small sample categories based on the information of the support set, improve the accuracy and robustness of detection, and avoid the problems of overfitting and catastrophic forgetting.

[0031] In the above embodiment, the query image and the support image can be respectively input into the detection sub-network model to obtain the query instance features of the query image and the support instance features of the support image. The query instance features are the features of the region including the query object predicted by the detection sub-network model in the query image, and the support instance features are the features of the region determined by the detection sub-network model according to the position of the candidate box in the annotation information. The query instance features and the support instance features can be input into the migration structure for reconstruction to obtain the target reconstruction query features and the target reconstruction support features, and the target loss value of the initial network model can be determined according to the target reconstruction query features and the target reconstruction support features.

[0032] In the above embodiment, after the initial detection model is obtained, the detector included in the initial detection model can be retrained on a small sample category with a small amount of data, and the small sample category can be optimized and adjusted, such as increasing category specificity, reducing category confusion, and improving category sensitivity. The purpose of small sample target detection is to improve the detection accuracy of the detector for small sample categories while ensuring the generalization ability of the detector.

[0033] Through the present invention, the target detection network for identifying small sample images is to train the initial network model through the query image and the support image to obtain the initial detection model, and then train the output detection model using the small sample annotated data to obtain the target detection network. Among them, the migration structure is used when training the output network model, and the query instance features of the query image and the support instance features of the support image are reconstructed through the migration results, shortening the feature distance of the similar parts between the support features and the query features, and at the same time pushing away the feature distance of the dissimilar parts between them, thereby improving the effect of feature alignment, and further improving the accuracy of locating the target object and determining the type of the target object. Therefore, the problem of inaccurate recognition of small sample objects existing in the related art can be solved, and the effect of improving the accuracy of identifying small sample objects can be achieved.

[0034] Optionally, the executor of the above steps may be a background processor, or other devices with similar processing capabilities, or a machine that integrates at least an image acquisition device and a data processing device, wherein the image acquisition device may include a graphics acquisition module such as a camera, and the data processing device may include a computer, a mobile phone or other terminal, but is not limited thereto.

[0035] In an exemplary embodiment, the query instance feature and the plurality of supporting instance features are input into the migration structure included in the initial network model for reconstruction, and obtaining the target reconstructed query feature and the target reconstructed support feature includes: reconstructing the query instance feature and the supporting instance feature based on the reconstruction module and the domain adversarial module included in the migration structure to obtain the target reconstructed query feature and the target reconstructed support feature. In this embodiment, the migration structure may include a reconstruction module and a domain adversarial module. There are two inputs to the migration structure, one is the supporting feature of the instance, i.e., the supporting instance feature, and the other is the query feature of the instance, i.e., the query instance feature. The supporting instance feature is a feature extracted from the labeled data of the source domain, which contains the category information and sample information of the source domain, and can be used as the prior knowledge of the target domain detection task. The query feature is a feature extracted from the unlabeled data of the target domain, which contains the data distribution and sample information of the target domain, and can be used as the input data of the target domain detection task.

[0036] In an exemplary embodiment, reconstructing the query instance feature and the support instance feature based on the reconstruction module and the domain confrontation module included in the migration structure includes: inputting the query instance feature and the multiple support instance features into the domain confrontation module to determine the first source domain of the query instance feature and any one of the target instance features included in the multiple support instance features; inputting the query instance feature, the multiple support instance features and the source domain into the reconstruction module to map the query instance feature and the multiple support instance features to the same feature space for reconstruction to obtain a first reconstructed query feature and a first reconstructed support feature; inputting the first reconstructed query feature and the first reconstructed support feature into the domain confrontation module to determine the second source domain of the first reconstructed query feature and any one of the target reconstruction features included in the first reconstructed support feature; mapping the first reconstructed query feature to the initial feature space of the query instance feature based on the second source domain for reconstruction to obtain a second reconstructed query feature, mapping the first reconstructed support feature to the initial feature space of the support instance feature for reconstruction to obtain a second reconstructed support feature; determining the second reconstructed query feature as the target reconstruction query feature, and determining the second reconstruction support feature as the target reconstruction support feature. In this embodiment, the reconstruction module may include an encoder and a decoder, the encoder maps the support instance features and the query instance features to the same feature space, and then sends the mapped support features and query features to the encoder for reconstruction, and then maps the reconstructed support features and query features back to their respective feature domains. The domain adversarial module introduces a domain discriminator before and after the feature mapping, the purpose of which is to determine whether the input features are from the source domain or the target domain, and to perform domain adversarial operations on the original support features and query features and the mapped support features and query features respectively. The goal of the domain discriminator is to maximize the domain difference between the support features and the query features, thereby prompting the encoder to minimize this difference. In this way, the feature distance of the similar parts between the support features and the query features can be shortened, while the feature distance of the dissimilar parts between them can be pushed away, thereby improving the effect of feature alignment. The output of the migration structure is the reconstructed support features and query features, which can be used for further detection tasks.

[0037] In the above embodiment, the encoder can be used to encode the input data into an implicit vector, and then the decoder can be used to decode the vector into output data, thereby realizing data reconstruction. The function of the encoder is to take a sequence of variable length as input and then turn it into a coded state of fixed shape. It is implemented through a recursive neural network layer, generally using a recurrent neural network or a long short-term memory network. The encoder converts each word element of the input sequence into a vector representation, and these vectors are finally encoded into a state representation of a fixed dimension after being transmitted through multiple layers. This state contains all the information of the input sequence for subsequent decoding processes. The task of the decoder is to generate the next word element based on the output state of the encoder and the part of the target sequence that has been generated. It also uses a recursive neural network layer, and generally a recurrent neural network or a long short-term memory network can also be used. At each time step, the decoder takes the currently generated partial sequence and the output state of the encoder as input, and then generates the next most likely word element. This process will be repeated until the complete target sequence is generated or the preset length is reached.

[0038] In this way, the reconstruction module can learn the intrinsic structure and features of the data, thereby extracting useful information. The application scenario of the reconstruction module is the migration optimization of support features and query features. Since support features and query features may come from different distributions, there is an offset in the feature space, which leads to a decrease in the performance of the reconstruction module. To solve this problem, the encoder of the reconstruction module maps the support features and query features to the same feature space, extracts higher-level semantic information, and facilitates information interaction between query features and support features. This can enhance the expressive power of the query feature and make it more suitable for the scenario of the target task. The decoder maps the feature back to the original space for subsequent classification and positioning tasks. The input of the network is the support features and the query features optimized by the graph attention network, and the output is the support features and query features after migration optimization. The query features are input into the detection head to output the detection results. Such a reconstruction module can effectively reduce the offset of the feature space and improve the accuracy and robustness of detection.

[0039] In the above embodiment, the domain adversarial module can use the domain discriminator and the feature extractor for adversarial learning, so that the feature extractor can extract cross-domain common features and remove domain-specific private features. The role of the domain discriminator is to determine which domain the feature comes from, and the role of the feature extractor is to extract features and deceive the domain discriminator so that it cannot distinguish features from different domains. In this way, the alignment and migration of feature space can be achieved, domain offset can be reduced, and generalization ability can be improved. The loss function of the domain discriminator is the binary cross entropy loss, which is a commonly used classification loss function. It calculates the mean of the cross entropy between the predicted value and the true value. Cross entropy is an indicator to measure the difference between two probability distributions. The smaller it is, the closer the two distributions are, and the larger it is, the farther the two distributions are. Binary cross entropy loss can reflect the classification ability of the domain discriminator, and the smaller the loss is, the higher the accuracy. The goal of the domain discriminator is to distinguish the support features of the source domain from the query features of the target domain. Therefore, the true label of the support features of the source domain can be set to 0, and the true label of the query features of the target domain can be set to 1, so that a binary classification problem can be constructed. The output of the domain discriminator is a probability value between 0 and 1, indicating the possibility that the feature belongs to the target domain. The closer it is to 0, the more likely it is from the source domain, and the closer it is to 1, the more likely it is from the target domain.

[0040] In an exemplary embodiment, the query image and multiple support images are input into the detection sub-network model included in the initial network model to obtain the query instance features of the query image and the support instance features of each support image, including: inputting the query image and the support images into the feature extraction network included in the detection sub-network model to obtain query features and support features; inputting the query features into the extraction candidate box network included in the detection sub-network model to obtain query global features; inputting the support features into the extraction candidate box network to obtain support global features; inputting the query global features into the pooling layer included in the detection sub-network model to obtain query instance features; inputting the support features into the pooling layer to obtain support instance features. In this embodiment, the feature extraction network can be a feature extraction network with shared weights, such as a twin network. The feature extraction network adopts the structure of a twin neural network, and the support set and the query set share the feature extraction network. The network processes a set of n categories of support images and a query image with the same model and parameters to obtain their global features. Support images are some images that have been labeled with targets and are used to assist training. Query images are some unknown images that are to be detected. A twin neural network is a network that can compare the similarity of two inputs. It consists of two identical sub-networks, each with the same structure and parameters. They share weights to ensure the consistency of feature extraction. Using a twin neural network can put the features of support images and query images in the same feature space, which facilitates subsequent feature matching and classification.

[0041] The extracted support features and query features are sent to the RPN network, that is, the candidate box extraction network. The purpose of the RPN network is to generate some candidate target areas, thereby reducing the search space and improving detection efficiency. The feature extraction network extracts global features, and performs RoI Pooling on the global features to obtain instance features. Instance features refer to the features of each target. It is a sub-region of the global feature, which contains the location and category information of the target. The purpose of using RoI Pooling is to extract instance features from global features, so as to achieve the positioning and classification of the target. The RoI Pooling methods for support features and query features are different. The border of the RoI Pooling of support features is given by the real positioning box, while the positioning box of the query feature RoI Pooling is a candidate box obtained by preliminary regression of the anchor box through the RPN network.

[0042] In an exemplary embodiment, the query image and the support image are input into the feature extraction network included in the detection subnetwork model to obtain the query feature, and the support feature includes: inputting the query image and the support image into the residual structure included in the twin network model to obtain the initial query feature and the initial support feature, wherein the feature extraction network includes the twin network model; inputting the initial query feature into the feature pyramid structure included in the twin network model to obtain the query feature; inputting the initial support feature into the feature pyramid structure to obtain the query feature. In this embodiment, the twin network is a weight-shared feature extraction network. The feature extraction network uses a network structure combining Resnet and FPN. Resnet uses a residual structure, which can effectively solve the gradient explosion problem after the network is deepened, and extract image features from bottom to top. FPN uses a feature pyramid structure, which is a solution to the multi-scale problem in object detection. It significantly improves the detection performance of small objects by simply changing the connection mode of the network without significantly increasing the computational amount of the original model, and connects the features extracted by Resnet horizontally from top to bottom.

[0043] In an exemplary embodiment, determining the target loss value of the initial network model based on the reconstructed query feature and the reconstructed support feature includes: determining a first mean square error loss between the query feature and the reconstructed query feature; determining a second mean square error loss between the support feature and the reconstructed support feature; determining the sum of the first mean square error loss and the second mean square error loss; and determining the target loss value based on the sum. In this embodiment, the loss of the reconstructed part is the mean square error loss of the query feature before the reconstruction module and the query feature after the reconstruction, plus the mean square error loss of the support feature before the reconstruction and the support feature after the reconstruction. The target loss value can be determined based on the sum of the two.

[0044] In an exemplary embodiment, determining the target loss value based on the sum value includes: determining the classification loss value and the positioning loss value of the classifier and the locator included in the initial network model; determining the target loss value based on the classification loss value, the positioning loss value and the sum value. In this embodiment, the initial network model may also include a classifier and a locator, which are respectively used to determine the category of the object in the query image and the position of the candidate box. The classifier can use a sampled cross entropy loss function to perform category regression of the detection box, and the locator can use a smooth L1 loss function to perform position regression of the detection box. The input of the detection head is the reconstructed query feature, and the output is the position and category of the detection box in the query image.

[0045] In the above embodiment, the classification loss value, the positioning loss value and the sum value can be determined as the target loss value, which are used to adjust the model parameters of the classifier, the model parameters of the locator and the model parameters of the migration structure respectively. The three can also be summed and the sum of the three can be used as the target loss value. The summation can include weighted summation and direct summation.

[0046] In the above embodiment, after obtaining the initial detection model, the detection head, i.e., the classifier and the locator, can be retrained on a small sample category with a small amount of data, and the small sample category can be optimized and adjusted, such as increasing category specificity, reducing category confusion, and improving category sensitivity.

[0047] In the above embodiment, the schematic diagram of the initial network model structure can be found in the attached Figure 3 ,like Figure 3 As shown in the figure, the initial network model can be based on the Meta RCNN network architecture to solve the problem of small sample target detection. This network uses the concepts of support set and query set. The support set, that is, the support images are some pictures of known categories, and the query set, that is, the query images are some pictures of unknown categories. The purpose is to enable the network to assist in detecting targets in the query set based on the information of the support set.

[0048] First, the Meta RCNN network inputs the support set image into the PRN network. The main part of the PRN network has the same feature extraction network structure as Faster R-CNN and shares weights, which can improve the generalization ability and efficiency of the network. The feature extraction network uses a network structure that combines Resnet and FPN. Resnet uses a residual structure, which can effectively solve the gradient explosion problem after the network is deepened and extract image features from the bottom to the top. FPN uses a feature pyramid structure, which is a solution to the multi-scale problem in object detection. It significantly improves the detection performance of small objects by simply changing the connection method of the network without significantly increasing the computational complexity of the original model. The features extracted by Resnet are connected horizontally from top to bottom. The feature extraction network adopts the structure of a twin neural network, and the support set and the query set share the feature extraction network. Then, the support features output by the PRN network are used element by element to obtain a set of corresponding attention vectors. The role of the attention vector is to weight the support features, highlight important features, and suppress irrelevant features. Next, the query set image is input into the Faster R-CNN network to obtain the corresponding feature map. The Faster R-CNN network is a classic target detection network, which consists of an RPN network and a detection head. The RPN network is responsible for generating candidate regions, and the detection head is responsible for classifying and regressing the candidate regions. Finally, the class attention vector generated by the PRN network is used to perform channel-level soft attention weighting on the RoI region feature map generated by the RPN network to obtain the fused features, which are then input into the detection head for prediction. In this way, the attention vector acts as a regulator, enabling the detection head to adapt to different categories based on the information of the support set and improve the accuracy of detection. The loss function design of the detection head is the conventional target detection loss function, which consists of the classification loss of the detection box and the positioning loss of the detection box. The advantage of this network is that it can effectively use a small amount of labeled data to learn the target detection task, and it can also adapt to new categories and has strong generalization ability.

[0049] The reason why Meta R-CNN is chosen as the basic framework is that the Meta R-CNN network architecture is relatively simple. Faster R-CNN is used as the detection framework, which is convenient for modification. The Meta R-CNN network is a metric learning-based network that can quickly learn and adapt on a small amount of labeled data and has strong generalization ability. The core idea of ​​the Meta R-CNN network is to use the information of the support set to adjust the parameters of the detector, thereby realizing the detection of small sample categories.

[0050] The following describes a method for detecting small sample objects in conjunction with a specific implementation method:

[0051] Figure 4 is a flow chart of a method for detecting a small sample object according to a specific embodiment of the present invention. Figure 4 As shown, the process includes:

[0052] Step S402, construct a small sample target detection model based on transfer learning. For example, FasterRCNN can be selected as the framework of the target detection algorithm, including a twin network, an RPN (region proposal network) network and RoIPooling (Region of interest pooling), a transfer structure, and a detection head.

[0053] Step S404, the basic training phase, is conducted on a public dataset with a large amount of rich annotated data to train the basic detection capabilities of the detector, such as feature extraction, candidate box generation, classification and regression, etc. The image input is input in parallel by the support image and the query image, and enters the weight-sharing feature extraction network, that is, the twin network, for feature extraction, and then the extracted support features and query features are input into the RPN to filter out the candidate boxes of potential targets. After obtaining the candidate boxes, RoI Pooling is performed, and then feature alignment is performed in the migration structure, and finally the regression of the category and detection box is completed in the detection head.

[0054] Step S406, optimization training phase, the detector is retrained on a small sample category with a small amount of data, and the small sample category is optimized and adjusted, such as increasing category specificity, reducing category confusion, improving category sensitivity, etc. The purpose of small sample target detection is to improve the detection accuracy of the detector for small sample categories while ensuring the generalization ability of the detector.

[0055] In the above-mentioned embodiment, the reconstruction module regards the feature space where the support features are located as the source domain and the feature space where the query features are located as the target domain, maps the features of the two domains to the same feature space for processing, and then maps them back to their respective feature spaces; the feature migration and alignment between the support set and the query set in small sample target detection is realized. The domain adversarial module determines whether the input features are from the source domain or the target domain, and conducts domain adversarial operations on the original support features and query features as well as the mapped support features and query features. The goal of the domain discriminator is to maximize the domain difference between the support features and the query features, thereby prompting the encoder to minimize this difference. Through adversarial training, the feature distance between the support set and the query set is reduced, and the generalization ability of the features is improved. The information of the support features is fully utilized to assist the query features, shorten the feature distance of the similar parts between the support features and the query features, and push the feature distance of the dissimilar parts between them, thereby improving the effect of feature alignment; the encoder-decoder and the domain discriminator are utilized, and the migration structure is fully utilized to enhance the query features to help subsequent detection; the network structure has always been the same, and no other network structures are added, so the memory space occupied is relatively small, and the time complexity is also small.

[0056] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0057] In the present embodiment, a detection device for a small sample object is also provided, and the device is used to implement the above-mentioned embodiments and preferred embodiments, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0058] Figure 5 is a structural block diagram of a detection device for a small sample object according to an embodiment of the present invention. Figure 5 As shown, the device comprises:

[0059] A detection module 52, used for inputting a small sample image into a target detection network to determine a target object, a target category of the target object, and a target candidate box of the target object included in the small sample image;

[0060] The target detection network is trained in the following way:

[0061] Input a query image and a plurality of support images into a detection sub-network model included in an initial network model, and obtain a query instance feature of the query image and a support instance feature of each of the support images, wherein the query image is an unlabeled image, the support images are labeled images, the support images include support objects and labeling information, the labeling information includes the type of the support object and a candidate box of the support object, the query instance feature is a feature of an area where the query object included in the query image is located predicted by the detection sub-network model, and the support instance feature is a feature of an area where the candidate box is located determined by the detection sub-network model based on the labeling information;

[0062] Inputting the query instance feature and the plurality of supporting instance features into the migration structure included in the initial network model for reconstruction, to obtain a target reconstruction query feature and a target reconstruction supporting feature;

[0063] Determining a target loss value of the initial network model based on the reconstruction query feature and the reconstruction support feature;

[0064] Determine, based on the target loss value, to iteratively update the network parameters of the initial network model to obtain an initial detection model;

[0065] The initial detection model is trained using the acquired small sample annotated data to obtain the target detection network.

[0066] In an exemplary embodiment, the detection module 52 can implement the following method to reconstruct the query instance feature and the multiple supporting instance features into the migration structure included in the initial network model to obtain the target reconstructed query feature and the target reconstructed support feature: reconstruct the query instance feature and the supporting instance feature based on the reconstruction module and the domain adversarial module included in the migration structure to obtain the target reconstructed query feature and the target reconstructed support feature.

[0067] In an exemplary embodiment, the detection module 52 can reconstruct the query instance feature and the supporting instance feature based on the reconstruction module and the domain adversary module included in the migration structure in the following manner: input the query instance feature and the multiple supporting instance features into the domain adversary module to determine the first source domain of the query instance feature and any one of the target instance features included in the multiple supporting instance features; input the query instance feature, the multiple supporting instance features and the source domain into the reconstruction module to map the query instance feature and the multiple supporting instance features to the same feature space for reconstruction, and obtain a first reconstructed query feature and a first source domain. a reconstruction support feature; inputting the first reconstruction query feature and the first reconstruction support feature into the domain adversarial module to determine the second source domain of the first reconstruction query feature and any one target reconstruction feature included in the first reconstruction support feature; mapping the first reconstruction query feature to the initial feature space of the query instance feature based on the second source domain to reconstruct the first reconstruction query feature, mapping the first reconstruction support feature to the initial feature space of the support instance feature to reconstruct the second reconstruction support feature; determining the second reconstruction query feature as the target reconstruction query feature, and determining the second reconstruction support feature as the target reconstruction support feature.

[0068] In an exemplary embodiment, the detection module 52 can input the query image and multiple supporting images into the detection sub-network model included in the initial network model to obtain the query instance features of the query image and the support instance features of each of the support images in the following manner: input the query image and the support images into the feature extraction network included in the detection sub-network model to obtain query features and support features; input the query features into the extraction candidate box network included in the detection sub-network model to obtain query global features; input the support features into the extraction candidate box network to obtain support global features; input the query global features into the pooling layer included in the detection sub-network model to obtain query instance features; input the support features into the pooling layer to obtain support instance features.

[0069] In an exemplary embodiment, the detection module 52 can input the query image and the support image into the feature extraction network included in the detection sub-network model to obtain the query feature and the support feature in the following manner: input the query image and the support image into the residual structure included in the twin network model to obtain the initial query feature and the initial support feature, wherein the feature extraction network includes the twin network model; input the initial query feature into the feature pyramid structure included in the twin network model to obtain the query feature; input the initial support feature into the feature pyramid structure to obtain the query feature.

[0070] In an exemplary embodiment, the detection module 52 can determine the target loss value of the initial network model based on the reconstructed query feature and the reconstructed support feature in the following manner: determine a first mean square error loss between the query feature and the reconstructed query feature; determine a second mean square error loss between the support feature and the reconstructed support feature; determine the sum of the first mean square error loss and the second mean square error loss; and determine the target loss value based on the sum.

[0071] In an exemplary embodiment, the detection module 52 can determine the target loss value based on the sum value in the following manner: determine the classification loss value and the positioning loss value of the classifier and the locator included in the initial network model; determine the target loss value based on the classification loss value, the positioning loss value and the sum value.

[0072] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0073] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0074] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0075] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0076] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0077] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the steps of the method in each embodiment of the present application when the computer program is executed by a processor.

[0078] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0079] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting small sample objects, characterized in that: include: Inputting the small sample image into the target detection network to determine the target object, the target category of the target object and the target candidate box of the target object included in the small sample image; The target detection network is trained in the following way: Input a query image and a plurality of support images into a detection sub-network model included in an initial network model, and obtain a query instance feature of the query image and a support instance feature of each of the support images, wherein the query image is an unlabeled image, the support images are labeled images, the support images include support objects and labeling information, the labeling information includes the type of the support object and a candidate box of the support object, the query instance feature is a feature of an area where the query object included in the query image is located predicted by the detection sub-network model, and the support instance feature is a feature of an area where the candidate box is located determined by the detection sub-network model based on the labeling information; Inputting the query instance feature and the plurality of supporting instance features into the migration structure included in the initial network model for reconstruction, to obtain a target reconstruction query feature and a target reconstruction supporting feature; Determining a target loss value of the initial network model based on the reconstruction query feature and the reconstruction support feature; Determine, based on the target loss value, to iteratively update the network parameters of the initial network model to obtain an initial detection model; Using the acquired small sample annotated data to train the initial detection model to obtain the target detection network; Inputting the query instance feature and the plurality of supporting instance features into the migration structure included in the initial network model for reconstruction to obtain a target reconstructed query feature and a target reconstructed supporting feature comprises: reconstructing the query instance feature and the supporting instance feature based on a reconstruction module and a domain adversarial module included in the migration structure to obtain the target reconstructed query feature and the target reconstructed supporting feature; The reconstruction module is used to map the query instance feature and the supporting instance feature to the same feature space, reconstruct the mapped query instance feature and the supporting instance feature, map the reconstructed query instance feature back to the feature domain where the query instance feature is located, and map the reconstructed supporting instance feature back to the feature domain where the supporting instance feature is located; The domain discrimination module is used to maximize the domain difference between the query instance features and the support instance features.

2. The method according to claim 1, characterized in that Reconstructing the query instance feature and the support instance feature based on the reconstruction module and the domain adversarial module included in the migration structure includes: Inputting the query instance feature and the plurality of supporting instance features into the domain adversarial module, and determining a first source domain of any one target instance feature included in the query instance feature and the plurality of supporting instance features; Inputting the query instance feature, the plurality of supporting instance features, and the source domain into the reconstruction module, so as to map the query instance feature and the plurality of supporting instance features to the same feature space for reconstruction, and obtain a first reconstructed query feature and a first reconstructed supporting feature; Inputting the first reconstruction query feature and the first reconstruction support feature into the domain adversarial module, and determining a second source domain of any target reconstruction feature included in the first reconstruction query feature and the first reconstruction support feature; Based on the second source domain, the first reconstructed query feature is mapped to the initial feature space of the query instance feature for reconstruction to obtain a second reconstructed query feature, and the first reconstructed support feature is mapped to the initial feature space of the support instance feature for reconstruction to obtain a second reconstructed support feature; The second reconstruction query feature is determined as a target reconstruction query feature, and the second reconstruction support feature is determined as a target reconstruction support feature.

3. The method according to claim 1, characterized in that Inputting a query image and a plurality of supporting images into a detection sub-network model included in the initial network model, obtaining a query instance feature of the query image and a supporting instance feature of each of the supporting images, including: Inputting the query image and the support image into the feature extraction network included in the detection sub-network model to obtain query features and support features; Inputting the query feature into the candidate box extraction network included in the detection sub-network model to obtain the query global feature; Inputting the supporting features into the candidate box extraction network to obtain supporting global features; Inputting the query global feature into the pooling layer included in the detection subnetwork model to obtain the query instance feature; The supporting features are input into the pooling layer to obtain supporting instance features.

4. The method according to claim 3, characterized in that The query image and the support image are input into the feature extraction network included in the detection sub-network model to obtain the query features, and the support features include: Inputting the query image and the support image into the residual structure included in the twin network model to obtain initial query features and initial support features, wherein the feature extraction network includes the twin network model; Inputting the initial query feature into the feature pyramid structure included in the twin network model to obtain the query feature; The initial support features are input into the feature pyramid structure to obtain the query features.

5. The method according to claim 1, characterized in that: Determining the target loss value of the initial network model based on the reconstruction query feature and the reconstruction support feature includes: determining a first mean square error loss between the query feature and the reconstructed query feature; determining a second mean square error loss between the support feature and the reconstructed support feature; Determine a sum of the first mean square error loss and the second mean square error loss; The target loss value is determined based on the sum value.

6. The method according to claim 5, characterized in that Determining the target loss value based on the sum value includes: Determine the classification loss value and the positioning loss value of the classifier and the locator included in the initial network model; The target loss value is determined based on the classification loss value, the positioning loss value, and the sum value.

7. A detection device for small sample objects, characterized in that: include: A detection module, used to input a small sample image into a target detection network to determine a target object, a target category of the target object, and a target candidate box of the target object included in the small sample image; The target detection network is trained in the following way: Input a query image and a plurality of support images into a detection sub-network model included in an initial network model, and obtain a query instance feature of the query image and a support instance feature of each of the support images, wherein the query image is an unlabeled image, the support images are labeled images, the support images include support objects and labeling information, the labeling information includes the type of the support object and a candidate box of the support object, the query instance feature is a feature of an area where the query object included in the query image is located predicted by the detection sub-network model, and the support instance feature is a feature of an area where the candidate box is located determined by the detection sub-network model based on the labeling information; Inputting the query instance feature and the plurality of supporting instance features into the migration structure included in the initial network model for reconstruction, to obtain a target reconstruction query feature and a target reconstruction supporting feature; Determining a target loss value of the initial network model based on the reconstruction query feature and the reconstruction support feature; Determine, based on the target loss value, to iteratively update the network parameters of the initial network model to obtain an initial detection model; Using the acquired small sample annotated data to train the initial detection model to obtain the target detection network; The detection module implements the following method to input the query instance feature and the plurality of supporting instance features into the migration structure included in the initial network model for reconstruction, thereby obtaining a target reconstruction query feature and a target reconstruction supporting feature: based on the reconstruction module and the domain adversarial module included in the migration structure, the query instance feature and the supporting instance feature are reconstructed to obtain the target reconstruction query feature and the target reconstruction supporting feature; The reconstruction module is used to map the query instance feature and the supporting instance feature to the same feature space, reconstruct the mapped query instance feature and the supporting instance feature, map the reconstructed query instance feature back to the feature domain where the query instance feature is located, and map the reconstructed supporting instance feature back to the feature domain where the supporting instance feature is located; The domain discrimination module is used to maximize the domain difference between the query instance features and the support instance features.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 6 when executed.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.