Cross-domain small sample target detection method based on generative 3D geometric prior enhancement

Abstract

The invention particularly relates to a cross-domain small sample target detection method based on generative 3D geometric prior enhancement, object instances under different visual angles are synthesized based on an object rendering engine expanded by Gaussian distribution, and meanwhile, a fidelity perception object selection strategy is provided to filter and generate objects so as to retain a high-fidelity synthesized image. Besides, the cross-view semantic consistency loss is introduced in the method, so that the semantic similarity between the prototype and the synthetic samples with the same object but different views is enhanced, and the cross-view invariant capability of the prototype is improved. Through the data augmentation and prototype enhancement strategy, the capability of generalizing the model to the target field only by using a small number of labeled samples in the cross-domain field can be further improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a cross-domain few-shot object detection method based on generative 3D geometric prior enhancement. BACKGROUND

[0002] Object detection is the core foundation in the technical field of image processing. In recent years, a series of deep learning models have made important progress in the task of natural scene image object detection. However, these models often rely on large-scale labeled data for training to exert their performance, such as training on the COCO dataset with a large amount of labeled data in natural scene data. In the extreme task field of underwater images, comic style images, and the like, it is not only costly to obtain high-quality image data, but also the professional annotation resources are scarce, making it very difficult to train a high-performance detector. Therefore, it is of great application value to effectively adapt the model trained in natural scene images to the extreme cross-domain field with scarce labeled data. Under this background, cross-domain few-shot object detection using only a small amount of labeled samples in the cross-domain field is an effective solution.

[0003] The document "Fu Y, Wang Y, Pan Y, et al. Cross-domain few-shot object detection via enhanced open-set object detector [C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 247-264." proposes a small sample image cross-domain detection method based on fine-tuning. The object detection model used by the method is the CD-Vito model (Cross-Domain Few-Shot Object Detection via Enhanced Open-Set Object Detector). This method uses prototype relearning and weighting to solve the problems of style, inter-class difference and boundary caused by domain difference, and proposes a domain simulator to simulate domain disturbance to further improve the cross-domain ability of the detector. However, the obvious cross-domain difference and the scarcity of labeled samples will destroy the robustness and discriminability of the prototype, resulting in a decrease in the generalization performance of the model for the cross-domain task.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The application provides a cross-domain small sample target detection method based on generative 3D geometric prior enhancement, which is used for solving the problem that the obvious cross-domain difference and the lack of labeled samples in the prior art will destroy the robustness and discriminability of the prototype, resulting in the reduction of the generalization performance of the model for the cross-domain task.

[0006] Other characteristics and advantages of the present application will become apparent from the following detailed description, or will be learned by practice of the present application.

[0007] According to a first aspect of the present application, a cross-domain small sample target detection method based on generative 3D geometric prior enhancement is provided, and the method comprises: obtaining training image data of a cross-domain target field; performing Gaussian rendering on the training image data to obtain synthetic samples; calculating the cosine similarity of the synthetic samples and the original support instances in the CLIP feature space, screening high-fidelity synthetic samples as enhancement samples according to the cosine similarity score, and constructing an enhancement sample set; feeding the original support instances and their corresponding enhancement samples into the backbone network of the target detection model to extract features, performing pooling operations on all features of the same category, and obtaining an optimized prototype; randomly sampling two samples of different perspectives from the enhancement sample set, feeding them into the backbone network to extract features, and calculating the category attribution probability distribution based on the features and the prototype; calculating the KL divergence based on the category attribution probability distribution of the original support instance and its different perspective enhancement samples, calculating the cross-perspective consistency loss based on the KL divergence, constructing a total loss function based on the cross-perspective consistency loss, the positioning loss and the classification loss, and updating the parameters of the target detection model through back propagation; In the inference stage, the image to be detected is input into the trained target detection model, and after the features of the image are extracted by the backbone network, the final detection result can be obtained through the detection and classification modules of the target detection model.

[0008] In some example embodiments, the Gaussian rendering on the training image data to obtain synthetic samples is specifically: segmenting the training image data through a SAM model to obtain a foreground object; randomly initializing a set of 3D Gaussian distributions, wherein each Gaussian distribution is represented as a parameter set containing a center position, a scaling factor, a rotation quaternion, an opacity and a color value; rendering the 3D Gaussian distribution into 2D images of different perspectives using a differentiable renderer; calculating the SDS loss based on the 2D images and the foreground object and updating the parameters of the 3D Gaussian distribution through back propagation; obtaining synthetic samples based on the 3D Gaussian distribution after parameter optimization.

[0009] In some example embodiments, the high-fidelity synthetic samples are screened as augmented samples according to the cosine similarity score, and an augmented sample set is constructed, specifically: All synthetic samples are sorted according to the similarity score, and a plurality of synthetic samples with high ranking are selected as augmented samples, and the augmented samples are put back into the original support instance background to construct an augmented sample set.

[0010] In some example embodiments, the similarity score calculation method is:

[0011] wherein, represents the CLIP feature of the original support instance, represents the feature of the i-th synthetic sample.

[0012] In some example embodiments, the prototype is set as a learnable parameter in the training process of the target detection model to further optimize through loss function gradient update.

[0013] In some example embodiments, the KL divergence is calculated based on the class attribution probability distribution of the original support instance and its different view augmented samples, and the cross-view consistency loss is calculated based on the KL divergence, specifically: For the original support instance, two different view samples are sampled from the augmented sample set thereof, and the predicted probability distribution is calculated respectively; The predicted probability distribution of the original support instance is taken as a pseudo label, the predicted probability distribution of the sample under different views is aligned with the pseudo label, and the KL divergence loss is calculated: The total cross-view consistency loss is obtained by summing all support samples. In some example embodiments, the target detection model is CD-ViTo.

[0014] According to a second aspect of the present application, a storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the cross-domain small sample target detection method based on generative 3D geometric prior enhancement according to the first aspect.

[0015] According to a third aspect of the present application, a computer program product is provided, which stores a computer program, and the computer program is executed by a processor to implement the cross-domain small sample target detection method based on generative 3D geometric prior enhancement according to the first aspect.

[0016] According to a fourth aspect of the present application, an electronic device is provided, comprising: a processor; and​ a memory for storing executable instructions of the processor; The processor is configured to implement the cross-domain small sample target detection method based on generative 3D geometric prior enhancement according to the executable instructions.

[0017] The cross-domain small sample target detection method based on generative 3D geometric prior enhancement provided by the embodiment of the present application can effectively adapt the visual base model trained in natural images to the small sample task in the extreme field by designing a target detection framework based on generative 3D geometric prior enhancement, and can be seamlessly integrated with the existing matching-based detector, thereby alleviating the realistic problem that the existing fine-tuning technology has weak generalization ability due to large distribution difference and less labeled data. The method of the present application has achieved good results in the small sample image detection task in the extreme field.

[0018] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated into and form part of the specification, illustrate one embodiment consistent with the present application and, together with the description, serve to explain the principles of the application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained from these drawings without creative labor for those skilled in the art.

[0020] Figure 1 The method framework diagram of the present application; Figure 2 The data engine flowchart proposed by the present application. DETAILED DESCRIPTION

[0021] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations may, however, be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the gist of the example implementations to those skilled in the art. The features, structures, or characteristics described can be combined in any suitable manner in one or more implementations.

[0022] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0023] To address the shortcomings and deficiencies of existing technologies, this exemplary embodiment provides a cross-domain few-shot object detection method based on generative 3D geometric prior enhancement. This method synthesizes object instances from different viewpoints using a Gaussian distribution-expanded object rendering engine, and proposes a fidelity-aware object selection strategy to filter generated objects, preserving high-fidelity synthesized images. Furthermore, this method introduces cross-viewpoint semantic consistency loss to enhance the semantic similarity between the prototype and synthesized samples with the same object but different viewpoints, improving the prototype's cross-viewpoint invariance. Through this data augmentation and prototype enhancement strategy, this invention can further improve the model's ability to generalize to the target domain using only a small number of labeled samples from across domains.

[0024] refer to Figure 1 As shown, the specific steps may include: Step 1: Gaussian Rendering. Given training image data from a cross-domain target area, the foreground object is first segmented using the SAM model. Then, a set of representations is randomly initialized. The 3D Gaussian distribution, in which The initial number of 3D Gaussians. Each Gaussian distribution... Represented as ,in Indicates the center position. Indicates the scaling factor. Represents a rotation quaternion. Indicates opacity. Indicates the color value.

[0025] To optimize the parameters of the 3D Gaussian distribution, a camera angle parameter will be randomly sampled. To simulate objects Small-scale movements, i.e., simulating different perspectives or slight changes. At a camera angle of... Under the given conditions, use a differentiable renderer to render a 3D Gaussian distribution into a 2D image. When rendering an image, use an instance image. Using a pre-trained diffusion model as a reference image, image denoising is performed, thereby improving... Detailed information. The final 2D image. Match the reference image The appearance, while presenting a slight change in perspective. Finally, using Calculate the SDS loss and backpropagate to update the parameters of the 3D Gaussian distribution. The gradient of the SDS loss function is:

[0026] in, Represents the parameters relative to the three-dimensional Gaussian model. gradient, Indicates the diffusion time step. This is a time-step weighting function used to adjust the contribution level of different time steps; This represents a pre-trained two-dimensional diffusion model. The predicted noise, These are real noise samples; This represents the gradient of the rendered image relative to the Gaussian distribution parameters.

[0027] To further improve the texture fidelity of images, this invention will render the image... Compared with reference image Alignment is achieved through image space reconstruction loss, thereby enhancing the consistency of appearance details such as color and texture. Ultimately, this improves the texture details of multi-view synthesized images of the 3D model from different camera angles.

[0028] Step 2: Sample Selection. To improve the quality and representativeness of the synthesized images, this invention proposes an automatic sample selection module. This module automatically filters the large number of synthesized samples generated by the object rendering engine based on Gaussian distribution expansion in Step 1, thereby avoiding the negative impact of low-quality samples that differ significantly from real samples in semantic expression or visual performance on the detector performance. This module calculates the synthesized samples... With original support instance Cosine similarity in the CLIP feature space measures the consistency of their semantic representation. The CLIP feature representation of the original supporting instances is as follows: , No. The features of each synthetic sample are represented as follows: The similarity score is calculated as follows:

[0029] All candidate samples are ranked according to similarity scores, and the top-ranked samples are selected to form the augmented dataset, ensuring that the samples constituting the augmented dataset have strong representativeness and informativeness. Finally, the selected synthetic samples are placed back into the background of the original supporting examples to construct the augmented training sample set.

[0030] Step 3: Prototype Optimization. This invention further proposes a prototype construction method based on augmented samples to improve the representation and discriminative capabilities of the prototype. The original supporting instances and their corresponding augmented samples are fed into the backbone network to extract instance features. Among them, the class... The corresponding number The original support examples are denoted as The corresponding number Each augmented sample is represented as The corresponding feature vector is represented as and .

[0031] The original support instances and augmented samples together constitute the support samples. Pooling is performed on the feature vectors of all support samples belonging to the same category to obtain the optimized prototype representation for that category. The set of prototypes for all categories is represented as follows: ,in Indicates the number of categories. Indicates the first The prototype representation of the class. The prototype is set as a learnable parameter during training, allowing it to be further optimized through gradient updates of the loss function of the object detection model.

[0032] Step 4: Cross-view semantic consistency. This invention further proposes a cross-view semantic consistency loss method to align features between generated samples from different viewpoints. Specifically, the first step... The class of Each supporting sample is denoted as From its corresponding augmented sample set Two augmented samples from different viewpoints were randomly sampled and denoted as follows: and Subsequently, the enhanced samples are fed into the backbone network to extract features, and the resulting feature vectors are represented as follows: and .

[0033] Based on the category prototype set constructed in step 3 Calculate the probability of a sample being classified into a specific category. This includes sample features. Category The probability is denoted as:

[0034] in This represents the cosine similarity measure between vectors. It incorporates sample features. The predicted probability distribution across all categories is denoted as Sample characteristics The corresponding predicted distribution is After obtaining the predicted probability distributions of two sets of samples from different perspectives, Kullback-Leibler divergence (KL divergence) is used to constrain the distribution similarity between samples from different perspectives, thereby achieving cross-perspective consistency. To ensure stable convergence of the model during training, this invention proposes using the original support instances... The predicted probability distribution is used as a pseudo-label, and the predicted distribution of samples from different perspectives is aligned with the pseudo-label. For samples The cross-perspective consistency loss is defined as:

[0035] in The KL divergence is represented as follows:

[0036]

[0037] in , and These represent the predicted probability distributions of the original support examples and their augmented viewpoint samples across all categories, respectively. Finally, for the entire set of support samples... Calculate cross-perspective semantic consistency loss:

[0038] During the network training phase, the image to be detected is first... Its corresponding augmented sample and The features are extracted from the input samples and fed into the backbone network. The feature vectors extracted from the augmented samples are then processed. and Further calculate the cross-perspective consistency loss. This is done to improve the consistency of feature representations from different perspectives. The image to be detected is then... eigenvectors A feature subspace is constructed together with predefined background features, and this feature subspace is input into a Region Proposal Network (RPN) to generate candidate boxes. Then, the feature subspace is input into a classification module to classify the candidate boxes. Simultaneously, the candidate boxes and feature subspace are input into a regression module to regress the candidate boxes, obtaining the final predicted boxes.

[0039] Compare the predicted bounding boxes with the ground truth bounding boxes. The positioning loss between them is marked as The classification loss is labeled as Finally, the total loss function of the network is defined as follows:

[0040] in, Loss due to domain simulator.

[0041] The gradient is generated based on the aforementioned loss function, and backpropagation is used to update the network parameters, thereby improving the image detection capability of the assigned samples in small-sample and extreme cross-domain scenarios. During the inference phase, only the image to be detected needs to be input. After the features are extracted by the backbone network, the final detection result can be obtained through the detection and classification module.

[0042] The target detection model used in this invention is CD-ViTo. First, class prototypes are obtained through a support image set. ,in Indicates the number of categories. Indicates the first The prototype representation of a class. For a given query image Input it into the region suggestion network to obtain a set of candidate boxes. ,in This represents the i-th candidate box. The query image is fed into the backbone network to obtain image features. Next, this set of candidate bounding boxes B and image features... The image features corresponding to the candidate boxes are obtained by feeding them into the region proposal alignment network. Image features Candidate boxes and class prototype The image features are then fed into the regression (localization) module for coordinate regression. Candidate box corresponds to image features The class prototype F is fed into the classification module for classification. In addition, CD-ViTo introduces a domain simulator to improve the detector's cross-domain detection capability by simulating perturbations in different domains.

[0043] This invention achieves superior classification performance in small-sample image target detection tasks in extreme domains. For example, based on average test results on the natural insect dataset ArTaxOr, the cartoon-style dataset Clipart1k, the underwater fish recognition dataset DeepFish, and the underwater target dataset UODD, the method of this invention achieves an average accuracy of 17.5% with one training sample per class, 31.9% with five training samples per class, and 36.0% with ten training samples per class, representing improvements of 2.0%, 2.4%, and 2.5% compared to baseline models.

[0044] It should be noted that, as another aspect, this application also provides a storage medium, which may be included in an electronic device or may exist independently without being assembled into the electronic device. The storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to perform the methods described in the following embodiments.

[0045] In one embodiment, this application provides a computer program product including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0046] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0047] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0048] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is defined only by the appended claims.

Claims

1. A cross-domain small-sample target detection method based on generative 3D geometric prior enhancement, characterized in that, The method includes: Acquire training image data across target domains; Gaussian rendering is applied to the training image data to obtain synthetic samples; Calculate the cosine similarity between the synthetic sample and the original supporting instance in the CLIP feature space, select high-fidelity synthetic samples as augmentation samples based on the cosine similarity score, and construct an augmentation sample set; The original support instances and their corresponding augmented samples are fed into the backbone network of the object detection model to extract features. Pooling is then performed on all features of the same category to obtain the optimized prototype. Samples from two different perspectives are randomly sampled from the augmented sample set and fed into the backbone network to extract features. Based on the features and prototypes, the probability distribution of category assignment is calculated. The KL divergence is calculated based on the class assignment probability distribution of the original support examples and the augmented samples from different perspectives. The cross-perspective consistency loss is then calculated based on the KL divergence. The total loss function is constructed based on the cross-perspective consistency loss, localization loss, and classification loss. The parameters of the object detection model are then updated through backpropagation. During the inference phase, the image to be detected is input into the trained target detection model. After the backbone network extracts its features, the final detection result can be obtained through the detection and classification modules of the target detection model.

2. The method according to claim 1, characterized in that, The process of performing Gaussian rendering on the training image data to obtain synthetic samples specifically involves: The training image data is segmented using the SAM model to obtain the foreground object; Randomly initialize a set of 3D Gaussian distributions, where each Gaussian distribution is represented as a set of parameters including center position, scaling factor, rotation quaternion, opacity, and color value; Use a differentiable renderer to render a 3D Gaussian distribution as a 2D image from different perspectives; The SDS loss is calculated based on the 2D image and foreground object, and the parameters of the 3D Gaussian distribution are updated by backpropagation. Synthetic samples are obtained by rendering based on a 3D Gaussian distribution with optimized parameters.

3. The method according to claim 1, characterized in that, The process of selecting high-fidelity synthetic samples as enhancement samples based on cosine similarity scores and constructing an enhancement sample set is as follows: All synthetic samples are sorted according to similarity scores, and the top-ranked synthetic samples are selected as augmented samples. The augmented samples are then placed back into the context of the original supporting examples to construct an augmented sample set.

4. The method according to claim 3, characterized in that, The similarity score is calculated as follows: in, This is represented by the CLIP feature of the original supporting instance. Represented as the first Features of a synthetic sample.

5. The method according to claim 1, characterized in that, The prototype is set as a learnable parameter during the training of the object detection model, so as to be further optimized by updating the loss function gradient.

6. The method according to claim 1, characterized in that, The KL divergence is calculated based on the class assignment probability distribution of the original support instances and their augmented samples from different perspectives. The cross-perspective consistency loss is then calculated based on the KL divergence. Specifically: For the original support instance, sample two different viewpoint samples from its augmented sample set and calculate the predicted probability distribution for each; Using the predicted probability distribution of the original supporting instances as pseudo-labels, the predicted probability distributions of samples from different perspectives are aligned with the pseudo-labels, and the KL divergence loss is calculated: The total cross-perspective consistency loss is obtained by summing over all supporting samples.

7. The method according to claim 1, characterized in that, The target detection model is CD-ViTo.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the cross-domain small sample target detection method based on generative 3D geometric prior enhancement as described in any one of claims 1 to 7.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the cross-domain small sample target detection method based on generative 3D geometric prior enhancement as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the cross-domain few-shot target detection method based on generative 3D geometric prior enhancement as described in any one of claims 1 to 7 by executing the executable instructions.

Images