A few-shot object-level visual perception method based on synthetic data cross-domain migration

Abstract

The application discloses a few-shot object-level visual perception method based on synthetic data cross-domain migration. Through high-precision three-dimensional reconstruction and PBR material modeling, the synthetic image keeps consistent reflection characteristics with the real object under different illumination and viewing angles, and provides a more reliable source data basis for subsequent domain randomization and style migration. Experiments show that after PBR material modeling, the peak signal-to-noise ratio index in the reconstruction task is increased by more than 15%, and the quality of the synthetic data is significantly improved. The application uses a generative three-dimensional model construction method to automatically generate a large number of interference objects with different morphologies, greatly enriching the complexity of the simulation scene without manual modeling. After introducing the generative interference object, the recognition accuracy of the model for unknown objects is increased by 8.2%, effectively enhancing the generalization ability of the model. The average precision index on the LM-O data set is increased by 3.4%, and the AP index on the YCB-V data set is increased by 3.6%.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and robot perception technology, and more specifically, to a few-sample object-level visual perception method based on cross-domain transfer of synthetic data. Background Technology

[0002] In industrial assembly, robotic operation, and intelligent manufacturing, object-level visual perception models are fundamental to achieving intelligent automation. These models require accurate detection, segmentation, and pose estimation of objects within the scene, and their performance directly impacts the success of subsequent tasks. Existing high-performance models typically rely on large amounts of high-precision labeled data, such as segmentation masks, 3D bounding boxes, and six-DOF poses. However, acquiring this data in real-world scenarios is costly, time-consuming, and suffers from a long-tail distribution problem, severely limiting the model's generalization ability in situations with rare objects or complex scenes.

[0003] To reduce reliance on real-world labeled data, researchers have attempted to train models using synthetic data. Existing methods mainly fall into the following categories:

[0004] (1) Synthetic Data Generation Based on 3D Modeling and Rendering: This method constructs object models through 3D reconstruction or manual modeling, and then renders annotated images in a simulation environment. To enhance the realism of the rendering, researchers introduced physically based rendering (PBR) technology to accurately model the material parameters of the object surface (such as reflectivity, roughness, and metallicity). At the same time, domain randomization technology enhances the diversity of data by randomly perturbing parameters such as lighting, viewpoint, and background. However, existing methods have the following limitations: most methods rely on manual modeling or simple 3D scanning, making it difficult to simultaneously guarantee geometric accuracy and material realism; the generation of interfering objects often relies on manual construction, which limits the complexity and diversity of the simulation scene.

[0005] (2) Style transfer-based image enhancement: To reduce the visual difference between synthetic and real images, researchers have proposed using generative adversarial networks or diffusion models to perform style transfer on synthetic images, making their visual style closer to that of real images. These methods effectively improve the realism of synthetic images, but they have the following problems: the style transfer process can easily change the physical geometric features of objects, affecting the accuracy of subsequent perception tasks; these methods are still limited to style alignment at the image level and do not solve the problem of domain offset of the model in the feature space.

[0006] In summary, the existing technology mainly has the following technical problems:

[0007] First, the synthetic data lacks realism in its physical properties. Existing methods have limited accuracy in modeling the material properties of objects (such as reflectivity, roughness, and metallicity) during 3D model construction, making it difficult to realistically reproduce the optical properties of objects under different lighting conditions in a simulation environment. This results in significant differences in the distribution of physical parameters between synthetic and real data.

[0008] Second, the diversity of simulation scenarios is limited. Existing methods largely rely on manual modeling to generate interference objects, making it difficult to generate a large number of interference objects with different geometric structures and appearances in a low-cost and efficient manner. This limits the complexity of the simulation scenarios and the model's ability to generalize to unknown objects.

[0009] Third, there is a significant difference in visual style between synthetic and real data. While existing style transfer methods can improve the visual realism of synthetic images, they tend to alter the geometric features of objects and remain limited to appearance alignment at the image level, failing to address the issue of inconsistent distribution between the synthetic and real domains in the deep feature space.

[0010] Fourth, a small number of real samples are insufficient to fully represent the distribution of the target domain. In practical applications, the number of available real labeled samples is extremely limited, making it difficult to accurately estimate the complete data distribution of the target domain with a small number of samples, thus limiting the model's generalization performance in real-world scenarios.

[0011] Fifth, feature space domain shift leads to a decline in generalization performance. Even when high-fidelity synthetic images are generated, the distribution difference between the synthetic and real domains in the feature space still exists. Models trained directly using synthetic data will show a significant performance degradation in real-world scenes, and existing methods lack effective feature-level domain alignment strategies.

[0012] Therefore, how to effectively align the synthetic domain with the real domain at multiple levels, such as physical properties, visual appearance, and deep features, by working together from two dimensions—data generation and model training—guided by a small number of real samples, is a key technical challenge that urgently needs to be addressed in current research. Summary of the Invention

[0013] This invention addresses the shortcomings of existing technologies by providing a few-sample object-level visual perception method based on cross-domain transfer of synthetic data. This invention is achieved through the following technical solution:

[0014] This invention discloses a few-shot object-level visual perception method based on cross-domain transfer of synthetic data, comprising:

[0015] Step 1: Obtain multi-view images of the target object and perform 3D reconstruction to obtain a 3D model of the target object;

[0016] Step 2: Generate 3D models of multiple interfering objects based on the 3D model;

[0017] Step 3: Randomly place the 3D models of the target object and the interference object in the simulation environment, randomly set the imaging parameters, render and generate source synthetic data, and construct the source domain from the source synthetic data;

[0018] Step 4: Obtain a small number of real-world scene images as anchor images, and then annotate them to obtain the foreground and background of the anchor images.

[0019] Step 5: Using a two-stage domain transfer network, a high-fidelity synthetic foreground is generated with the foreground of the source synthetic data as content and the foreground of the anchor image as style. Using an image inpainting model, the background of the anchor image is expanded and fused with the high-fidelity synthetic foreground to obtain a high-fidelity synthetic image.

[0020] Step 6: Obtain a large amount of unlabeled physical data, and construct an extended target domain by combining high-fidelity synthetic images, a small number of anchor point images, and a large amount of unlabeled physical data. Based on the source domain and the extended target domain, jointly train the visual perception model.

[0021] As a further improvement, step 1 of the present invention involves obtaining multi-view images of the target object and performing 3D reconstruction to obtain a 3D model of the target object, specifically including:

[0022] A neural implicit surface reconstruction method is adopted to optimize the symbolic distance function field based on the acquired multi-view images in order to reconstruct the three-dimensional geometric mesh of the target object;

[0023] The reconstructed 3D geometric mesh of the target object is imported into the differentiable rendering pipeline. By jointly optimizing the ambient lighting and the physically based rendering materials of the 3D model, a 3D model of the target object with realistic reflection properties is obtained. The physically based rendering materials of the 3D model include reflectivity, roughness, and metallicity.

[0024] As a further improvement, step 2 of this invention, which generates 3D models of multiple interfering objects based on a 3D model generation model, specifically includes:

[0025] The latent encoding of random sampling is input into the 3D model to generate a model, generating a signed distance function field and a texture field;

[0026] The mesh is extracted from the symbolic distance function field using a surface extraction algorithm, and the corresponding texture is assigned to the texture field to obtain a 3D model of the interfering object.

[0027] As a further improvement, step 5 of this invention utilizes a two-stage domain transfer network to generate a high-fidelity synthetic foreground using the foreground of the source synthetic data as content and the foreground of the anchor image as style. Specifically, this includes:

[0028] In the first stage, the foreground of the source synthetic data is used as the content image, and the anchor point real foreground is used as the style image. After wavelet transform of the image features, whitening and color transformation are performed on both low-frequency and high-frequency components to realize the transfer of the camera imaging artistic style and obtain a coarsely aligned image.

[0029] In the second stage, the coarsely aligned image output from the first stage is used as the content image, and the real foreground at the same anchor point is used as the style image. After wavelet transform of the image features, only the low-frequency components are whitened and tinted, while the high-frequency components directly inherit the high-frequency features from the first stage, realizing the transfer of large-scale texture style and finally outputting a high-fidelity synthesized foreground.

[0030] As a further improvement, step 5 of this invention utilizes an image restoration model to expand the background of the anchor point image, specifically including:

[0031] The anchor point image background is randomly cropped, rotated, and affine transformed in sequence, and finally input into the image restoration model to complete the image, resulting in the expanded anchor point image background.

[0032] As a further improvement, step 6 of this invention, which involves jointly training a visual perception model based on the source domain and the extended target domain, specifically includes:

[0033] Based on the source domain and extended target domain, a visual perception model is jointly trained through a student-teacher collaborative learning framework. During the training process, the category prototypes of the source domain and the target domain are calculated and aligned to minimize the domain differences in the feature space. At the same time, through contrastive learning, the class features across data domains can be separated.

[0034] As a further improvement, the student-teacher collaborative learning framework of the present invention specifically includes:

[0035] The teacher model is an exponential moving average copy of the student model, and is updated by the student model during training;

[0036] For unlabeled data in the expanded target domain, pseudo-labels are generated by the teacher model;

[0037] The student model receives both source domain data and extended target domain data simultaneously. Based on the data label type, it is jointly trained by combining the supervised loss and the pseudo-supervised loss corresponding to the pseudo-label.

[0038] As a further improvement, the calculation and alignment of the category prototypes of the source and target domains in this invention to minimize domain differences in the feature space specifically includes:

[0039] For each object category, calculate the source domain prototype and the target domain prototype separately. The prototype is the statistical average of the foreground features of all samples in that category.

[0040] Construct a domain discriminator that takes the source domain prototype and the target domain prototype as input and outputs the domain category.

[0041] Based on the domain category, adversarial training optimizes the domain discriminator to accurately distinguish between the source and target domains, while also optimizing the feature extractor of the visual perception model to make the extracted source and target domain features difficult for the domain discriminator to differentiate, thereby minimizing domain differences.

[0042] As a further improvement, the present invention achieves cross-data domain separability of category features through contrastive learning, specifically including:

[0043] Calculate the cross-domain prototype contrast loss, which is used to narrow the feature distance between the source domain prototype and the target domain prototype within the same category, and to widen the feature distance between prototypes of different categories.

[0044] Cross-domain prototype comparison loss is implemented using the InfoNCE loss function, the formula of which is as follows:

[0045] ;

[0046] in, For object categories, the source domain prototypes and target domain prototypes aggregated during training are respectively and The features extracted in each batch are Harmony , Temperature parameters for controlling the sharpness of the distribution, This is the function for calculating cosine similarity.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. This invention addresses the problem of insufficient physical parameter distribution coverage in traditional synthetic data: By employing high-precision 3D reconstruction and PBR material modeling, the synthesized image maintains reflectivity consistent with real objects under different lighting and viewing angles, providing a more reliable source data foundation for subsequent domain randomization and style transfer. Experiments show that using PBR material modeling improves the peak signal-to-noise ratio (PSNR) by more than 15% in reconstruction tasks, significantly enhancing the quality of the synthetic data.

[0049] 2. This invention addresses the limitation of simulation scene diversity: Utilizing a generative 3D model construction method, it automatically generates a large number of diverse interference objects, greatly enriching the complexity of the simulation scene without requiring manual modeling. Experiments show that introducing generative interference objects improves the model's accuracy in recognizing unknown objects by 8.2%, effectively enhancing the model's generalization ability.

[0050] 3. This invention addresses the significant visual style difference between synthetic and real data: It proposes a high-fidelity synthetic data generation method based on a two-stage domain transfer network, which significantly reduces the visual distribution difference between synthetic and real data while preserving the geometric structure of objects. Experiments show that training segmentation tasks with the transferred high-fidelity synthetic data improves the average precision (AP) by 3.4% on the LM-O dataset and by 3.6% on the YCB-V dataset.

[0051] 4. This invention addresses the problem of insufficient representation of the target domain distribution by a small number of real samples: By explicitly incorporating high-fidelity synthetic images into the target domain, a small number of real samples (e.g., 48 images) are expanded into an extended target domain containing thousands of high-quality samples, resulting in more accurate target domain distribution estimation. Experiments show that on the YCB-V dataset, using only 48 real images combined with 3000 high-fidelity synthetic images achieves model performance comparable to using 113,000 real images (AP 76.0% vs 76.3%), significantly reducing data annotation costs.

[0052] 5. This invention addresses the degradation in generalization performance caused by feature space domain shift: By expanding the target domain, this invention utilizes a student-teacher collaborative learning framework, combined with cross-domain prototype alignment and contrastive learning, to achieve effective alignment between the synthetic and real domains at the feature level. Through this combined effect, the model achieves leading performance on multiple public datasets. Furthermore, in the BOP (Benchmark for 6D Object Pose Estimation) public challenge, this method ranked first on the public leaderboard for segmentation tasks on both the LM-O and YCB-V datasets, achieving a leap from "visual similarity" to "feature usability."

[0053] 6. Achieved high-performance object-level visual perception under limited sample conditions: This invention innovates collaboratively from two dimensions: "data generation" and "model training". Through a full-chain domain alignment strategy, it can train an object detection and segmentation model with excellent generalization performance in real-world scenarios with only a small number of real labeled samples, providing an efficient and low-cost solution for data-scarce scenarios such as industrial assembly and robot operation. Attached Figure Description

[0054] Figure 1 This is an overall flowchart of the method of the present invention. Detailed Implementation

[0055] This invention proposes a few-shot object-level visual perception method based on cross-domain transfer of synthetic data, which mainly includes the following steps. A few-shot object-level visual perception method based on cross-domain transfer of synthetic data includes:

[0056] Step 1: Obtain multi-view images of the target object and perform 3D reconstruction to obtain a 3D model of the target object; specifically, this includes:

[0057] A neural implicit surface reconstruction method is adopted to optimize the symbolic distance function field based on the acquired multi-view images in order to reconstruct the three-dimensional geometric mesh of the target object;

[0058] The reconstructed 3D geometric mesh of the target object is imported into the differentiable rendering pipeline. By jointly optimizing the ambient lighting and the physically based rendering materials of the 3D model, a 3D model of the target object with realistic reflection properties is obtained. The physically based rendering materials of the 3D model include reflectivity, roughness, and metallicity.

[0059] Step 2: Generate 3D models of multiple interfering objects based on the 3D model; specifically including:

[0060] The latent encoding of random sampling is input into the 3D model to generate a model, generating a signed distance function field and a texture field;

[0061] The mesh is extracted from the symbolic distance function field using a surface extraction algorithm, and the corresponding texture is assigned to the texture field to obtain a 3D model of the interfering object.

[0062] Step 3: Randomly place the 3D models of the target object and the interference object in the simulation environment, randomly set the imaging parameters, render and generate source synthetic data, and construct the source domain from the source synthetic data;

[0063] Step 4: Obtain a small number of real-world scene images as anchor images, and then annotate them to obtain the foreground and background of the anchor images.

[0064] Step 5: Using a two-stage domain transfer network, a high-fidelity synthetic foreground is generated with the foreground of the source synthetic data as content and the foreground of the anchor image as style. An image inpainting model is then used to expand the background of the anchor image and fuse it with the high-fidelity synthetic foreground to obtain a high-fidelity synthetic image. Specifically, generating the high-fidelity synthetic foreground using a two-stage domain transfer network with the foreground of the source synthetic data as content and the foreground of the anchor image as style includes:

[0065] In the first stage, the foreground of the source synthetic data is used as the content image, and the anchor point real foreground is used as the style image. After wavelet transform of the image features, whitening and color transformation are performed on both low-frequency and high-frequency components to realize the transfer of the camera imaging artistic style and obtain a coarsely aligned image.

[0066] In the second stage, the coarsely aligned image output from the first stage is used as the content image, and the real foreground at the same anchor point is used as the style image. After wavelet transform of the image features, only the low-frequency components are whitened and tinted, while the high-frequency components directly inherit the high-frequency features from the first stage, realizing the transfer of large-scale texture style and finally outputting a high-fidelity synthesized foreground.

[0067] Using an image inpainting model, the background of the anchor point image is expanded, specifically including:

[0068] The anchor point image background is randomly cropped, rotated, and affine transformed in sequence, and finally input into the image restoration model to complete the image, resulting in the expanded anchor point image background.

[0069] Step 6: Obtain a large amount of unlabeled object data. Construct an extended target domain using high-fidelity synthetic images, a small number of anchor point images, and the large amount of unlabeled object data. Jointly train the visual perception model based on the source domain and the extended target domain. Specifically, this joint training of the visual perception model based on the source domain and the extended target domain includes:

[0070] Based on the source domain and extended target domain, a visual perception model is jointly trained through a student-teacher collaborative learning framework. During the training process, the category prototypes of the source domain and the target domain are calculated and aligned to minimize the domain differences in the feature space. At the same time, through contrastive learning, the class features across data domains can be separated.

[0071] The student-teacher collaborative learning framework specifically includes:

[0072] The teacher model is an exponential moving average copy of the student model, and is updated by the student model during training;

[0073] For unlabeled data in the expanded target domain, pseudo-labels are generated by the teacher model;

[0074] The student model receives both source domain data and extended target domain data simultaneously. Based on the data label type, it is jointly trained by combining the supervised loss and the pseudo-supervised loss corresponding to the pseudo-label.

[0075] Calculate and align the category prototypes of the source and target domains to minimize domain differences in the feature space, specifically including:

[0076] For each object category, calculate the source domain prototype and the target domain prototype separately. The prototype is the statistical average of the foreground features of all samples in that category.

[0077] Construct a domain discriminator that takes the source domain prototype and the target domain prototype as input and outputs the domain category.

[0078] Based on the domain category, adversarial training optimizes the domain discriminator to accurately distinguish between the source and target domains, while also optimizing the feature extractor of the visual perception model to make the extracted source and target domain features difficult for the domain discriminator to differentiate, thereby minimizing domain differences.

[0079] Through contrastive learning, cross-data domain categorical feature separability is achieved, specifically including:

[0080] Calculate the cross-domain prototype contrast loss, which is used to narrow the feature distance between the source domain prototype and the target domain prototype within the same category, and to widen the feature distance between prototypes of different categories. Cross-domain prototype comparison loss is implemented using the InfoNCE loss function, the formula of which is as follows:

[0081] ;

[0082] in, For object categories, the source domain prototypes and target domain prototypes aggregated during training are respectively and The features extracted in each batch are Harmony , Temperature parameters for controlling the sharpness of the distribution, This is the function for calculating cosine similarity.

[0083] The implementation methods for each step are explained in detail below with specific examples. Figure 1 This is an overall flowchart of the method of the present invention.

[0084] Step 1: Obtain multi-view images of the target object and perform 3D reconstruction to obtain a 3D model of the target object.

[0085] This step is used to build a high-precision 3D model of the target object and give it realistic material properties, providing high-quality source data for subsequent rendering.

[0086] The input for this step is the actual target object, taking a mobile phone front camera bracket as an example. First, a multi-view image acquisition platform is built, consisting of an industrial camera (such as a RealSense D435i) and a U5 robotic arm. The camera is fixed to the end of the robotic arm, and by changing the pose of the robotic arm, approximately 100 images from different perspectives are acquired, ensuring that each image covers different sides of the object. Simultaneously, a checkerboard calibration board is used to calibrate the camera, obtaining the camera's intrinsic parameter matrix K and distortion coefficients.

[0087] This step employs a two-stage reconstruction approach to decouple the object's geometry from its texture and material:

[0088] The first stage involves geometric reconstruction. The acquired multi-view images and their corresponding camera poses are input into the neural network, employing a neural implicit surface reconstruction method (such as a variant of NeRF). The network optimizes the Signed Distance Function (SDF) field through differentiable volume rendering. The SDF field is represented by a dense mesh with a resolution of 256³, and each mesh vertex stores one SDF value. During training, the rendering resolution is set to 512×512, the number of ray sampling points is 128, and the SDF gradient is constrained using Eikonal regularization loss. After training, the zero isosurface is extracted from the SDF field to obtain the initial mesh model. To further improve geometric accuracy, a differentiable Marching Tetrahedra (DMTet) mesh is used to refine the mesh, optimizing vertex positions to align with the SDF zero isosurface.

[0089] The second stage involves explicitly modeling the ambient light and reconstructing the PBR materials of the objects. After obtaining the geometric mesh, the mesh model is imported into the differentiable rendering pipeline and rendered using physically based rendering equations. The materials are represented by spatially varying PBR parameters: diffuse albedo, roughness, and metallicity. Lighting is represented using a High Dynamic Range (HDR) environment map with a resolution of 64×128.

[0090] The output of this step is a high-precision 3D model of the target object with PBR material parameters, which will be used for the simulation scene construction in subsequent steps.

[0091] Step 2: Generate 3D models of multiple interfering objects based on the 3D model.

[0092] This step is used to automatically generate a large number of interference objects of various shapes to enrich the complexity of the simulation scene and improve the model's ability to generalize to unknown objects.

[0093] The inputs for this step are a random noise vector and a 3D model generation model. The 3D model generation model employs a generative adversarial network-based architecture, capable of mapping Gaussian noise into a hybrid representation composed of a tri-plane network and a small-parameter network based on a multi-layer perceptron (MLP). This representation simultaneously represents both RGB and SDF fields. The 3D model generation model is trained using multiple categories (such as chairs, tables, and cars) from the ShapeNet dataset. The learning rates for the generator and discriminator are set to 0.0015 and 0.0005, respectively. The Adam optimizer is used, and the model is trained on four RTX 3090 GPUs for a total of 600,000 iterations.

[0094] Subsequently, noise vectors are randomly sampled from a standard normal distribution and input into the trained 3D model to generate a unique textured mesh for each noise vector. The number of generated meshes is set to 500, covering different geometries and texture styles. The generated meshes are extracted from the SDF field using a marching cubes algorithm and automatically assigned texture maps.

[0095] The output of this step is a 3D model library containing 500 interfering objects, stored in .obj format, and an index library is built for subsequent rendering.

[0096] Step 3: Randomly place the 3D models of the target object and the interference object in the simulation environment, randomly set the imaging parameters, and render to generate source composite data. The source composite data is constructed as the source domain.

[0097] This step involves constructing a highly randomized simulation scene based on a physical rendering engine, and generating source composite data and corresponding precise annotations in batches.

[0098] The input for this step is a 3D model of the target object and a library of 3D models of interfering objects. Using the BlenderProc rendering engine, a blank scene is created in the simulation environment, with a ground plane and four walls added as the base environment. Random textures can be applied to the ground and walls. 2-5 target objects are randomly selected from the target object model library, and 3-10 interfering objects are randomly selected from the interfering object model library. The positions and orientations of all objects are randomly generated within the scene constraint space: horizontal positions are uniformly sampled within [-0.5m, 0.5m], and heights are slightly perturbed according to the ground height; the object placement postures are randomly perturbed to simulate a slight tilt of natural placement. Simple collision detection is performed; if the overlap volume between the bounding box of a newly placed object and an already placed object exceeds a threshold (30% of the smaller object's volume), the position is resampled.

[0099] When setting up ambient lighting, randomly select an HDR map from the Poly Heaven HDR environment map library and rotate it randomly around the vertical axis. Simultaneously add 1-3 additional point or area lights, with their positions randomly sampled in the upper hemisphere of the scene. The camera model is randomly selected as either a pinhole or fisheye model, with a 50% probability each. The camera position uses spherical coordinates, with the radius r uniformly sampled between 0.8m and 1.5m, the azimuth angle θ uniformly sampled between 0 and 2π, and the elevation angle φ uniformly sampled between π / 6 and π / 3. The camera's gaze point is the center of the target object plus a random offset, with the offset uniformly sampled between -0.1m and 0.1m. The camera focal length is randomly perturbed based on a reference focal length of 24mm.

[0100] Each scene uses different camera parameters, rendering 15-20 images.

[0101] The output of this step is source synthetic data, including RGB images, instance segmentation masks, depth maps, and object pose information. This data will be used as the source domain for subsequent domain adaptation training.

[0102] Step 4: Obtain a small number of real-world scene images as anchor images, and then use annotations to identify the foreground and background of the anchor images.

[0103] This step is used to obtain a small number of real-world scene images as anchor points, and to extract the foreground and background of the anchor point images through annotation.

[0104] The input for this step is a small number of real-world scene images. A small number (e.g., 48) of real-world scene images are collected as anchor images. The foreground mask of the target object is extracted by a segmentation model (e.g., Segment Anything Model, SAM) or by manual annotation to obtain the foreground and background of the anchor images.

[0105] The output of this step is the anchor point foreground image and the anchor point image background. The anchor point foreground image will be used as a style image for the subsequent high-fidelity composite data foreground generation, while the anchor point background image will be expanded and used as the background of the high-fidelity composite data.

[0106] Step 5: Using a two-stage domain transfer network, a high-fidelity synthetic foreground is generated with the foreground of the source synthetic data as content and the foreground of the anchor image as style. An image inpainting model is used to expand the background of the anchor image and fuse it with the high-fidelity synthetic foreground to obtain a high-fidelity synthetic image.

[0107] This step is used to transfer the source synthetic data to a realistic visual style and blend diverse backgrounds to generate a high-fidelity synthetic image.

[0108] The input for this step is the foreground image and anchor point image of the source synthetic data. This step includes two sub-steps: a two-stage foreground visual domain transfer and a background expansion and fusion.

[0109] The two-stage foreground visual domain transfer sub-step uses a two-stage transfer network to transfer data. The two-stage transfer network consists of two cascaded feedforward style transfer networks, both employing an encoder-decoder structure, and introducing wavelet transform and a foreground-aware mask into the feature space. The first four convolutional layers (conv1_1 to conv4_1) of a VGG-19 pre-trained dataset on ImageNet are used as fixed feature extractors. After each encoding layer, a Discrete Wavelet Transform (DWT) is performed on the feature map, decomposing it into low-frequency and high-frequency components. The low-frequency components continue to propagate along the backbone, while the high-frequency components are directly fed into the corresponding decoder layer via skip connections.

[0110] The first stage performs artistic style transfer of camera imaging, using the source synthesized foreground as the content image and the anchor image foreground as the style image, which are then input into the encoder-decoder network of the first stage. This stage performs whitening and coloring transformation (WCT) on both low-frequency and high-frequency features, but introduces Gaussian blur into the high-frequency skip connections to suppress excessive high-frequency injection.

[0111] The second stage performs large-scale texture style transfer. In this stage, WCT transformation is performed only on the low-frequency backbone path, while high-frequency skip connections directly transfer the high-frequency features from the first stage (without style transformation) to preserve the transferred high-frequency details. The final high-fidelity composite foreground is output.

[0112] In the background expansion and fusion sub-step, the background is first randomly cropped, randomly rotated, and randomly perspective-transformed to obtain the background image to be incomplete. The background image to be incomplete is then input into an image incomplete diffusion model (such as StableDiffusion Inpainting) to fill in the hole regions and boundaries in the image, resulting in a high-fidelity synthesized background.

[0113] Finally, using image fusion technology, the migrated high-fidelity composite foreground and the expanded high-fidelity composite background are fused to obtain a high-fidelity composite image.

[0114] The output of this step is a high-fidelity synthesized image, which will be incorporated into the expanded target domain for subsequent model training.

[0115] Step 6: Obtain a large amount of unlabeled physical data, and construct an extended target domain by combining the high-fidelity synthesized image, a small amount of anchor point image, and a large amount of unlabeled physical data. Based on the source domain and the extended target domain, jointly train the visual perception model.

[0116] This step involves constructing datasets that extend the target and source domains. A student-teacher collaborative learning framework is employed, combining adversarial and contrastive learning, to jointly train a high-performance visual perception model.

[0117] The input to this step includes a large number of unlabeled real images, source synthetic data (as the source domain), anchor images and their annotations, and high-fidelity synthetic images; the output of this step is the trained student model, which can be directly applied to visual perception tasks such as object segmentation and detection in real-world scenes.

[0118] First, we construct datasets for both the extended target domain and the source domain. The source domain dataset consists of source synthetic data. All source domain data has corresponding annotations for subsequent training. The extended target domain dataset contains three types of data: anchor images obtained in step 4, high-fidelity synthetic images generated in step 5, and a large number of unlabeled real images. Except for the newly obtained large number of unlabeled real images, all other extended target domain data has corresponding annotations for subsequent training. All data is organized by object category to ensure that each category has sufficient samples in the extended target domain.

[0119] Secondly, joint training was conducted. In joint training, both the student model and the teacher model adopted the same network architecture, and the initial weights of the models were the results of pre-training on the COCO dataset.

[0120] First, the student model is pre-trained (e.g., for 5 epochs) in the source domain to acquire basic object perception capabilities. Then, the teacher model is initialized as a copy of the student model.

[0121] For each object category, a source domain prototype and an extended target domain prototype are maintained. The prototype is the statistical average of the foreground features of all samples in that category, with features taken from the instance-level features (dimension 256) output by the decoder. During training, after each batch, the prototype is updated online based on the sample features of the current batch (using momentum update, with the momentum coefficient set to 0.1). For unlabeled ground truth images, their pseudo-labels are generated by the teacher model and participate in the update of the extended target domain prototype.

[0122] The teacher model generates predictions for unlabeled data in the extended target domain, retaining only predictions with a confidence level higher than a threshold (e.g., 0.85) as pseudo-labels.

[0123] The student model receives both source domain data and target domain data: for source domain data, the supervised loss is calculated using real labels; for labeled data in the extended target domain (real labels and high-fidelity synthetic data), the supervised loss is also calculated; for unlabeled data, the pseudo-supervised loss is calculated using pseudo labels generated by the teacher model.

[0124] The teacher model parameters are updated by the student model using an exponential moving average (EMA).

[0125] Simultaneously, calculate the cross-domain prototype adversarial loss: construct a lightweight domain discriminator (such as a three-layer MLP), taking prototype features as input and outputting domain labels (source domain or extended target domain). Through adversarial training, on the one hand, optimize the discriminator to accurately distinguish between the source domain and the extended target domain prototypes; on the other hand, optimize the feature extractor so that the extracted features are difficult for the discriminator to distinguish.

[0126] This approach employs contrastive learning to narrow the gap between source domain prototypes and extended target domain prototypes within the same category, while simultaneously widening the gap between prototypes from different categories. It utilizes the InfoNCE loss function, which is added to the total loss with a weight of 0.1.

[0127] The total loss is the weighted sum of the losses mentioned above. Training was performed using the Adam optimizer with an initial learning rate of 1e-4, a total of 50,000 iterations, and a batch size of 16. Training was conducted on four NVIDIA RTX 3090 graphics processors and took approximately three days.

[0128] The output of this step is the trained student model, which can be directly applied to visual perception tasks such as object segmentation and detection in real-world scenarios.

[0129] Through the coordinated work of steps 1 to 6 above, this invention achieves a closed loop from data construction to model training, and obtains a high-performance object-level visual perception model under the condition of a small number of real samples.

[0130] Performance comparison experiments on public datasets (BOP segmentation challenge comparison experiments):

[0131] To comprehensively evaluate the performance of the complete workflow proposed in this work, this section selects representative state-of-the-art methods from the BOP (Benchmark for 6D Object Pose Estimation) open challenge for comparison. The BOP is currently the most representative public benchmark in the field of 6D object pose estimation, providing a unified dataset, evaluation protocol, and leaderboard for fair comparison of the performance of different algorithms. These methods cover different technical paradigms, ensuring the breadth and depth of the comparison. All comparison methods reproduce or cite their published results under the same public dataset and the same evaluation protocol:

[0132] 1) Zero-shot training-free methods: These methods do not rely on explicit training on the target dataset. They usually perform inference directly through a general visual model or geometric prior, and have strong versatility. However, they are often limited by feature representation ability under complex occlusion and realistic imaging conditions.

[0133] SAM6D: SAM6D is a zero-shot 6D pose estimation method based on the Segment Anything Model (SAM). This method utilizes a large-scale pre-trained segmentation model to perform instance-level segmentation of the target and combines geometric constraints to complete subsequent pose inference.

[0134] CNOS: CNOS is a zero-shot target recognition and pose estimation method based on class-independent feature matching, achieving target localization through cross-modal or cross-viewpoint feature similarity. This method does not require training on a specific dataset and has good generalization potential.

[0135] 2) Supervised learning-based training methods: Unlike zero-shot methods, these methods typically rely on large-scale synthetic or real data for training, enabling them to learn more discriminative feature representations and often have higher performance ceilings on specific tasks.

[0136] DLZDet: DLZDet is an anonymous method in the BOP challenge segmentation leaderboard. Its training data includes real object images and source synthetic data.

[0137] CosyPose: CosyPose employs a two-stage strategy. First, it obtains a coarse pose estimate through a detection network, and then performs fine alignment using a rendering-based optimization process.

[0138] ZebraPose: ZebraPose is a 6D pose estimation method based on keypoint regression and geometric constraints, focusing on robustness in occluded and symmetrical scenes. It improves the stability of perception by dividing the object surface into discriminative regions.

[0139] HccePose: HccePose improves the ability to distinguish occlusion and similar objects in complex scenes by introducing hierarchical geometric consistency and contextual constraints. This method learns multi-level feature representations during the training phase and combines local and global geometric constraints to solve for pose during inference.

[0140] To systematically verify the generalization ability of the method of this invention under different data characteristics and scenarios, this invention designs detailed synthetic data generation experiments based on multiple authoritative datasets:

[0141] For the BOP benchmark dataset, since the source synthetic data generation method of this invention is similar to that of the BOP benchmark dataset, this invention directly uses the 50,000 physically based rendered images provided by the BOP challenge as source synthetic data. In this invention, it is defined as standard "source synthetic data," and its main function is to construct an initial reference training set that has inherent domain differences from the real world, thereby providing a clear performance starting point for the subsequent evaluation of the effectiveness of the domain transfer method in this invention.

[0142] For the LM-O dataset, this experiment aims to verify the effectiveness of the style transfer method proposed in this invention under extreme conditions. This invention intentionally uses only a single real-world scene image as the style reference anchor point, and based on this, generates 3000 high-fidelity synthetic data images. This setting simulates the challenge of extremely scarce real-world reference data that may be faced in practical applications. Successfully generating visually coherent and diverse data from such a single-sample anchor point strongly demonstrates the advantages of the proposed transfer method in terms of data efficiency and generalization ability.

[0143] The experimental design on the YCB-V dataset focuses on testing the robustness of the method to complex domain shifts. This invention selects one real image as an anchor point for each of the first 48 training scenes, generating a total of 3500 high-fidelity synthetic images. Crucially, the lighting conditions and background environments of these selected anchor images differ significantly and systematically from those in the final test set. This intentionally introduced "misalignment" aims to simulate the inevitable variations in training and testing environments in real-world deployments, thereby testing whether the data generated by the method of this invention can help the model learn to isolate the interference of essential object features and apparent changes.

[0144] To systematically evaluate the effectiveness of the proposed synthetic data generation, transfer, and domain adaptation strategies in object perception tasks, this section presents comparative experiments on object segmentation performance under different training data configurations on multiple publicly available benchmark datasets. All experiments were trained using the same perceptual network structure to ensure that performance differences primarily stem from the training data and learning strategies themselves, rather than differences in model structure.

[0145] Table 1.1 Segmentation performance metrics of different methods and data on the LM-O dataset

[0146]

[0147] This invention selects MaskDINO as the basic perception model for instance segmentation and object detection tasks. MaskDINO is an end-to-end instance perception framework based on Transformer, which inherits the idea of ​​ensemble prediction and simultaneously models the object's category, location, and mask information through a unified decoder structure. Compared to traditional methods based on candidate boxes or region proposals, this type of model avoids complex post-processing steps and exhibits better global consistency and robustness in complex scenes.

[0148] The experiments strictly followed the official data partitioning and evaluation protocol of the BOP challenge, using physically based rendering (PBR) source synthetic data as the basic input for the synthetic data portion. Based on this, and depending on the experimental setup, high-fidelity synthetic data after transfer, a small amount of real-world labeled data, and unlabeled physical object data were introduced. Furthermore, domain adaptation training strategies were incorporated in some experiments.

[0149] Tables 1.1 and 1.2 present the experimental results on the LM-O and YCB-V datasets, respectively. The two tables have the same structure, and their meanings are explained as follows: "Labeled Real Data" represents the number of real labeled images used for training; "Unlabeled Real Data" represents the number of real images used for training but without human annotation; "Source Synthetic Data" represents the number of synthetic images obtained using domain randomization rendering; "High-Fidelity Synthetic Data" represents the number of synthetic images generated using the data transfer method proposed in this invention. AP represents average precision, AP50 and AP75 represent the average precision at intersection-over-union (IoU) thresholds of 0.5 and 0.75, respectively, and APS represents the average precision of small-scale targets. "-" in the tables indicates that this type of data was not used.

[0150] The experimental results in the table can be roughly divided into four categories according to the different types of training data: zero-shot methods that do not require training and perform inference directly during the testing phase; supervised methods that use only source synthetic data or source synthetic data and real data for training; methods that introduce transfer synthetic data on the basis of source synthetic data, but still use traditional supervised learning methods; and training schemes that introduce transfer synthetic data and combine it with domain adaptation learning.

[0151] Note that for the readability of the table data, the indicator values ​​are 100 times the calculated results, and the best indicator in the table is highlighted in bold, while the second best and the second-best indicators are marked with underlines.

[0152] Table 1.2 Segmentation performance metrics of different methods and data on the YCB-V dataset

[0153]

[0154] By analyzing the experimental results, the following three conclusions can be drawn:

[0155] 1) Under the zero-shot setting without training, the overall accuracy is significantly lower than that of learning-based methods. The first set of experiments with LM-O and YCB-V shows that SAM6D and CNOS can complete object segmentation without any training, achieving AP of approximately 40% to 60% on both datasets. This indicates that methods based on general feature matching or cue-driven approaches have some generalization ability in industrial object scenarios. However, compared to subsequent models trained on synthetic data, their overall performance and performance on small-to-medium scale targets are significantly lower, especially in high-occlusion scenarios like LM-O, where the performance drop is particularly pronounced for small targets. This result suggests that methods relying solely on training-independent approaches are insufficient to fully characterize fine-grained semantic differences in complex scenes.

[0156] 2) Introducing high-fidelity synthetic data after transfer, combined with a very small number of real anchor point samples, can significantly improve segmentation performance and exhibit consistent gains across multiple scales. When the high-fidelity synthetic data generated and transferred according to this invention is introduced on top of PBR synthetic data, MaskDINO shows significant performance improvements on both datasets. Taking LM-O as an example, using only one real-labeled image per scene as an anchor point can improve AP to 52.8%. After further introducing unlabeled real object data, AP reaches 53.1%, significantly better than the results trained using only PBR data. In the YCB-V dataset, 48 real-labeled images combined with transferred synthetic data can achieve performance levels comparable to models trained on hundreds of thousands of real data. More importantly, this performance improvement is stable on AP, AP50, and AP75, indicating that the transferred synthetic data not only improves overall accuracy but also enhances the representation ability of targets at different scales.

[0157] 3) Introducing domain adaptation training based on transferred data can achieve further and stable performance improvements on the real data distribution. Comparing the results of using only transferred synthetic data with those of simultaneously introducing unlabeled physical data and performing domain adaptation training reveals that the latter achieves additional performance gains on both LM-O and YCB-V. Although this improvement is more limited compared to the data transfer stage, it shows a consistent positive improvement on the overall and mesoscale targets. This indicates that when the synthetic data has already become quite close to the target domain in appearance through the transfer strategy, the main role of domain adaptation training is to refine and align the feature space, thereby mitigating the remaining distribution shift. This result verifies the collaborative design concept of "synthetic data transfer as the foundation and domain adaptation learning as a supplement" in the method of this invention.

[0158] Obviously, the above embodiments are merely examples for clear illustration and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A few-shot object-level visual perception method based on cross-domain transfer of synthetic data, characterized in that, include: Step 1: Obtain multi-view images of the target object and perform 3D reconstruction to obtain a 3D model of the target object; Step 2: Generate 3D models of multiple interfering objects based on the 3D model; Step 3: Randomly place the 3D models of the target object and the interference object in the simulation environment, randomly set the imaging parameters, and render and generate source synthetic data. The source synthetic data is constructed as the source domain. Step 4: Obtain a small number of real-world scene images as anchor images, and then annotate them to obtain the foreground and background of the anchor images. Step 5: Using a two-stage domain transfer network, a high-fidelity synthetic foreground is generated with the foreground of the source synthetic data as content and the foreground of the anchor image as style. An image inpainting model is used to expand the background of the anchor image and fuse it with the high-fidelity synthetic foreground to obtain a high-fidelity synthetic image. Step 6: Obtain a large amount of unlabeled physical data, and construct an extended target domain by combining the high-fidelity synthesized image, a small amount of anchor point image, and a large amount of unlabeled physical data. Based on the source domain and the extended target domain, jointly train the visual perception model. Step 6, which involves jointly training a visual perception model based on the source domain and the extended target domain, specifically includes: Based on the source domain and the extended target domain, a visual perception model is jointly trained through a student-teacher collaborative learning framework. During the training process, the category prototypes of the source domain and the target domain are calculated and aligned to minimize the domain differences in the feature space. At the same time, through contrastive learning, the category features across data domains can be separated.

2. The method according to claim 1, characterized in that, Step 1 involves obtaining multi-view images of the target object and performing 3D reconstruction to obtain a 3D model of the target object. Specifically, this includes: A neural implicit surface reconstruction method is adopted to optimize the symbolic distance function field based on the acquired multi-view images in order to reconstruct the three-dimensional geometric mesh of the target object; The reconstructed 3D geometric mesh of the target object is imported into the differentiable rendering pipeline. By jointly optimizing the ambient lighting and the physically based rendering material of the 3D model, a 3D model of the target object with realistic reflection characteristics is obtained. The physically based rendering material of the 3D model includes reflectivity, roughness, and metallicity.

3. The method according to claim 1, characterized in that, Step 2, which involves generating 3D models of multiple interfering objects based on a 3D model, specifically includes: The latent encoding of random sampling is input into the 3D model to generate a model, generating a signed distance function field and a texture field; The mesh is extracted from the symbolic distance function field using a surface extraction algorithm, and the corresponding texture is assigned to the texture field to obtain the three-dimensional model of the interfering object.

4. The method according to claim 1, 2, or 3, characterized in that, Step 5, which involves using a two-stage domain transfer network to generate a high-fidelity synthetic foreground using the foreground of the source synthetic data as content and the foreground of the anchor image as style, specifically includes: In the first stage, the foreground of the source synthetic data is used as the content image and the anchor point real foreground is used as the style image. After wavelet transform of the image features, whitening and color transformation are performed on both low-frequency and high-frequency components to realize the transfer of the camera imaging artistic style and obtain a coarsely aligned image. In the second stage, the coarsely aligned image output from the first stage is used as the content image, and the real foreground at the same anchor point is used as the style image. After performing wavelet transform on the image features, only the low-frequency components are whitened and tinted, while the high-frequency components directly inherit the high-frequency features from the first stage, thereby realizing the transfer of large-scale texture style and finally outputting the high-fidelity synthesized foreground.

5. The method according to claim 4, characterized in that, Step 5, which utilizes an image inpainting model to expand the background of the anchor point image, specifically includes: The anchor point image background is randomly cropped, rotated, and subjected to affine transformation in sequence, and finally input into an image restoration model to complete the image, resulting in an expanded anchor point image background.

6. The method according to claim 5, characterized in that, The aforementioned student-teacher collaborative learning framework specifically includes: The teacher model is an exponential moving average copy of the student model, and is updated by the student model during training; For unlabeled data in the extended target domain, pseudo-labels are generated by the teacher model; The student model receives source domain data and extended target domain data simultaneously, and performs joint training based on the data label type, combining the supervised loss with the pseudo-supervised loss corresponding to the pseudo-label.

7. The method according to claim 5, characterized in that, The calculation and alignment of category prototypes between the source and target domains to minimize domain differences in the feature space specifically includes: For each object category, a source domain prototype and a target domain prototype are calculated separately. The prototype is the statistical average of the foreground features of all samples in that category. Construct a domain discriminator that takes the source domain prototype and the target domain prototype as input and outputs the domain category. Based on the domain category, adversarial training optimizes the domain discriminator to accurately distinguish between the source and target domains, while simultaneously optimizing the feature extractor of the visual perception model to make the extracted source and target domain features difficult for the domain discriminator to differentiate, thereby minimizing the domain differences.

8. The method according to claim 5, 6, or 7, characterized in that, The aforementioned method of achieving cross-data domain separability through contrastive learning specifically includes: Calculate the cross-domain prototype contrast loss, which is used to narrow the feature distance between the source domain prototype and the target domain prototype under the same category, and to widen the feature distance between prototypes of different categories. The cross-domain prototype comparison loss is implemented using the InfoNCE loss function, the formula of which is as follows: ； in, For object categories, the source domain prototypes and target domain prototypes aggregated during training are respectively and The features extracted in each batch are Harmony , Temperature parameters for controlling the sharpness of the distribution, This is the function for calculating cosine similarity.

Images