A generative data augmentation method and device for body perception model learning

Abstract

The application discloses a generative data enhancement method with body perception model learning, comprising: evaluating the average precision of the body perception model on the source domain verification set to establish the category prior weight; collecting unmarked observation images in the target domain environment to be perceived, calculating the uncertainty of all detection instances in each observation image based on the target detection result of the observation image by the body perception model, and calculating the difficulty score of each observation image by combining the category prior weight of the corresponding category, screening seed images from the observation images based on the difficulty score; performing physical boundary constraint and logical boundary constraint extraction on the seed images, and generating decoupled long and short texts; in the diffusion model denoising stage, the latent variable is updated in opposition through the feature gradient and the spatial mask guide, and finally the difficult example images are generated, which are mixed with the source domain training set for learning of the body perception model. The application solves the defects that the existing data enhancement is blindly expanded and difficult to touch the model decision boundary.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and machine vision. It is an application that combines computer vision algorithms, generative large models and robot active learning technology. In particular, it relates to a generative data augmentation method and device for embodied perception model learning. Background Technology

[0002] Currently, embodied robots primarily rely on detectors based on deep convolutional neural networks (such as the YOLO series) for target perception in their environment. However, when an embodied robot migrates from its training environment to an unseen target scene, shifts in the distribution of lighting, texture, and object layout can cause severe "perceptual blind spots" in the detector, resulting in a significant drop in confidence or complete missed detections.

[0003] To address the domain offset problem, traditional methods often employ online test-time adaptation or data augmentation. Online model updates are prone to catastrophic forgetting in variable environments; while conventional generative data augmentation methods typically only perform global illumination and color changes, failing to deeply alter the underlying feature space upon which the model's decisions rely, resulting in minimal improvement for targets with poor detection performance. Furthermore, when dealing with large scenes, manually labeling target data frame by frame is extremely time-consuming and costly. Therefore, accurately identifying a small number of high-value, difficult examples for pinpoint annotation and generating highly challenging samples at the feature level that can effectively probe the boundaries of the model's decision-making process has become a pressing technical challenge. Summary of the Invention

[0004] To address the performance degradation caused by distribution offset when deploying embodied perception models across domains, and the lack of specificity in conventional generative augmentation methods, this invention proposes a generative data augmentation method and apparatus for embodied perception model learning. This method combines unsupervised multimodal heuristic search with latent spatial feature-level adversarial attacks: in the data screening stage, it automatically locates hard example images where the embodied perception model is weak without target domain labeling; in the data generation stage, by applying feature gradient guidance to the diffusion model's denoising process, it generates augmented samples that can effectively probe the decision boundaries of the embodied perception model in deep feature dimensions while maintaining the target topology, thus overcoming the shortcomings of conventional data augmentation methods that can only perform surface pixel transformations and lack specificity.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] The first aspect of this invention provides a generative data augmentation method for embodied perception model learning, comprising:

[0007] Evaluate the average accuracy of the embodied perception model for each instance category on the source domain validation set, and calculate the category prior weights corresponding to each category based on the average accuracy.

[0008] In the target domain environment to be perceived, an embodied robot is used to collect unlabeled observation images. Based on the target detection results of the observation images by the embodied perception model, the instance uncertainty of all detected instances in each observation image is calculated. Based on the instance uncertainty and the category prior weight of the corresponding category, the difficulty score of each observation image is calculated. Based on the difficulty score, several seed images are selected from all observation images. The seed images are labeled to obtain the true bounding boxes of the instances.

[0009] Physical boundary constraints and logical boundary constraints are extracted from the seed images respectively. The extracted logical boundary constraints are decoupled into a long text description containing scene description and a short text containing only the real category.

[0010] The seed image's depth features are extracted through the intermediate layer of the embodied perception model and used as static feature anchors. The seed image with real bounding boxes, physical boundary constraints, long text, and short text are input into a diffusion model with a control net for denoising generation. During denoising, the feature interference loss between the current feature and the static feature anchor is calculated. The latent variables are updated based on the feature interference loss, and the updated latent variables are returned to the diffusion model to finally obtain a clear hard example image. The hard example image and the source domain training set are then mixed to learn the embodied perception model.

[0011] In some embodiments, the formula for calculating the category prior weight is:

[0012]

[0013] in, For instance category The corresponding category prior weights; The instance categories for evaluating the embodied perception model on the source domain validation set. Average accuracy; This is the minimum weight threshold.

[0014] In some embodiments, the difficulty score of the observed image is calculated according to the following formula:

[0015]

[0016]

[0017]

[0018] in, For observation images Difficulty score; For observation images The collection of detection instances within; For the first Instance uncertainty of individual instances; The embodied perception model's response to the input observed image The extracted first Individual Detection Boxes Corresponding instance category The prediction confidence level; According to the first The prior weight of the class corresponding to the predicted instance class of each detection box; For the first Difficulty score for each instance;

[0019] If the embodied perception model perceives the input observed image If no instances are detected, it is determined to be a severe perception blind spot, and the observation image is assigned accordingly. Difficulty score .

[0020] In some embodiments, a number of seed images are selected from all observed images based on the difficulty score, including:

[0021] Sort all the observation images by difficulty score in descending order, and select the top K observation images as seed images.

[0022] In some embodiments, extracting physical boundary constraints from the seed image includes:

[0023] The seed image is input into the segmentation model, and the obtained segmentation result is subjected to edge capture and texture noise filtering to obtain a clean edge map that preserves the complete contour of the target, which serves as the physical boundary constraint.

[0024] Extracting logical boundary constraints from the seed image includes:

[0025] The seed image is input into a large language vision model, which combines the constraints of the annotation list on the seed image to generate a long text describing the scene material and lighting, while constructing a short text based on the target category extracted from the real bounding box.

[0026] In some embodiments, the denoising process calculates the feature interference loss between the current feature and the static feature anchor, and updates the latent variables based on the feature interference loss, including:

[0027] Within the defined iterative attack window, perform the following operations at each time step:

[0028] Decode the latent variables of the current time step into an image tensor;

[0029] The image tensor is input into the embodied perception model to extract the current features at the same layer as the static feature anchor, the feature interference loss between the current features and the static feature anchor is calculated, and the gradient of the current features is obtained by taking the derivative with respect to the latent variables.

[0030] Generate a binary spatial mask of the potential spatial scale based on the target bounding box, and use the binary spatial mask to filter and normalize the current feature gradient;

[0031] Calculate the dynamically decaying attack step size and update the latent variables at the current time step by combining the normalized current feature gradient.

[0032] In some embodiments, the formula for calculating the current feature gradient is:

[0033]

[0034]

[0035] in, The gradient of the current feature; The current latent variables are in sequence format; The current feature is the interference loss; For the current features, by using image tensors The embodied perception model is input to trigger the forward hook function to obtain the result; The static feature anchor point; Mean square error; It is the negative amplification constant.

[0036] In some embodiments, the current feature gradient is filtered and normalized using the binary space mask, and the calculation formula includes:

[0037]

[0038]

[0039] in, The serialization mask is obtained by performing dimensional reshaping and permutation operations on the binary space mask. With current latent variables The dimensions are the same; The gradient of the current feature; The feature gradient is filtered element by element by element through the serialized mask, retaining only the gradient components within the region corresponding to the target bounding box; The gradient is the normalized value. To prevent constants with a denominator of zero; This is the Hadama product operation.

[0040] In some embodiments, the current latent variable is updated according to the following formula:

[0041]

[0042]

[0043]

[0044] in, For the updated current latent variables; For the current latent variables; The gradient is the normalized value. For dynamic attack step size; For the current time step Attack window Linear decay rate parameter within; This is the preset maximum attack scaling ratio.

[0045] A second aspect of the present invention provides a generative data augmentation apparatus for embodied perception model learning, comprising:

[0046] The prior evaluation module is configured to evaluate the average accuracy of the embodied perception model for each instance category on the source domain validation set, and calculate the category prior weights corresponding to each category based on the average accuracy.

[0047] The difficult instance mining module is configured to, in the target domain environment to be perceived, use an embodied robot to collect unlabeled observation images, calculate the instance uncertainty of all detected instances in each observation image based on the target detection results of the observation images by the embodied perception model, calculate the difficulty score of each observation image based on the instance uncertainty and the category prior weight of the corresponding category, select a number of seed images from all observation images according to the difficulty score, and label the seed images to obtain the true bounding boxes of the instances.

[0048] The constraint extraction module is configured to extract physical boundary constraints and logical boundary constraints from the seed image respectively, and decouple the extracted logical boundary constraints into a long text description containing scene description and a short text containing only the real category.

[0049] The adversarial generation module is configured to extract the depth features of the seed image through the intermediate layer of the embodied perception model as static feature anchors; input the seed image with real bounding boxes, physical boundary constraints, long text, and short text into a diffusion model with a control net for denoising generation; calculate the feature interference loss between the current feature and the static feature anchor during denoising; update the latent variables based on the feature interference loss; return the updated latent variables to the diffusion model; finally obtain a clear hard example image; and mix the hard example image and the source domain training set with the learning of the embodied perception model.

[0050] Compared with the prior art, the present invention has at least the following beneficial effects:

[0051] (1) This invention abandons the traditional hard example mining method that requires a large number of target domain labels to calculate the loss, and proposes to combine source domain priors and low confidence outputs to calculate the image-level difficulty score. This can reduce the computational overhead of embodied intelligent agents and lower the requirements for manual annotation.

[0052] (2) This invention introduces a rigorous dual-text encoding decoupling structure into the denoising pipeline of the diffusion model, which not only avoids the illusion problem of language models, but also utilizes Spatial masking strictly limits gradient attacks to the topological boundaries of the target, preventing meaningless background collapse.

[0053] (3) This invention performs feature-level adversarial attacks at the midpoint of the diffusion process. The generated images are not only visually diverse, but also highly deceptive adversarial examples in the deep feature space of the embodied perception model. After fine-tuning with data containing such high-dimensional feature-difficult examples, the decision boundary of the embodied perception model is forced to shift from fragile surface textures to robust geometric topologies. Experiments show that after fine-tuning with the adversarial examples generated by this invention, the embodied perception model is significantly improved in the following aspects: First, the average detection accuracy of the target domain is further improved compared with conventional enhancement methods, especially in difficult categories where the embodied perception model originally had weak detection capabilities; Second, the recall rate is significantly improved, effectively reducing missed detections in complex scenes; Third, because the adversarial examples introduce universal perturbations in the deep feature space that are not dependent on the appearance of a specific room, the fine-tuned embodied perception model exhibits stronger generalization ability in completely unseen zero-sample environments. Attached Figure Description

[0054] Figure 1 A flowchart of a generative data augmentation method provided in a first aspect embodiment of the present invention.

[0055] Figure 2 This is a logical structure block diagram of an embodiment of a generative data augmentation device provided in a second aspect of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0057] Conversely, this application covers any alternatives, modifications, equivalent methods, and schemes made within the spirit and scope of this application as defined by the claims. Furthermore, to provide the public with a better understanding of this application, certain specific details are described in detail below. However, this application can be fully understood by those skilled in the art even without these detailed descriptions.

[0058] This invention discloses a generative data augmentation method for learning an embodied perception model, wherein the embodied perception model is mounted on an embodied robot. In this embodiment, the embodied robot operates in indoor home scenarios such as a kitchen and living room, using a visual perception model to perform real-time target detection of everyday objects to support interactive tasks such as navigation and grasping. The method of this invention involves the following models: a pre-trained embodied perception model (using the YOLO architecture) for target detection and feature extraction; a general segmentation model (using Mask2Former) for background mask extraction; a large language vision model (using Qwen2.5-VL) for scene description generation; a diffusion model (using FLUX with dual text encoders T5 and CLIP) and a control network (using ControlNet) for image generation and adversarial attacks. The industrial control computer of the embodied robot is equipped with an Ubuntu operating system and an NVIDIA GPU to support tensor operations of the embodied perception model and image generation and adversarial attack inference based on the diffusion model.

[0059] See Figure 1 The first aspect of this invention provides a generative data augmentation method for embodied perception model learning, comprising the following steps:

[0060] Step S100, Source Domain Prior Evaluation: Evaluate the average accuracy of the embodied perception model for each instance category on the source domain validation set, and calculate the category prior weights corresponding to each category based on the average accuracy;

[0061] Step S200, Unsupervised Difficult Instance Mining and Seed Selection: In the target domain environment to be perceived, an embodied robot is used to collect unlabeled observation images. Based on the target detection results of the observation images by the embodied perception model, the instance uncertainty of all detected instances in each frame of the observation image is calculated. Based on the instance uncertainty and the prior weight of the corresponding category, the difficulty score of each frame of the observation image is calculated respectively. Based on the difficulty score, several seed images are selected from all observation images. All seed images are labeled to obtain the true bounding boxes of the instances.

[0062] Step S300, Semantic and Structural Constraint Extraction: Physical boundary constraints and logical boundary constraints are extracted from the selected seed images respectively. The extracted logical boundary constraints are decoupled into long text descriptions containing scene descriptions and short texts containing only real categories.

[0063] Step S400, Adversarial Generation in the Latent Space: The deep features of the seed image are extracted through the intermediate layer of the embodied perception model and used as static feature anchors. The seed image with real bounding boxes, physical boundary constraints, long text and short text are input into the diffusion model for denoising and generation. During the denoising process, the feature interference loss between the current feature and the set static feature anchor is calculated. The latent variables are updated based on the feature interference loss. The updated latent variables are returned to the diffusion model, and finally a clear hard example image is obtained. The hard example image and the source domain training set are mixed for the learning of the embodied perception model.

[0064] In some embodiments, step S100 aims to perform source domain prior evaluation and parameter setting, specifically including:

[0065] Step S101: Obtain the source domain dataset: In one specific embodiment of this application, the source domain dataset is collected from six kitchen layouts in the AI2-THOR simulation environment. The robot moves along a preset navigation trajectory and captures RGB images frame by frame, with a resolution of 1024×1024 pixels, stored in JPEG format. Each image is accompanied by a YOLO format annotation file with the same name, recording the bounding boxes of 28 everyday item categories with normalized center coordinates and width and height. A total of 600 valid images are collected, divided into a 500-image training set and a 100-image validation set, where the validation set is used for subsequent average accuracy evaluation.

[0066] Step S102: Obtain the target detection index of the embodied perception model on the source domain validation set, that is, the average accuracy of each category within the range of intersection-union threshold of 0.5 to 0.95. .

[0067] Step S103: Calculate the category of each instance. Corresponding category prior weights :

[0068]

[0069] in, This is a set minimum weight threshold to prevent the model's weights from dropping to zero when the average precision for a particular instance class reaches 1.0. It is a constant, and in this embodiment, it is taken as... Save the prior weights of all instance categories as a prior mapping table. The prior weights established in this step are inversely allocated according to the detection accuracy, so that subsequent enhancement resources are automatically focused on categories where the model is weak, avoiding wasting computational resources on categories that have been fully learned.

[0070] Understandably, compared to conventional data augmentation methods that apply the same transformation to all categories indiscriminately, step S100 in this embodiment establishes a quantitative prior difficulty for each category based on the objective evaluation of the source domain validation set. This allows subsequent hard case mining and generation to allocate resources according to the actual weakness distribution of the embodied perception model, ensuring from the root that limited annotation and generation resources are concentrated on the categories that can best improve the overall performance of the embodied perception model.

[0071] In some embodiments, step S200 specifically includes:

[0072] Step S201: In the unknown target domain environment to be perceived, the embodied robot captures unlabeled observation images along the planned trajectory.

[0073] Step S202, Difficulty Score Calculation

[0074] A single observed image is input into the embodied perception model, and the embodied perception model is set at a confidence threshold. Set to 0.05, Intersection over Union (IoU) threshold Inference is performed with a value set to 0.45. Specifically, the first... Individual Detection Boxes Corresponding instance category Prediction confidence Calculate the first Instance uncertainty of individual instances :

[0075]

[0076] And according to the first The predicted instance category corresponding to each detection box Retrieve the corresponding category prior weight from the prior mapping table obtained in step S100. .

[0077] Calculate the first according to the following formula. Difficulty score of each instance :

[0078]

[0079] For the input single observation image If the embodied perception model fails to detect any valid target, it is determined to be a severe perception blind spot, and the observed image is assigned accordingly. Image-level difficulty score If a target is detected, then the observed image... Image-level difficulty score Take observation image The maximum value of the difficulty score for all instances within the range, i.e.:

[0080]

[0081] in, For a single observation image The collection of detection instances within.

[0082] Step S203: Seed image selection and annotation

[0083] Traverse all observed images and press Sort in descending order and extract the top-ranked items. (In this embodiment) The observed images are used as seed image sequences, and the true bounding boxes of each instance in the seed images are obtained through manual annotation.

[0084] In other embodiments, the results can be automatically exported through the instance segmentation interface built into the simulation environment, or the high-confidence prediction results of the embodied perception model can be used as pseudo-real bounding boxes when the confidence level meets preset conditions. This application does not make specific limitations in this regard.

[0085] This step can automatically filter out a very small number of high-value images with the weakest model perception without target domain annotation, reducing full frame-by-frame annotation to fixed-point sparse annotation, which greatly reduces the time and economic cost of annotation.

[0086] Understandably, compared to the limitations of traditional hard case mining, which relies on target domain labeling or active learning that can only select from existing samples, step S200 in this embodiment can automatically complete the difficulty ranking of a massive number of unlabeled observation images without target domain labeling, and only label a very small number of high-value seeds, achieving an efficient balance between labeling costs and enhancement benefits.

[0087] In some embodiments, step S300 aims to extract physical and logical boundaries from all seed images selected in step S200, specifically including:

[0088] Step S301: Extraction of physical boundary constraints (spatial structure constraints)

[0089] A clean edge map that preserves the complete contour of the target is obtained from the seed image as a physical boundary constraint, specifically including:

[0090] Background masking and edge protection: The background region mask of the seed image is extracted using a general segmentation model (in this embodiment, the pre-trained instance segmentation model Mask2Former is used). Use size is Using an all-one matrix as the structural kernel, the background region is masked. Two morphological etching operations were performed to obtain the etched background region mask. This causes the white background area to shrink outwards by pixels, preventing damage to the target outline during subsequent edge erasure.

[0091] Low-threshold edge capture: The seed image is converted to grayscale, Gaussian blur is applied, and then the Canny operator with double thresholds set to (10, 50) is used to detect and obtain the original edge map containing faint lines. This is to ensure that even the faintest object outline information is obtained.

[0092] Texture noise filtering: Filters the original edge map The middle part belongs to the eroded background area mask. The pixel is set to 0, that is:

[0093]

[0094] Then, a connected component search algorithm is called to obtain the contour vectors of all edge segments and calculate their geometric arc lengths. A minimum length threshold is set. The value is 30. Unstructured texture fragments with arc lengths below this threshold are removed, and only long lines representing the structure of the object are retained. The retained long lines are redrawn on the blank tensor to generate a clean edge map that retains the complete outline of the target as input to the control net.

[0095] Step S302: Extraction of logical boundary constraints (semantic constraints)

[0096] Controlled Scene Description: The seed image is input into the large language vision model (in this embodiment, the large language vision model is Qwen2.5-VL). During the generation process, the large language vision model is forced to describe only the target categories and their physical attributes within the manually labeled list, ignoring background objects outside the manually labeled list to ensure semantic consistency with the detection labels. After generating the output, the format is parsed and decoupled into: a long text containing scene descriptions such as material and lighting, and a short text containing only descriptions of the real categories, wherein the short text is constructed based on the target categories determined by extracting the real bounding boxes.

[0097] Understandably, compared to conventional generative augmentation which lacks constraints and is prone to target deformation or semantic confusion, in step S300 of this embodiment, structural constraints lock the geometric contour of the target through a clean edge map, and semantic constraints eliminate the object illusion of the large language model through a whitelist mechanism. From the two dimensions of physical boundary and logical semantics, the spatial fidelity and content accuracy of the subsequently generated image are ensured, providing a controllable perturbation space for the gradient attack in the subsequent step S400.

[0098] In some embodiments, step S400 is designed to implement latent spatial feature gradient guidance and adversarial updates.

[0099] Before updating, the diffusion model and control network are initialized. A forward hook function is registered using layer 9 of the embodied perception model's backbone network. The seed image is input to extract dimensional features, and the gradient backpropagation of these features is cut off, serving as static feature anchors. Layer 9 was chosen because it strikes a balance between semantic abstraction and spatial resolution, effectively guiding the diffusion model's decisions from fragile textures to robust geometric topologies.

[0100] A manual embedding mapping approach is employed: based on a dual-encoder decoupling strategy, short texts are fed into the CLIP encoder of the diffusion model (CLIP excels at short text category anchoring but has a token cap of 77) to obtain pooled output features for anchoring the target category; long texts are fed into the T5 encoder of the diffusion model (T5 excels at long text scene description and supports 512 token expansion) to obtain sequence output features to enrich scene texture, thereby suppressing cross-modal semantic confusion. By feeding short and long texts into their respective adapted encoders, semantic confusion or truncation caused by mixing them is avoided.

[0101] Set the total number of steps to generate. Attack window and maximum attack scaling , These are the starting and ending step sizes of the attack window, respectively, satisfying... In one specific embodiment of this application, take , , (Considering that the latent variable noise is too strong in the early stages of denoising, the image semantics are not yet formed, and the features have no guiding value; in the later stages of denoising, the image is close to convergence, and gradient attacks can only disturb the surface texture. Steps 10 to 20 are in the intermediate stage where the semantic structure is initially formed but not yet solidified. At this time, applying feature gradients can effectively change the deep decision response of the embodied perception model while maintaining the overall rationality of the image. Therefore, in this embodiment, the total number of generation steps is taken.) The first half of the time window is used as the attack window. .

[0102] After completing the above initialization and parameter settings, at each time step within the attack window The iterative calculation is as follows:

[0103] Step S401: Latent variable unpacking and decoding

[0104] For diffusion models based on the Transformer architecture, the current latent variables are in sequence format. Reconstructing it into a spatial potential tensor with height and width through unpacking operations. In FLUX, each sequence token corresponds to a 2×2 pixel block in the latent space, which is restored to the complete spatial dimension after unpacking.

[0105] Utilizing the inherent scaling factor of the variational autoencoder (VAE) built into the diffusion model For the aforementioned spatial potential tensor After normalization and restoration, the signal is input to the decoder of the variational autoencoder. Map the current latent variables to pixel space, restore them to an image tensor, and crop them to the target value range:

[0106]

[0107] in, For image tensors; The unpacked potential tensor of the space; is the scaling factor of the variational autoencoder, and its value is the empirical standard deviation of the encoder output distribution after the variational autoencoder is trained. It is used to normalize the magnitude of the latent variable to near the unit variance to stabilize the denoising process of the diffusion model. In this example, it is taken from the configuration file of the FLUX pre-trained model and is approximately equal to 0.3611. For the decoder of the variational autoencoder, the output value range for ; This is a truncation function used to truncate functions that might exceed the limits after linear mapping. A small overflow is constrained within a defined range to generate image tensors for the input format of the embodied perception model. To truncate the boundaries, the upper and lower limits of the input image pixels required by the embodied perception model are taken respectively; For truncation function The linear transform coefficients in the expression are derived from the output range of the decoder of the variational autoencoder. Sure; , , , All are fixed values, determined by the data preprocessing conventions of the embodied perception model.

[0108] In one specific embodiment of this application, the input value range required by the embodied perception model for The decoded output is divided by Add After completing the linear mapping, a truncation operation is performed to ensure the numerical stability of subsequent feature extraction. If other diffusion model architectures are used, the above scaling factors... The corresponding model configuration and range mapping coefficients should be read. This depends on the output convention of its codec.

[0109] Step S402: Feature Interference Loss and Feature Gradient Extraction

[0110] Image tensor Inputting the embodied perception model triggers a hook to obtain the current features. ;

[0111] Calculate current features Static feature anchors with cut gradient Mean square error between And multiplied by the negative amplification constant Calculate the current feature disturbance loss and feature interference loss Current latent variables regarding sequence format The gradient of the current feature is obtained by automatic differentiation. The calculation formula is as follows:

[0112]

[0113]

[0114] In this embodiment, the negative amplification constant Set as This value is used to amplify the mean squared error loss of the layer features to an magnitude that can be effectively propagated to the latent space. Combined with subsequent normalization processing, this ensures that each attack step acts on the target region with a stable intensity. If other embodied perception models or feature layers are used, this value can be adjusted according to the magnitude of the actual loss, ideally with the magnitude before gradient normalization reaching single digits to tens digits.

[0115] Step S403: Spatial mask filtering and gradient normalization

[0116] Analyze the coordinates of the target bounding box ,in, The normalized center coordinates of the target bounding box. The normalized width and height of the target bounding box; Mapping to the latent tensor resolution, we get ,in , Let x and y coordinates be the top-left corner of the target bounding box in the latent space. , These are the x and y coordinates of the bottom right corner of the target bounding box in the potential space. Within this region... Set the corresponding pixel to 1 to generate a binary space mask. Through dimensional reshaping and permutation operations, it is packaged into a serialization mask that is completely consistent with the dimensions of the current latent variables. .

[0117] Use serialization mask For the current feature gradient Perform element-wise filtering and flatten the result on the non-batch dimension. Normalization of norms:

[0118]

[0119]

[0120] in, The feature gradient is filtered element by element by element through the serialized mask, retaining only the gradient components within the region corresponding to the target bounding box; The gradient is the normalized value. To prevent small constants with a denominator of zero, this embodiment... ; This is the Hadama product operation.

[0121] Step S404: Dynamic step decay and adversarial update based on attack progress

[0122] Calculate the current time step Attack window Linear decay rate parameter within :

[0123]

[0124] Calculate dynamic attack step size :

[0125]

[0126] Normalized feature gradient after applying mask filtering With dynamic attack step size Complete the local adversarial update of the serialized latent variables at the current time step:

[0127]

[0128] Update the current latent variables Return to the diffusion denoising pipeline.

[0129] After the iteration, the VAE of the diffusion model decodes the image into a feature-level hard example image.

[0130] It is understood that step S400 of this embodiment uses the deep features of the embodied perception model as the attack anchor point, and applies feature-level adversarial gradient updates to the latent variables during the diffusion denoising process, so that the generated image constitutes an effective perturbation in the deep feature space on which the embodied perception model's decision depends, fundamentally breaking through the limitation that conventional enhancement can only change surface pixels.

[0131] After the above iterative update is completed, multiple variants are generated for each seed image. In this embodiment, a total of 100 clear hard example images are generated. The generated hard example images and their corresponding labels are mixed with 500 images from the source domain dataset to form a training set, which is used for fine-tuning the embodied perception model.

[0132] In one specific embodiment of this application, the obtained training set is used to fine-tune the embodied perception model using the stochastic gradient descent (SGD) algorithm. The momentum is set to 0.937, the initial learning rate to 0.001, and the weight decay coefficient to 0.001. The first 10 layers of the embodied perception model are frozen, and offline parameter fine-tuning is performed only on its deep feature extractor and detector head for 100 iterations. This completes the offline repair of perception blind spots, significantly enhancing the cross-domain perception capability of the embodied agent. This step mixes the generated hard examples with the source domain data for offline parameter fine-tuning of the embodied perception model. While improving the target domain perception capability, it maintains the overall generalization of the embodied perception model, avoiding the catastrophic forgetting problem caused by continuous parameter updates in adaptive methods during online testing.

[0133] Experiments show that using only 25 seed images, with a labeling amount only one-quarter that of fully supervised methods, the method of this invention achieves a mean average accuracy (mAP) of 70.9% in a kitchen scene, a 3.2 percentage point improvement over standard augmentation methods without gradient guidance. The effect is particularly outstanding in difficult categories, with improvements from 21.4% to 78.2% for trash cans and from 37.8% to 86.0% for indoor plants. In zero-shot generalization tests, the method of this invention even exceeds fine-tuning with full real-image annotations by 5 to 7 percentage points. Compared to existing methods that can only perform surface pixel transformations, this step directly perturbs the decision space of the embodied perception model through feature-level adversarial gradients, fundamentally breaking through the performance ceiling of conventional augmentations.

[0134] See Figure 2 A generative data augmentation apparatus for embodied perception model learning, provided in a second aspect embodiment of the present invention, includes:

[0135] The prior evaluation module is configured to evaluate the average accuracy of the embodied perception model for each instance category on the source domain validation set, and calculate the category prior weights corresponding to each category based on the average accuracy.

[0136] The difficult instance mining module is configured to, in the target domain environment to be perceived, use an embodied robot to collect unlabeled observation images, calculate the instance uncertainty of all detected instances in each observation image based on the target detection results of the observation images by the embodied perception model, calculate the difficulty score of each observation image based on the instance uncertainty and the category prior weight of the corresponding category, select a number of seed images from all observation images according to the difficulty score, and label the seed images to obtain the true bounding boxes of the instances.

[0137] The constraint extraction module is configured to extract physical boundary constraints and logical boundary constraints from the seed image respectively, and decouple the extracted logical boundary constraints into a long text description containing scene description and a short text containing only the real category.

[0138] The adversarial generation module is configured to extract the depth features of the seed image through the intermediate layer of the embodied perception model as static feature anchors; input the seed image with real bounding boxes, physical boundary constraints, long text, and short text into a diffusion model with a control net for denoising generation; calculate the feature interference loss between the current feature and the static feature anchor during denoising; update the latent variables based on the feature interference loss; return the updated latent variables to the diffusion model; finally obtain a clear hard example image; and mix the hard example image and the source domain training set with the learning of the embodied perception model.

[0139] It should be noted that the foregoing explanation of the embodiments of the generative data augmentation method for embodied perception model learning also applies to the generative data augmentation device for embodied perception model learning in this embodiment, and will not be repeated here.

[0140] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0141] Although embodiments of this disclosure have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A generative data augmentation method with body perception model learning, characterized in that, include: Evaluate the average accuracy of the embodied perception model for each instance category on the source domain validation set, and calculate the category prior weights corresponding to each category based on the average accuracy. In the target domain environment to be perceived, an embodied robot is used to collect unlabeled observation images. Based on the target detection results of the observation images by the embodied perception model, the instance uncertainty of all detected instances in each observation image is calculated. Based on the instance uncertainty and the category prior weight of the corresponding category, the difficulty score of each observation image is calculated. Based on the difficulty score, several seed images are selected from all observation images. The seed images are labeled to obtain the true bounding boxes of the instances. Physical boundary constraints and logical boundary constraints are extracted from the seed images respectively. The extracted logical boundary constraints are decoupled into a long text description containing scene description and a short text containing only the real category. The depth features of the seed image are extracted through the intermediate layer of the embodied perception model and used as static feature anchors. The seed image with real bounding boxes, physical boundary constraints, long text, and short text are input into a diffusion model with a control net for denoising generation. During the denoising process, the feature interference loss between the current feature and the static feature anchor is calculated. The latent variables are updated based on the feature interference loss. The updated latent variables are returned to the diffusion model to finally obtain a clear hard example image. The hard example image and the source domain training set are mixed to learn the embodied perception model.

2. The generative data augmentation method of claim 1, wherein, The formula for calculating the prior weights of the categories is: wherein, is an instance class is a corresponding class prior weight; is an instance class evaluated on the source domain validation set for the embodied perception model is an average precision of the instance class is a floor weight threshold.

3. The generative data augmentation method of claim 1, wherein, The difficulty score of the observed image is calculated according to the following formula: in, For observation images Difficulty score; For observation images The collection of detection instances within; For the first Instance uncertainty of individual instances; The embodied perception model's response to the input observed image The extracted first Instance detection box Corresponding instance category The prediction confidence level; According to the first The prior weight of the class corresponding to the predicted instance class of each detection box; For the first Difficulty score for each instance; If the embodied perception model detects any instances in the input observation image If no instances are detected, a severe perception blind spot is determined and a difficulty score is assigned to the observation image .​ 4. The generative data augmentation method of claim 1, wherein, Based on the difficulty score, a number of seed images are selected from all observed images, including: Sort all the observation images by difficulty score in descending order, and select the top K observation images as seed images.

5. The generative data augmentation method of claim 1, wherein, Extracting physical boundary constraints from the seed image includes: The seed image is input into the segmentation model, and the obtained segmentation result is subjected to edge capture and texture noise filtering to obtain a clean edge map that preserves the complete contour of the target, which serves as the physical boundary constraint. Extracting logical boundary constraints from the seed image includes: The seed image is input into a large language vision model, which combines the constraints of the annotation list on the seed image to generate a long text describing the scene material and lighting, while constructing a short text based on the target category extracted from the real bounding box.

6. The generative data augmentation method of claim 1, wherein, The denoising process calculates the feature interference loss between the current feature and the static feature anchor point, and updates the latent variables based on the feature interference loss, including: Within the defined iterative attack window, perform the following operations at each time step: Decode the latent variables of the current time step into an image tensor; The image tensor is input into the embodied perception model to extract the current features at the same layer as the static feature anchor, the feature interference loss between the current features and the static feature anchor is calculated, and the gradient of the current features is obtained by taking the derivative with respect to the latent variables. Generate a binary spatial mask of the potential spatial scale based on the target bounding box, and use the binary spatial mask to filter and normalize the current feature gradient; Calculate the dynamically decaying attack step size and update the latent variables at the current time step by combining the normalized current feature gradient.

7. The generative data augmentation method of claim 6, wherein, The formula for calculating the current feature gradient is: wherein, is the current feature gradient; is the current latent variable in sequence format; is the current feature interference loss; is the current feature, by transforming the image tensor inputting the embodied perception model triggers a forward hook function to obtain; is the static feature anchor point; is the mean square error; is the negative amplification constant.

8. The generative data augmentation method of claim 6, wherein, The current feature gradient is filtered and normalized using the binary space mask, and the calculation formula includes: wherein, is a serialized mask, obtained by dimension reshaping and permutation operations on the binary spatial mask, is the same dimension as the current latent variable ; is the current feature gradient; is the feature gradient filtered by the serialized mask element-wise, only keeping the gradient components within the region corresponding to the target bounding box; is the normalized gradient; is a constant to prevent the denominator from being zero; is a Hadamard product operation.

9. The generative data augmentation method of claim 6, wherein, The current latent variables are updated according to the following formula: wherein, is the updated current latent variable; is the current latent variable; is the normalized gradient; is the dynamic attack step size; is the current time step is a linear decay schedule parameter within an attack window is a linear decay schedule parameter within an attack window is a preset maximum attack scaling ratio.

10. A generative data augmentation device for embodied perception model learning, characterized in that, include: The prior evaluation module is configured to evaluate the average accuracy of the embodied perception model for each instance category on the source domain validation set, and calculate the category prior weights corresponding to each category based on the average accuracy. The difficult instance mining module is configured to, in the target domain environment to be perceived, use an embodied robot to collect unlabeled observation images, calculate the instance uncertainty of all detected instances in each observation image based on the target detection results of the observation images by the embodied perception model, calculate the difficulty score of each observation image based on the instance uncertainty and the category prior weight of the corresponding category, select a number of seed images from all observation images according to the difficulty score, and label the seed images to obtain the true bounding boxes of the instances. The constraint extraction module is configured to extract physical boundary constraints and logical boundary constraints from the seed image respectively, and decouple the extracted logical boundary constraints into a long text description containing scene description and a short text containing only the real category. The adversarial generation module is configured to extract deep features of the seed image through the intermediate layer of the embodied perception model as static feature anchors; The seed image with real bounding boxes, physical boundary constraints, long text, and short text are input into a diffusion model with a control net for denoising generation. During the denoising process, the feature interference loss between the current feature and the static feature anchor is calculated. The latent variables are updated based on the feature interference loss. The updated latent variables are returned to the diffusion model to finally obtain a clear hard example image. The hard example image and the source domain training set are mixed to learn the embodied perception model.

Images