Artificial intelligence based data set self-adaptive optimization method

Abstract

The application provides an artificial intelligence-based data set adaptive optimization method, and belongs to the technical field of artificial intelligence environment simulation; firstly, a physical robot platform device is deployed to collect real world reference data sets; at the same time, the environment is reproduced in a simulation engine to generate an initial simulation data set with initial guess parameters. Subsequently, a set of parallel multi-dimensional simulation-reality gap quantifiers is constructed to comprehensively quantify the gap between simulation and reality from multiple dimensions. The quantified gap score is accurately attributed to specific rendering parameters, physical parameters or missing scene types by using dynamic evaluation and simulation defect attribution methods. Finally, based on the attribution results, a closed-loop feedback is realized through an adaptive VR environment optimization module, and a programmed content generation engine is driven to supplement the missing rare scenes, thereby finally generating an optimized simulation data set with high fidelity. The simulation environment fidelity is comprehensively quantitatively evaluated.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence environment simulation technology, and in particular relates to an adaptive optimization method for datasets based on artificial intelligence. Background Technology

[0002] In the field of artificial intelligence (AI) robotics, training reinforcement learning strategies using virtual reality (VR) simulation environments is a necessary step in research and development. However, the gap between simulation and reality, which exists between the simulation environment and the real world, is a core pain point of this technology. This gap causes robot strategies trained perfectly in simulation to perform poorly or even fail completely in the real world. The current technological gap is mainly manifested on two levels: Visual perception: The rendering effects of object materials, textures, and lighting reflections in the simulated environment differ statistically from images captured by RGB cameras or depth cameras in the real world. This causes models that rely solely on vision to fail to generalize when transferred to reality.

[0003] Physical Interaction: Physical parameters in the simulation engine, such as the coefficient of friction, mass, inertia tensor, and elastic recoil coefficient, do not conform to the physical laws of the real world. This causes the precise interaction strategies learned by the robot in the simulation, such as grasping and pushing, to fail in reality due to the mismatch of the dynamic model.

[0004] Existing simulation-reality integration methods mainly include the following: Visual domain randomization-based methods force AI models to learn features robust to visual differences by randomizing rendering parameters such as texture, lighting, and camera position in a simulated environment. This improves the model's generalization ability to visual variations. However, this method typically neglects the fidelity of physical interactions and cannot address the failure of interaction strategies caused by physical parameter mismatches.

[0005] Physical parameter identification-based methods focus on identifying and calibrating physical parameters in simulation engines to align them with the real world. Their advantage lies in enhancing the realism of physical interactions. However, they fail to bridge the visual perception gap and typically rely on manual or specific calibration procedures, making it difficult to automate the combined visual and physical impacts.

[0006] The manual tuning and evaluation method relies on domain experts manually adjusting the rendering and physical parameters of the simulation environment and subjectively evaluating fidelity through observation. Its advantage is that it may be effective in specific, simple scenarios. Its disadvantages are that the process is highly dependent on expert experience, time-consuming and labor-intensive, lacks objective and quantifiable evaluation standards, and is difficult to automate closed-loop optimization.

[0007] In summary, existing methods often focus only on a single aspect, either visual or physical, or rely on manual adjustments, lacking a comprehensive solution that can simultaneously automate the evaluation and optimization of both visual and physical fidelity. Summary of the Invention

[0008] To address the above problems, this invention proposes an adaptive optimization method for datasets based on artificial intelligence, comprising the following steps: S1 involves deploying a physical robot platform in diverse real-world environments and collecting its visual perception and physical interaction data as a real-world reference dataset; then, a digital model of the robot and its environment is constructed in a virtual reality simulation engine to generate a set of unoptimized initial simulation data as an initial simulation dataset. S2, construct and train a visual perception similarity quantizer, a physical interaction similarity quantizer, and a scene coverage diversity quantizer, and use a real-world reference dataset as a benchmark to automatically perform multi-dimensional quantitative scoring on the fidelity of the initial simulation dataset. S3, based on the three quantizers of S2, establishes a mapping relationship between the quantized gap score and various simulation parameters in the initial simulation dataset. It decomposes and attributes the gap values ​​output by the three quantizers to the simulation parameters that cause the gap, and obtains a visual attribution heatmap, a list of physical defect parameters, and a rare scene description image. S4, based on the results of S3, adaptively adjusts the rendering parameters and physical parameter weights of the virtual reality simulation engine, and generates rare scenes in a targeted manner, ultimately generating a high-fidelity optimized simulation dataset.

[0009] Preferably, the visual perception similarity quantifier specifically comprises: A convolutional neural network is constructed as a visual perception similarity measurer. Its structure consists of M convolutional layers, activation layers, pooling layers, and fully connected layers at the end. Finally, a scalar value between 0 and 1 is output through a sigmoid activation function. The model training input is a dataset mixed in a 1:1 ratio, which includes a real-world reference dataset and an initial simulation dataset. The real-world visual data in the real-world reference dataset is labeled 1, and the simulation visual data in the initial simulation dataset is labeled 0. This trains a domain discriminator model that can distinguish between real and simulation images. A batch of simulation visual data from the initial simulation dataset is input, and a visual similarity score is output. This score is the average probability that the domain discriminator model incorrectly identifies a simulation image as real, i.e., outputs 1. The closer the score is to 0.5, the higher the visual similarity between the simulation and the real image.

[0010] Preferably, the physical interaction similarity quantifier specifically comprises: Two identical dynamic models, a real dynamic model and a simulated dynamic model, are constructed. Specifically, a feedforward neural network is constructed, which contains N fully connected layers and ReLU activation layers. The real dynamics model: The model training input is real-world interaction data from a real-world reference dataset, set as a state. and actions The output is the corresponding state at the next time step. The model is trained through supervised learning to minimize the mean squared error between the predicted state and the true state. The simulation dynamics model: The model training input is the initial simulation interaction data in the initial simulation dataset, which is set as the state. and actions The output is the corresponding state at the next time step. After the model training is completed, a set of test action sequences containing real-world action and state sequences reserved from S1 is input into the trained real dynamics model and the simulated dynamics model. The output is a physics gap score, which is calculated by comparing the mean square error between the real dynamics model and the simulated dynamics model for predicting the state at the next moment on the set of test action sequences. The larger the error, the greater the gap between simulated physics and real physics.

[0011] Preferably, the scene coverage diversity quantizer is specifically: A variational autoencoder model is employed, consisting of a convolutional encoder and a deconvolutional decoder. The convolutional encoder comprises P stacked 2D convolutional layers and ReLU activation layers, using strided convolutions to progressively reduce the spatial resolution of the feature map, and finally connecting to two independent fully connected layers, which are used to output the mean vector and log-variance vector of the latent space vector, respectively. The deconvolutional decoder comprises multiple stacked 2D transposed convolutional layers and ReLU activation layers, with its structure reversed from that of the encoder, and is responsible for progressively upsampling from the latent space vector to reconstruct the original image resolution. The encoder is responsible for compressing the input image into a low-dimensional latent space distribution, which is defined by the mean vector and the log-variance vector. This latent space distribution is determined to be a multidimensional Gaussian distribution, with its center determined by the mean vector, and its variance in each latent dimension determined by the exponential operation of the corresponding elements of the log-variance vector. The variational autoencoder model uses all real-world visual data in the real-world reference dataset as training input. The training objective is to minimize the reconstruction loss, which is the sum of the mean square error between the decoder reconstructed image and the original input image and the KL divergence in the latent space, and optimize its lower bound of evidence. The model inputs real-world visual data and simulated visual data into the trained feature encoder model and outputs a diversity gap score.

[0012] Preferably, the visual attribution heatmap is obtained through visual defect attribution, specifically as follows: A gradient-weighted class activation mapping method is used to locate specific image regions that result in low visual similarity scores. For images judged by the trained domain discriminator model, if the domain discriminator model determines that the image is a simulation image, the gradient of the feature map output by the last convolutional layer of the model is calculated. This gradient information is used to generate a weighted activation map as a visual attribution heatmap. This heatmap accurately displays the pixel regions in the input image that contribute the most to the domain discriminator model's judgment as false in a highlighted form. These regions are the most distorted parts of the simulation rendering.

[0013] Preferably, the list of physical defect parameters is obtained through physical defect attribution, specifically as follows: A parameter sensitivity analysis method is employed to identify the simulation physics parameters that contribute most to the physical interaction gap. Inputs include the physical gap score calculated by the physical interaction similarity quantifier as the objective function, and the initial simulation object attribute data from the initial simulation dataset. First, based on the initial simulation object attribute data, a small, deterministic perturbation is applied to each physics parameter in the list. Second, after each perturbation, the system reruns the complete evaluation process of the physical interaction similarity quantifier and calculates the change in the physical gap score, thereby obtaining the numerical partial derivative of the score with respect to this specific parameter. Finally, a list of physical defect parameters is output, sorted from high to low by sensitivity (i.e., the absolute value of the partial derivative), indicating the simulations that contribute most to the simulation-reality physical gap.

[0014] Preferably, the rare scene description image is obtained through diversity defect attribution, specifically as follows: First, a Gaussian mixture model is constructed based on the set of mean vectors generated by the scene coverage diversity quantizer for the simulated visual data. This Gaussian mixture model learns the distribution of the mean vectors in the simulated world, representing the probability distribution of scenes that can be generated in the simulated environment. Second, each mean vector generated by the scene coverage diversity quantizer for the real-world visual data is input into the Gaussian mixture model, and a likelihood score is calculated for each real-world mean vector. Real-world mean vectors with a likelihood score lower than a preset threshold are identified as rare or uncovered samples, representing scenes that exist in the real world but are considered highly unlikely to be generated by the simulation probability model. Finally, one or more representative vectors are sampled from the identified mean vectors with low likelihood scores to output a set of rare scene description images.

[0015] Preferably, the rendering parameters of the virtual reality simulation engine are adaptively adjusted, specifically as follows: A Bayesian optimizer is employed, which internally uses a Gaussian process as a surrogate model. The inputs include the visual similarity score calculated by the visual similarity quantifier as the objective function to be maximized, and the visual attribution heatmap obtained by S3. This heatmap is used to define a local rendering parameter search space, which specifically includes the rendering parameters in the simulation engine corresponding to the highlighted areas of the heatmap, including the metallicity and roughness parameters of the material, and the filtering radius of the shadow. The Bayesian optimizer iteratively searches within this search space to find the parameter settings that maximize the visual similarity score, and finally outputs a set of optimized rendering parameter combinations.

[0016] Preferably, the physical parameters are adaptively adjusted, specifically as follows: An independent Bayesian optimizer is employed, which internally uses a Gaussian process as a surrogate model. The input includes the physical gap score calculated by the physical interaction similarity quantifier as the objective function to be minimized, and a list of physical defect parameters output by S3. This list is used to define the parameter search space to be optimized, and the optimizer will focus on searching for the physical parameters with the highest sensitivity in the list. The Bayesian optimizer iteratively adjusts these parameters and re-evaluates the physical gap score until it finds the parameter settings that minimize the physical gap score. Finally, a set of optimized physical parameter combinations is output.

[0017] Preferably, the rare scene targeted generation specifically includes: The rare scene-oriented generation model architecture comprises a convolutional neural network backbone and a multilayer perceptron regression head. The output layer of the regression head is designed to output a set of continuous scene parameter values. It employs a procedural content generation engine built into a virtual reality simulation engine. The input consists of rare scene description images obtained from S3, which are parsed into a set of scene generation instructions by a pre-trained scene parameter inverse estimation network. The model training data is a set of pre-generated data pairs containing a one-to-one correspondence between parameters and images. These data pairs are obtained by generating a massive number of images with random parameters through the procedural content generation engine. During training, the images generated by the procedural content generation engine are used as input, and the corresponding parameters used to generate the images are used as supervision labels. The engine performs a targeted generation task and outputs a supplementary dataset of rare scenes to enhance diversity.

[0018] Preferably, a high-fidelity optimized simulation dataset is generated, and the specific process is as follows: The adjusted and optimized combination of rendering parameters and physical parameters are applied to the NVIDIA Isaac Sim virtual reality simulation engine, updating its internal configuration to a high-fidelity state. Subsequently, the initial simulation dataset and the output rare scene supplementary dataset are merged at the data level. The final output is a structured database, serving as a high-fidelity optimized simulation dataset.

[0019] Compared with the prior art, the present invention has the following beneficial effects: (1) Parallel multi-dimensional gap quantization model: This invention abandons the single-dimensional evaluation and innovatively constructs four parallel gap quantizers, including a visual quantizer based on domain discriminator CNN, a physical quantizer based on feedforward neural network dynamics model, a diversity quantizer based on the potential spatial distance of variational autoencoder, and a task completeness quantizer based on the performance of downstream grasping strategy network, realizing a comprehensive quantitative evaluation of the fidelity of the simulation environment.

[0020] (2) Automated multimodal defect attribution: This invention proposes a defect attribution mechanism. It uses gradient weighted class activation mapping technology to generate a visual attribution heatmap to accurately locate the image region with rendering distortion; at the same time, it uses parameter sensitivity analysis to automatically identify the key physical parameters that contribute the most to the physical gap; and it uses the probability density analysis of variational autoencoder and the decoder reconstruction to visualize the rare scenes missing in the simulation environment.

[0021] (3) Attribution-based closed-loop adaptive optimization: This invention uses attribution results to guide the optimization process. For visual defects, the Bayesian optimizer searches within the rendering parameter space defined by the attribution heatmap to maximize visual similarity; for physical defects, the Bayesian optimizer searches within the physical parameter list space with the highest sensitivity to minimize physical difference scores; for diversity defects, the inverse estimation network and programmatic content generation engine are used to generate the identified rare scenes in a targeted manner, forming an automated closed-loop optimization system. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating the overall implementation logic of the present invention.

[0023] Figure 2 This is a flowchart of the modal reference data and initial simulation data acquisition module of the present invention.

[0024] Figure 3 This is a schematic diagram of the visual perception similarity quantifier of the present invention.

[0025] Figure 4 This is a schematic diagram of the physical interaction similarity quantifier of the present invention.

[0026] Figure 5 This is a schematic diagram of the scenario coverage diversity quantifier of the present invention.

[0027] Figure 6 This is a schematic diagram of the dynamic evaluation and simulation defect attribution module of the present invention.

[0028] Figure 7 This is a schematic diagram of the adaptive VR environment optimization module based on gap feedback of the present invention.

[0029] Figure 8 This is a schematic diagram of the task performance completeness quantizer of the present invention.

[0030] Figure 9 This is a diagram showing the potential spatial distribution and diversity differences of the variational autoencoder in an embodiment of the present invention.

[0031] Figure 10 This is a heatmap for sensitivity analysis of physical defect attribution parameters in an embodiment of the present invention. Detailed Implementation

[0032] This invention provides a method for dynamic quality assessment and adaptive optimization of high-quality datasets based on artificial intelligence, such as... Figure 1 As shown, its main process is as follows: S1 involves deploying a physical robot platform in diverse real-world environments and collecting its visual perception and physical interaction data as a real-world reference dataset; then, a digital model of the robot and its environment is constructed in a virtual reality simulation engine to generate a set of unoptimized initial simulation data as an initial simulation dataset. S2, construct and train a visual perception similarity quantizer, a physical interaction similarity quantizer, and a scene coverage diversity quantizer, and use a real-world reference dataset as a benchmark to automatically perform multi-dimensional quantitative scoring on the fidelity of the initial simulation dataset. S3, based on the three quantizers of S2, establishes a mapping relationship between the quantized gap score and various simulation parameters in the initial simulation dataset. It decomposes and attributes the gap values ​​output by the three quantizers to the simulation parameters that cause the gap, and obtains a visual attribution heatmap, a list of physical defect parameters, and a rare scene description image. S4, based on the results of S3, adaptively adjusts the rendering parameters and physical parameter weights of the virtual reality simulation engine, and generates rare scenes in a targeted manner, ultimately producing a high-fidelity optimized simulation dataset. A task performance completeness quantizer is constructed to evaluate the quality of the high-fidelity optimized simulation dataset.

[0033] The invention will be further described below with reference to specific embodiments.

[0034] S1. Multimodal Reference Data and Initial Simulation Data Acquisition: This step is the data foundation construction stage of the method of this invention. Its core objective is to establish the following two datasets: a real-world reference dataset as a real-world benchmark, and an initial simulation dataset as the starting point for evaluation and optimization. This step firstly involves deploying a physical robot platform in diverse real-world environments to accurately collect its visual perception data and physical interaction data as a real-world reference dataset; secondly, a digital model of the robot and its environment is constructed in a virtual reality simulation engine, and a set of unoptimized initial simulation data is generated as the initial simulation dataset. The flowchart of S1 is as follows. Figure 2 As shown.

[0035] S1.1 Construction of Real-World Reference Dataset: This step involves data acquisition by deploying a robotic platform in diverse real-world indoor environments. These environments cover various lighting conditions, including bright sunlight, low evening light, and indoor lighting; they also encompass backgrounds of varying complexity, including simple desktops and cluttered environments. The robotic platform is a precisely kinematically calibrated seven-axis robotic arm, with a high-resolution RGB-D camera and a six-dimensional torque sensor rigidly fixed to its end effector flange. All sensor coordinate systems are calibrated to the robot's base coordinate system. First, real-world visual data is acquired by the RGB-D camera. This step ensures that the real-world RGB image and real-world depth image are time-stamped and pixel-space aligned at the hardware level, forming a registered RGB-D data stream. Secondly, as the robot performs a series of standard actions under control, including pushing wooden blocks of different materials at a constant speed, grasping water cups at different speeds, and releasing objects at different heights, its end effector torque sensor and joint encoder synchronously record the real-time position and velocity of each joint, the six-dimensional torque feedback of the end effector, and the synchronous action commands issued by the controller at a high sampling frequency of 1000Hz. This forms real-world interactive data that accurately reflects the robot's action commands, end effector force feedback, and actual position changes. Finally, a high-precision 3D laser scanner is used to scan each object to obtain its 3D geometric model, which will be used to define the precise collision body in the simulation. An electronic balance is used to measure its mass. The suspension method is used to determine the center of mass position of each object, and the trifilar pendulum method is used to accurately calculate its inertial tensor by measuring the oscillation period. A force gauge and an inclinometer are used to measure the static and dynamic friction coefficients between the object and a standard plane. Finally, for a rubber ball, a standard height drop test is conducted and the rebound height is measured to calculate its elastic recovery coefficient. All the above parameters, including the object's 3D geometric model, total mass, center of mass position, inertia tensor, static friction coefficient, kinetic friction coefficient, and elastic recoil coefficient, are compiled into real-world object attribute data, serving as reference ground truth values ​​for subsequent physics parameter optimization. Finally, this step organizes the real-world visual data, real-world interaction data, and real-world object attribute data into a structured database, named the Real-World Reference Dataset. All data in this database includes precise timestamps for rigorous alignment with simulation data later.

[0036] S1.2 Initial Simulation Dataset Generation: This step generates data using the NVIDIA Isaac Sim virtual reality simulation engine. The core objective of this step is to construct a simulation baseline dataset that fully corresponds to the real-world reference dataset in S1.1 in terms of data structure, format, and execution sequence. First, a virtual robot model is built in the simulation engine. This model achieves precise matching with the seven-axis robotic arm model in S1.1 in kinematic parameters, including link lengths, joint constraints, and transmission ratios, as well as basic dynamic properties, including link mass and center of mass. Second, virtual sensor models corresponding to the sensor suite in S1.1 are constructed. The model parameters of the virtual RGB-D camera, including its camera intrinsic matrix, distortion coefficients, depth measurement range, and basic noise model, are all set to be consistent with the physical camera in S1.1. The virtual six-dimensional torque sensor and virtual joint encoder are also configured to output data at a frequency of 1000Hz. Finally, this step reproduces the same set of standard motion sequences defined in S1.1 in the simulation environment. By sending a command stream to the virtual robot's controller that is identical to the synchronized motion commands recorded in the real-world interaction data of S1.1, the virtual robot is driven to perform actions such as uniform pushing, grasping, and releasing. During this process, the simulation engine synchronously generates simulation visual data containing simulation RGB images and simulation depth images; and records the initial simulation interaction data calculated by the physics engine at a frequency of 1000Hz. Finally, this step configures the initial simulation object attribute data for the interactive objects in the simulation environment, including wooden blocks, water cups, and rubber balls. The 3D geometric models of these objects directly reuse the models scanned in S1.1 to ensure consistency in collision bodies. However, all other key physical parameters, including total mass, center of mass position, inertia tensor, static friction coefficient, dynamic friction coefficient, and elastic recoil coefficient, are set to a set of unoptimized initial guesses. These guesses either come from the simulation software's default material library or from rough estimates manually annotated without precise measurements in S1.1; together, they constitute the baseline state of the simulation environment before optimization.

[0037] Finally, this step organizes the simulation visual data, initial simulation interaction data, and initial simulation object attribute data into a structured database, named the Initial Simulation Dataset. The data format, simulation timestamps, and action sequences in this database strictly correspond to the real-world reference dataset, providing paired inputs for the gap quantization in step S2.

[0038] S2. Construction of Multi-Dimensional Simulation-Reality Gap Quantification Model: This step aims to construct a set of parallel gap quantifier models. These models use the real-world reference dataset from S1.1 as a benchmark to automatically perform multi-dimensional quantitative scoring on the fidelity of the initial simulation dataset from S1.2, providing accurate numerical basis for subsequent defect attribution and optimization.

[0039] S2.1 Visual Perception Similarity Metric: This step constructs a Convolutional Neural Network (CNN) as a visual perception similarity metric. This network is designed as a domain discriminator, consisting of M convolutional layers, activation layers, pooling layers, and fully connected layers at the ends, ultimately outputting a scalar value between 0 and 1 through a sigmoid activation function. The model training input is a dataset mixed in a 1:1 ratio, containing the real-world reference dataset from S1.1 and the initial simulation dataset from S1.2. Real-world visual data from the real-world reference dataset is labeled 1, while simulated visual data from the initial simulation dataset is labeled 0. This step ultimately yields a trained domain discriminator model capable of distinguishing between real and simulated images. After training, a batch of simulated visual data from the initial simulation dataset from S1.2 is input. A visual similarity score is output. This score represents the average probability that the domain discriminator model incorrectly identifies a simulated image as real, i.e., outputs a 1. The closer the score is to 0.5, indicating near-random guessing, the higher the visual similarity between the simulation and reality. This step uses a visual perception similarity quantifier, such as... Figure 3 As shown.

[0040] S2.2 Physical Interaction Similarity Metric: This step constructs two identical real-world dynamics models and a simulated dynamics model, specifically implemented as a feedforward neural network (FFN) containing N fully connected layers and ReLU activation layers. 1. Real-world Dynamics Model: The model training input is the real-world interaction data from the S1.1 real-world reference dataset, set as the state. and actions The output is the corresponding state at the next time step. The model is trained through supervised learning to minimize the mean squared error between the predicted and actual states. 2. Simulation Dynamics Model: The model training input is the initial simulation interaction data from the S1.2 initial simulation dataset, set as the state. and actions The output is the corresponding state at the next time step. The model is trained independently in the same manner. After training, a set of test action sequences, containing real-world action and state sequences reserved in S1.1, is input into the trained real dynamics model and the simulated dynamics model, outputting a physics gap score. This score is calculated by comparing the mean squared error between the real dynamics model and the simulated dynamics model's predictions of the state at the next moment on the test action sequence set. The larger the error, the greater the gap between simulated physics and real physics. The physics interaction similarity quantifier in this step is as follows: Figure 4 As shown.

[0041] S2.3 Scene Cover Diversity Quantizer: The scene cover diversity quantizer employs a variational autoencoder (VAE) model. This model consists of a convolutional encoder and a deconvolutional decoder. The convolutional encoder contains P stacked 2D convolutional layers and ReLU activation layers, using straddle convolutions to progressively reduce the spatial resolution of the feature maps, ultimately connecting to two independent fully connected layers to output the mean vector and log-variance vector of the latent space vector, respectively. The deconvolutional decoder contains multiple stacked 2D transposed convolutional layers and ReLU activation layers, with its structure reversed from the encoder, responsible for progressively upsampling from the latent space vector to reconstruct the original image resolution. The encoder is responsible for compressing the input image into a low-dimensional latent space distribution, which is defined by the mean vector and the log-variance vector. Specifically, the latent space distribution is determined to be a multidimensional Gaussian distribution, with its center determined by the mean vector, and its variance in each latent dimension determined by the exponential operation of the corresponding elements of the log-variance vector. This variational autoencoder model uses all real-world visual data from the S1.1 real-world reference dataset as training input. The training objective is to minimize the reconstruction loss, which is the sum of the mean squared error between the decoder's reconstructed image and the original input image and the KL divergence in the latent space, optimizing its evidence lower bound. In the evaluation phase, the real-world visual data from S1.1 and the simulated visual data from S1.2 are input to the trained feature encoder model, outputting a diversity disparity score. This score is calculated through the following steps: First, a variational autoencoder model is used to transform the real-world visual data and the simulated visual data into two latent space distributions respectively; second, the mathematical distance between these two distributions is calculated, specifically using the Maximum Mean Discrepancy algorithm. This step covers scenarios such as diversity quantizers... Figure 5 As shown.

[0042] S3. Dynamic Evaluation and Simulation Defect Attribution: This step aims to establish a mapping relationship between the gap scores quantized in S2 and the simulation parameters in the initial simulation dataset of S1.2, using the quantizer models trained in S2. The core of this step is to decompose and attribute the gap values ​​output by S2 to the simulation parameters that cause the gap, thereby providing a clear and actionable optimization objective and search space for the adaptive optimization module in S4. The specific process is as follows... Figure 6 As shown.

[0043] S3.1 Visual Defect Attribution: This step employs a gradient-weighted class activation mapping method to locate specific image regions that lead to low visual similarity scores. For images judged by the domain discriminator model trained in 2.1, if the domain discriminator model determines the image to be a simulation image, the gradient of the feature map output by the last convolutional layer of the model is calculated. This gradient information is used to generate a weighted activation map as a visual attribution heatmap. This heatmap precisely highlights the pixel regions in the input image that contribute the most to the domain discriminator model's determination of a false image. These regions are the most distorted parts of the simulation rendering, specifically manifested as unrealistic metallic reflections, excessive repetition of wood textures, or incorrect rendering of shadow edges.

[0044] S3.2 Physical Defect Attribution: This step employs parameter sensitivity analysis to identify the simulation physics parameters that contribute most to the physical interaction gap. The inputs to this step include the physical gap score calculated in S2.2 as the objective function, and the initial simulation object attribute data from the initial simulation dataset in S1.2. First, using the initial simulation object attribute data as a baseline, the system applies a small, deterministic perturbation to each physics parameter in the list, specifically increasing the parameter value by one percent. Second, after each perturbation, the system reruns the complete evaluation process of S2.2 and calculates the change in the physical gap score, thus obtaining the numerical partial derivative of the score with respect to that specific parameter. Finally, the output of this step is a list of physical defect parameters. This list is sorted from highest to lowest sensitivity (i.e., the absolute value of the partial derivative), precisely identifying the simulation parameters that contribute most to the simulation-reality physical gap and clearly defining the primary cause of the physical gap.

[0045] S3.3 Diversity Deficiency Attribution: This step performs a probability density comparative analysis. First, based on the set of mean vectors generated for the simulated visual data in the S2.3 evaluation phase, this method constructs a probability density model, specifically a Gaussian mixture model. This Gaussian mixture model learns the distribution of the simulated world mean vectors, representing the probability distribution of scenes that the simulated environment can generate. Second, this method inputs each mean vector generated for the real-world visual data in the S2.3 evaluation phase into the constructed probability density model representing the simulated world, and calculates the likelihood score for each real-world mean vector. A low likelihood score indicates that the probability of this particular real-world mean vector appearing in the simulated world probability model is extremely low. That is, the scene it represents is extremely unlikely to appear in the simulated world. Those real-world mean vectors with likelihood scores below a preset threshold are identified as rare or uncovered samples. They represent scenes that exist in the real world but that the simulation probability model considers extremely unlikely to generate. Finally, one or more representative vectors are sampled from these identified mean vectors with low likelihood scores. The output of this step is a set of rare scene description images. These rare scene description images are generated by applying the deconvolutional decoder of S2.3 to the sampled low-likelihood mean vector, reconstructing it into a visual image. These reconstructed images visually demonstrate the specific scene types missing from the simulation environment.

[0046] S4. Adaptive VR Environment Optimization Based on Gap Feedback: This step is the final execution module to achieve closed-loop control. This step receives all defect attribution results from S3, including the visual attribution heatmap from S3.1, the physical defect parameter list from S3.2, and the rare scene description image from S3.3. Based on these results, it adaptively adjusts the rendering parameters, physical parameters, and scene generation weights of the virtual reality simulation engine, ultimately generating a high-fidelity optimized simulation dataset. A schematic diagram of this step is shown below. Figure 7 As shown.

[0047] S4.1 Adaptive Rendering Parameter Tuning: This step employs a Bayesian optimizer, internally using a Gaussian process as a surrogate model. The inputs to this step include the visual similarity score calculated in S2.1 as the objective function to be maximized, and the visual attribution heatmap output from S3.1. This heatmap is used to define a local rendering parameter search space, specifically including the rendering parameters in the simulation engine corresponding to the highlighted areas of the heatmap, including material metallicity, roughness parameters, and shadow filtering radii. The Bayesian optimizer iteratively searches within this search space to find the parameter settings that maximize the visual similarity score in S2.1. The output of this step is a set of optimized rendering parameter combinations.

[0048] S4.2 Adaptive Tuning of Physical Parameters: This step employs a separate Bayesian optimizer, which internally uses a Gaussian process as a surrogate model. The inputs to this step include the physical gap score calculated in S2.2 as the objective function to be minimized, and the list of physical defect parameters output in S3.2. This list is used to define the search space of the parameters to be optimized; the optimizer will focus on searching for the physical parameters with the highest sensitivity in the list. The Bayesian optimizer iteratively adjusts these parameters and re-evaluates the score in S2.2 until it finds the parameter settings that minimize the physical gap score. The output of this step is a set of optimized physical parameter combinations.

[0049] S4.3 Rare Scene Directed Generation Model: This step utilizes the procedural content generation engine built into the virtual reality simulation engine. The input to this step is the rare scene description images output from S3.3. These reconstructed images are parsed into a set of scene generation instructions by a pre-trained scene parameter inverse estimation network. The model structure includes a convolutional neural network backbone, specifically a ResNet-34 model (with the same structure as in S4.5), and a multilayer perceptron regression head. The output layer of this regression head is designed to output a continuous set of scene parameter values. The training data consists of a set of pre-generated data pairs containing a one-to-one correspondence between parameters and images. These data pairs are obtained by driving the procedural content generation engine to generate a massive number of images with random parameters. During training, the images generated by the procedural content generation engine are used as input, and the corresponding parameters used to generate those images are used as supervision labels. The parameter weighting module of the procedural content generation engine significantly increases the sampling weights of the scenes corresponding to the generated rare scene description images based on these scene generation instructions. Subsequently, the engine executes a directed generation task. The output of this step is a supplementary dataset of rare scenes to enhance diversity.

[0050] S4.4 Final Optimized Dataset Generation: This step is the final output of the method of this invention. The system first applies the optimized rendering parameter combination output from S4.1 and the optimized physical parameter combination output from S4.2 to the virtual reality simulation engine NVIDIA Isaac Sim in S1.2, updating its internal configuration to a high-fidelity state. Subsequently, the system merges the initial simulation dataset from S1.2 with the rare scene supplementary dataset output from S4.3 at the data level. The final output of this step is a structured database named the High-Fidelity Optimized Simulation Dataset. This high-fidelity optimized simulation dataset is highly consistent with the real-world reference dataset in terms of visual fidelity, physical fidelity, and scene diversity, and can be directly used to train a robot AI model with high generalization ability.

[0051] S4.5 Task Performance Completeness Quantification: This step employs a robot grasping policy network, an end-to-end deep neural network model designed to directly map visual input to the robot's grasping actions. Its model structure includes a convolutional neural network backbone, specifically a ResNet-34 model, which sequentially includes an initial convolutional layer and a max-pooling layer, followed by four residual stages composed of stacked basic residual blocks, and finally a global average pooling layer to extract high-dimensional visual feature vectors from the input visual data containing RGB and depth images. It also includes a multilayer perceptron policy head, which takes the flattened feature vectors output from the convolutional neural network backbone as input and outputs a set of continuous parameters defining the grasping actions, specifically: the target position of the robotic arm's end effector in 3D space, the pose represented by quaternions, and the gripper's opening and closing commands. During model training, this step executes two completely independent training processes with identical model structures and training hyperparameters. The first training process uses the real-world reference dataset S1.1 as input, resulting in a trained real-world robot grasping policy network, i.e., Model A. The second training process uses the initial simulation dataset S1.2 as input, resulting in a converged simulation training policy model file with its network weights saved, i.e., Model B. In the evaluation phase, this step inputs a reserved real-world validation set from the S1.1 real-world reference dataset into Models A and B. This real-world validation set contains a set of standardized grasping tasks defined in a real physical environment. The evaluation process first loads the real-world training policy, i.e., Model A, onto the physical robot platform in S1.1 and performs grasping tests on the tasks defined in the real-world validation set, calculating the grasping success rate, denoted as success rate A. Secondly, this step loads the simulation training policy, i.e., Model B, onto the same physical robot platform and performs grasping tests on the exact same real-world validation set tasks, calculating the grasping success rate, denoted as success rate B. Here, the operational definition of successful grasping is the percentage of times the robot successfully grasps the target object from its initial position, lifts it to a predetermined height, and holds it stably for three seconds without dropping it within a specified number of attempts. Finally, this step outputs a task performance gap value, which is the difference between success rate A and success rate B. The larger this gap value, the more severe the performance degradation of the strategy (model B) trained solely on simulation data in the real world, indicating a poorer task completeness of the initial simulation dataset. The task performance completeness quantification in this step is as follows: Figure 8 As shown.

[0052] To verify the effectiveness of the method proposed in this invention, this experiment uses the real-world reference dataset and the initial simulation dataset collected in step S1 as the research objects to evaluate the comprehensive performance of this invention in multi-dimensional simulation-reality gap quantification and simulation defect attribution. The core indicators of the experiment include: the visual similarity score mentioned in S2.1, the physical gap score mentioned in S2.2, and the diversity gap score mentioned in S2.3.

[0053] The experiment first verifies the ability of the S2.3 scene coverage diversity quantizer to quantize the visual diversity gap between the real world and the initial simulation environment. This experiment uses the convolutional encoder of the variational autoencoder trained in S2.3 to extract the mean vectors of the real-world visual data in S1.1 and the simulation visual data in S1.2, and then reduces them to a two-dimensional latent space for visualization.

[0054] like Figure 9 As shown, the potential spatial distribution range of real-world data is much larger than that of the initial simulation data, and the distribution centers of the two are significantly offset. This intuitively demonstrates the serious inadequacy of the initial simulation environment in terms of scene diversity. The maximum mean difference score calculated in S2.3 is significantly higher than the threshold, quantifying this gap.

[0055] Secondly, to verify the effectiveness of the S3.2 physical defect attribution identification of key defect parameters, this experiment conducted a parameter sensitivity analysis on the initial simulated object attribute data in S1.2. By applying small perturbations to each physical parameter and recalculating the physical gap score in S2.2, this experiment obtained the numerical bias of the score with respect to each parameter.

[0056] like Figure 10 The physical parameter sensitivity heatmap shown depicts the initial physical parameters in S1.2 on the horizontal axis and the sensitivity of each parameter, i.e., the absolute value of its partial derivative, on the vertical axis. Experimental results indicate that the static friction coefficient and elastic restitution coefficient are far more sensitive than parameters such as total mass or center of mass position. This demonstrates that S3.2 successfully identified the primary cause of the physical interaction gap and provided a highly relevant list of physical defect parameters and search space for the Bayesian optimizer in S4.2, avoiding ineffective searches on irrelevant parameters.

[0057] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0058] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. An artificial intelligence-based data set self-adaptive optimization method, characterized in that, The process includes the following: S1 involves deploying a physical robot platform in diverse real-world environments and collecting its visual perception and physical interaction data as a real-world reference dataset; then, a digital model of the robot and its environment is constructed in a virtual reality simulation engine to generate a set of unoptimized initial simulation data as an initial simulation dataset. S2, construct and train a visual perception similarity quantizer, a physical interaction similarity quantizer, and a scene coverage diversity quantizer, and use a real-world reference dataset as a benchmark to automatically perform multi-dimensional quantitative scoring on the fidelity of the initial simulation dataset. The scene coverage diversity quantizer is specifically: A variational autoencoder model is employed, consisting of a convolutional encoder and a deconvolutional decoder. The convolutional encoder comprises P stacked 2D convolutional layers and ReLU activation layers, using strided convolutions to progressively reduce the spatial resolution of the feature map, and finally connecting to two independent fully connected layers, which are used to output the mean vector and log-variance vector of the latent space vector, respectively. The deconvolutional decoder comprises multiple stacked 2D transposed convolutional layers and ReLU activation layers, with its structure reversed from that of the encoder, and is responsible for progressively upsampling from the latent space vector to reconstruct the original image resolution. The encoder is responsible for compressing the input image into a low-dimensional latent space distribution, which is defined by the mean vector and the log-variance vector. This latent space distribution is determined to be a multidimensional Gaussian distribution, with its center determined by the mean vector, and its variance in each latent dimension determined by the exponential operation of the corresponding elements of the log-variance vector. The variational autoencoder model uses all real-world visual data in the real-world reference dataset as training input. The training objective is to minimize the reconstruction loss, which is the sum of the mean square error between the decoder reconstructed image and the original input image and the KL divergence in the latent space, and optimize its lower bound of evidence. The real-world visual data and simulated visual data are input into the trained feature encoder model, and a diversity gap score is output. S3, based on the three quantizers of S2, establishes a mapping relationship between the quantized gap score and various simulation parameters in the initial simulation dataset. It decomposes and attributes the gap values ​​output by the three quantizers to the simulation parameters that cause the gap, and obtains a visual attribution heatmap, a list of physical defect parameters, and a rare scene description image. The rare scene description image is obtained through diversity defect attribution, specifically: First, a Gaussian mixture model is constructed based on the set of mean vectors generated by the scene coverage diversity quantizer for the simulated visual data. This Gaussian mixture model learns the distribution of the mean vectors in the simulated world, representing the probability distribution of scenes that can be generated in the simulated environment. Second, each mean vector generated by the scene coverage diversity quantizer for the real-world visual data is input into the Gaussian mixture model, and a likelihood score is calculated for each real-world mean vector. Real-world mean vectors with a likelihood score lower than a preset threshold are identified as rare or uncovered samples, representing scenes that exist in the real world but are considered highly unlikely to be generated by the simulation probability model. Finally, one or more representative vectors are sampled from the identified mean vectors with low likelihood scores to output a set of rare scene description images. S4, based on the results of S3, adaptively adjusts the rendering parameters and physical parameter weights of the virtual reality simulation engine, and generates rare scenes in a targeted manner, ultimately generating a high-fidelity optimized simulation dataset.

2. The method of claim 1, wherein: The visual perception similarity quantifier is specifically: Convolutional neural networks are constructed as visual perception similarity quantifiers. The structure consists of M convolutional layers, activation layers, pooling layers, and a fully connected layer at the end. Finally, a scalar value between 0 and 1 is output through a sigmoid activation function. The model training input is a dataset that is mixed in a 1:1 ratio. This dataset contains a real-world reference dataset and an initial simulation dataset. The real-world visual data in the real-world reference dataset is assigned the label 1, and the simulation visual data in the initial simulation dataset is assigned the label 0. The training results in a domain discriminator model that can distinguish between real and simulation images. Input a batch of simulated visual data from the initial simulation dataset and output a visual similarity score. This score is the average probability that the domain discriminator model incorrectly identifies the simulated image as real, i.e., outputs 1. The closer the score is to 0.5, the higher the visual similarity between the simulation and the real image.

3. The method of claim 1, wherein: The physical interaction similarity quantifier is specifically: Two identical dynamic models, a real dynamic model and a simulated dynamic model, are constructed. Specifically, a feedforward neural network is constructed, which contains N fully connected layers and ReLU activation layers. The real dynamics model: the model training input is real world interaction data in a real world reference data set, set to state and action , the output is the corresponding next time state ; the model is trained by supervised learning to minimize the mean square error of the predicted state and the real state; The simulation dynamics model: The model training input is the initial simulation interaction data in the initial simulation dataset, which is set as the state. and actions The output is the corresponding state at the next time step. After the model training is completed, a set of test action sequences containing real-world action and state sequences reserved from S1 is input into the trained real dynamics model and the simulated dynamics model, and a physical gap score is output. This score is calculated by comparing the average mean square error between the real dynamics model and the simulated dynamics model for predicting the state at the next moment on the set of test action sequences. The larger the error, the greater the gap between the simulated physics and the real physics.

4. The dataset adaptive optimization method based on artificial intelligence as described in claim 2, characterized in that: The visual attribution heatmap is obtained through visual defect attribution, specifically as follows: A gradient-weighted class activation mapping method is used to locate specific image regions that result in low visual similarity scores. For images judged by the trained domain discriminator model, if the domain discriminator model determines that the image is a simulation image, the gradient of the feature map output by the last convolutional layer of the model is calculated. This gradient information is used to generate a weighted activation map as a visual attribution heatmap. This heatmap accurately displays the pixel regions in the input image that contribute the most to the domain discriminator model's judgment as false in a highlighted form. These regions are the most distorted parts of the simulation rendering.

5. The dataset adaptive optimization method based on artificial intelligence as described in claim 3, characterized in that: The list of physical defect parameters is obtained through physical defect attribution, specifically as follows: A parameter sensitivity analysis method is used to identify the simulation physical parameters that contribute most to the physical interaction gap. The inputs include the physical gap score calculated by the physical interaction similarity quantifier as the objective function to be analyzed, and the initial simulation object attribute data in the initial simulation dataset. First, based on the initial simulation object attribute data, a small, deterministic perturbation is applied to each physical parameter in the list. Secondly, after each perturbation, the system reruns the complete evaluation process of the physical interaction similarity quantifier and calculates the change in the physical gap score, thereby obtaining the numerical partial derivative of the score with respect to this specific parameter. Finally, it outputs a list of physical defect parameters, which is sorted from high to low according to sensitivity, i.e., the absolute value of the partial derivative, indicating the simulation that contributes the most to the simulation-reality physical gap.

6. The dataset adaptive optimization method based on artificial intelligence as described in claim 4, characterized in that: The rendering parameters of the virtual reality simulation engine are adaptively adjusted, specifically as follows: A Bayesian optimizer is employed, which internally uses a Gaussian process as a surrogate model. The inputs include the visual similarity score calculated by the visual similarity quantifier as the objective function to be maximized, and the visual attribution heatmap obtained by S3. This heatmap is used to define a local rendering parameter search space, which specifically includes the rendering parameters in the simulation engine corresponding to the highlighted areas of the heatmap, including the metallicity and roughness parameters of the material, and the filtering radius of the shadow. The Bayesian optimizer iteratively searches within this search space to find the parameter settings that maximize the visual similarity score, and finally outputs a set of optimized rendering parameter combinations.

7. The dataset adaptive optimization method based on artificial intelligence as described in claim 5, characterized in that: The physical parameters are adaptively adjusted, specifically as follows: An independent Bayesian optimizer is employed, which internally uses a Gaussian process as a surrogate model. The input includes the physical gap score calculated by the physical interaction similarity quantifier as the objective function to be minimized, and a list of physical defect parameters output by S3. This list is used to define the parameter search space to be optimized, and the optimizer will focus on searching for the physical parameters with the highest sensitivity in the list. The Bayesian optimizer iteratively adjusts these parameters and re-evaluates the physical gap score until it finds the parameter settings that minimize the physical gap score. Finally, a set of optimized physical parameter combinations is output.

8. The dataset adaptive optimization method based on artificial intelligence as described in claim 1, characterized in that: The rare scene targeted generation is specifically as follows: The rare scene-oriented generation model structure includes a convolutional neural network backbone network and a multilayer perceptron regression head. The output layer of the regression head is designed to output a set of continuous scene parameter values. It utilizes the built-in procedural content generation engine of the virtual reality simulation engine; The input is rare scene description images obtained by S3, which are parsed into a set of scene generation instructions by a pre-trained scene parameter inverse estimation network; The model training data is a set of pre-generated data pairs containing a one-to-one correspondence between parameters and images. These data pairs are obtained by generating a massive number of images with random parameters through a programmable content generation engine. During training, the images generated by the programmable content generation engine are used as input, and the corresponding parameters that generate the images are used as supervision labels. The engine performs a targeted generation task and outputs a supplementary dataset of rare scenes to enhance diversity.

Images