Imperfect demonstration imitation learning method, device and storage medium

Abstract

The application discloses an imperfect demonstration imitation learning method and device and a storage medium, relates to the technical field of imitation learning, and comprises the following steps: acquiring a sparse expert data set and a suboptimal data set in an imitation learning scene; training a diffusion model based on the sparse expert data set, performing distribution-level correction on the suboptimal data set through the diffusion model, and obtaining a corrected data set; training a discriminator based on the sparse expert data set and the corrected data set, performing dynamics consistency screening on the corrected data set by using the discriminator, and obtaining a high-quality enhanced data set; and fusing the sparse expert data set and the high-quality enhanced data set, training a policy network through a behavior cloning algorithm, and outputting a target policy network suitable for the sparse expert data scene. The application realizes purification and enhancement of poor data through diffusion correction and discrimination screening, and effectively alleviates the problems of cumulative error and insufficient generalization ability caused by the scarcity of expert samples in traditional imitation learning.

Description

Technical Field

[0001] This application relates to the field of imitation learning technology, and more particularly to an imperfect demonstration imitation learning method, device, and storage medium. Background Technology

[0002] Imitation learning, as an important branch of artificial intelligence, is widely used in scenarios such as robot control, autonomous driving, and language modeling. Its core is to enable intelligent agents to learn effective strategies from expert demonstrations. Offline imitation learning further focuses on strategy learning under conditions without environmental interaction and has extremely high application value in practical scenarios such as industrial robots and medical operations.

[0003] Behavioral cloning is the most widely used method in traditional imitation learning. Traditional behavioral cloning methods rely on a large number of expert trajectories, which can easily lead to cumulative errors when the number of expert demonstrations is limited, causing the agent to deviate from the safe state. This problem is more prominent in offline environments. Existing methods that combine diffusion models need to rely on a large number of expert trajectories to model the expert distribution. When expert data is extremely scarce, the model's generalization ability drops significantly, and the reliability of the generated guidance signals is insufficient. At the same time, diffusion models are also prone to generating invalid samples that conform to statistical distributions but violate physical and dynamic constraints, further affecting the policy learning effect. Summary of the Invention

[0004] The main purpose of this application is to provide an imperfect demonstration imitation learning method, device and storage medium, which aims to effectively alleviate the problems of accumulated error and insufficient generalization ability caused by the scarcity of expert samples in traditional imitation learning when relying only on a small amount of sparse expert data and a large amount of imperfect suboptimal data.

[0005] To achieve the above objectives, this application proposes an imperfect demonstration imitation learning method, the method comprising: Obtain sparse expert datasets and suboptimal datasets in imitation learning scenarios; A diffusion model is trained based on the sparse expert dataset. The diffusion model is then used to perform distribution-level correction on the suboptimal dataset to obtain a corrected dataset. The distribution-level correction is achieved by using the diffusion model to perform forward noise addition and reverse noise removal on the suboptimal samples in the suboptimal dataset, gradually correcting the distribution of the suboptimal data to the expert manifold to obtain the corrected dataset. A discriminator is trained based on the sparse expert dataset and the calibration dataset. The discriminator is then used to perform dynamic consistency screening on the calibration dataset to obtain a high-quality augmented dataset. The dynamic consistency screening involves the discriminator performing expert behavior differentiation and dynamic verification on the calibration samples in the calibration dataset, and filtering out invalid samples that violate physical and dynamic constraints to obtain a high-quality augmented dataset. By fusing the sparse expert dataset with the high-quality augmented dataset, the policy network is trained using the behavior cloning algorithm, and a target policy network suitable for sparse expert data scenarios is output.

[0006] In one possible implementation, obtaining the sparse expert dataset and the suboptimal dataset in the imitation learning scenario includes: Acquire expert trajectory data generated by expert operations in an imitation learning scenario, and construct a sparse expert dataset based on the expert trajectory data; Acquire real suboptimal trajectory data obtained from non-expert operation records, intermediate results of strategy iteration, or noise sensor logs, and inject Gaussian noise into the expert actions in the expert trajectory data to generate simulated suboptimal trajectory data. A suboptimal dataset is constructed based on the real suboptimal trajectory data and the simulated suboptimal trajectory data.

[0007] In one possible implementation, the diffusion model trained based on the sparse expert dataset is used to perform distribution-level correction on the suboptimal dataset to obtain a corrected dataset, including: Based on the distribution characteristics of the sparse expert dataset, a correction function is constructed to correct the suboptimal distribution to the expert distribution; Using the state-action pairs in the sparse expert dataset as training samples, a diffusion model matching the correction function is trained by forward noise addition and reverse noise removal. The diffusion model is used to perform diffusion correction on the suboptimal dataset to obtain a corrected dataset that closely approximates the expert distribution.

[0008] In one possible implementation, the step of using the diffusion model to perform diffusion correction on the suboptimal dataset to obtain a corrected dataset that closely approximates the expert distribution includes: Noise is added to the suboptimal samples in the suboptimal dataset to make the noise pattern of the sample distribution overlap with that of the expert distribution. Based on the diffusion model, reverse denoising is performed on the noisy suboptimal samples to gradually correct each suboptimal sample to the expert manifold. The corrected dataset is obtained by summarizing and organizing all the suboptimal samples that have been corrected.

[0009] In one possible implementation, the discriminator is trained based on the sparse expert dataset and the calibration dataset, and the discriminator is used to perform dynamic consistency screening on the calibration dataset to obtain a high-quality augmented dataset, including: The sparse expert dataset is used as positive samples and the calibration dataset is used as negative samples to construct the discriminator training data. Based on the discriminator training data, a discriminator network is trained using a binary classification learning method to obtain a discriminator that can distinguish between expert behavior and non-expert behavior. The trained discriminator is used to perform dynamic consistency detection on the corrected dataset to select high-quality augmented datasets.

[0010] In one possible implementation, the step of using a trained discriminator to perform dynamic consistency detection on the corrected dataset to filter and obtain a high-quality augmented dataset includes: The discriminator calculates the proximity index between the corrected samples and expert behavior in the corrected dataset, thereby quantifying the degree of matching between the corrected samples and the expert distribution. The proximity index is used to filter out invalid samples that have failed correction, deviated from the expert manifold, or violated dynamic constraints. By retaining and integrating valid samples, a high-quality augmented dataset that meets the requirements of dynamic continuity is obtained.

[0011] In one possible implementation, the fusion of the sparse expert dataset and the high-quality augmented dataset, followed by training the policy network using a behavior cloning algorithm, outputs a target policy network suitable for sparse expert data scenarios, including: The sparse expert dataset and the high-quality augmented dataset are merged to construct the expanded target training dataset; With the goal of learning expert-level behavioral policies, the policy network is iteratively trained using a behavior cloning algorithm based on the target training dataset. The output of the trained policy network is used as the target policy network suitable for sparse expert data scenarios.

[0012] In one possible implementation, the method further includes: The trained target policy network is loaded into the robot control system to execute the corresponding robot control tasks; During task execution, the robot's real-time status information and action execution information are collected and input into the target policy network for inference and calculation. Based on the decision results output by the target strategy network, the robot is controlled to complete the specified operation, achieving stable operation in sparse expert data scenarios.

[0013] Furthermore, to achieve the above objectives, this application also proposes an imperfect demonstration imitation learning device, which includes: The acquisition module is used to acquire sparse expert datasets and suboptimal datasets in imitation learning scenarios. The correction module is used to train a diffusion model based on the sparse expert dataset, and to perform distribution-level correction on the suboptimal dataset through the diffusion model to obtain a corrected dataset. The distribution-level correction is performed by using the diffusion model to perform forward noise addition and reverse noise removal on the suboptimal samples in the suboptimal dataset, and gradually correcting the distribution of the suboptimal data to the expert manifold to obtain the corrected dataset. A filtering module is used to train a discriminator based on the sparse expert dataset and the calibration dataset, and to use the discriminator to perform dynamic consistency filtering on the calibration dataset to obtain a high-quality augmented dataset. The dynamic consistency filtering is performed by the discriminator to distinguish expert behavior and perform dynamic verification processing on the calibration samples in the calibration dataset, and to filter out invalid samples that violate physical dynamic constraints in order to obtain a high-quality augmented dataset. The learning module is used to fuse the sparse expert dataset with the high-quality augmented dataset, complete the policy network training through the behavior cloning algorithm, and output a target policy network suitable for sparse expert data scenarios.

[0014] Furthermore, to achieve the above objectives, this application also proposes an imperfect demonstration imitation learning device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the imperfect demonstration imitation learning method as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the imperfect demonstration imitation learning method described above.

[0016] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the imperfect demonstration imitation learning method described above.

[0017] This application provides an imperfect demonstration imitation learning method, device, and storage medium. The method acquires a sparse expert dataset and a suboptimal dataset in an imitation learning scenario, then trains a diffusion model based on the sparse expert dataset, performs distribution-level correction on the suboptimal dataset using the diffusion model to obtain a corrected dataset, trains a discriminator based on the sparse expert dataset and the corrected dataset, uses the discriminator to perform dynamic consistency screening on the corrected dataset to obtain a high-quality augmented dataset, then merges the sparse expert dataset and the high-quality augmented dataset, completes policy network training using a behavior cloning algorithm, and outputs a target policy network suitable for sparse expert data scenarios. Thus, relying only on a small amount of sparse expert data and a large amount of imperfect suboptimal data, it achieves the purification and enhancement of inferior data through diffusion correction and discriminative screening, effectively alleviating the problems of accumulated error and insufficient generalization ability caused by the scarcity of expert samples in traditional imitation learning, and significantly improving the robustness and control accuracy of the policy in offline environments. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an imperfect demonstration of the imitation learning method in Implementation Example 1 of this application. Figure 2 Performance comparison curves of various methods under different numbers of expert trajectories to provide an imperfect demonstration of the imitation learning method in this application; Figure 3 A comparative diagram showing the training time of different methods provided to imperfectly demonstrate the imitation learning method in this application; Figure 4 A schematic diagram showing the performance comparison of ablation experiments at different noise levels, providing an imperfect demonstration of the imitation learning method in this application. Figure 5 This is an imperfect illustration of the module structure of the imitation learning device in the embodiments of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the imperfect demonstration of the imitation learning method in this application embodiment.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] Imitation learning (IL) is an important branch of artificial intelligence, aiming to enable agents to learn efficient policies from expert demonstrations. It has achieved significant results in various fields such as robot control, autonomous driving, and language modeling. Among traditional imitation learning methods, behavioral cloning (BC) is widely used due to its simplicity and effectiveness. BC learns policies by minimizing the differences between experts and agents in temporal state-action pairs within a supervised learning framework.

[0024] With a large number of expert trajectories available, BC (Browser-Based Learning) performs well. However, when the number of expert demonstrations is limited, BC is often affected by accumulated errors, causing the agent to deviate from familiar or safe states. This problem is particularly pronounced in offline environments, where the agent cannot correct its behavior through interaction with the environment. Therefore, offline imitation learning algorithms, including BC, typically perform poorly when only a few expert trajectories are available.

[0025] An intuitive solution is to collect more expert demonstrations. However, in practical applications such as robot control and medicine, acquiring large amounts of high-quality expert data is often impractical or extremely costly. For example, when dealing with covariate shift, requiring continuous operator intervention and providing correction signals is usually prohibitive. Therefore, to achieve robust policy performance and help agents recover from small deviations, the key challenge lies in how to leverage expert guidance to transform imperfect demonstrations into reliable training signals, beyond the limitations of limited expert demonstrations.

[0026] To address this problem, previous studies have attempted to extract approximate expert behavioral signals from imperfect trajectories to guide agents to revert to expert behavior when deviating from the expected path. However, such methods typically require measurable proximity between expert data and imperfect data in the state space; when imperfect data deviates significantly from the expert distribution, the recovery process becomes unreliable.

[0027] Meanwhile, diffusion models have achieved significant success in generative tasks due to their powerful distribution modeling capabilities, typically generating samples with higher quality and diversity compared to variational autoencoders (VAEs). Based on this advantage, recent research has attempted to incorporate diffusion models into imitation learning processes to enhance or refine training signals. However, these methods often rely on a large number of expert trajectories to model the expert distribution; when expert data is extremely scarce (e.g., only one demonstration), the model's generalization ability significantly decreases, and the learned guidance signals become unreliable.

[0028] In summary, this application proposes a hierarchical expert-guided imitation learning framework based on uncertainty perception, which solves the problem of policy learning under limited and imperfect demonstration data through cascaded distributed-level correction and transfer-level selection.

[0029] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0030] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device, big data service platform, or imperfect demonstration imitation learning system capable of realizing the above functions. The following description uses an imperfect demonstration imitation learning system as an example to illustrate this embodiment and the subsequent embodiments.

[0031] Based on this, the embodiments of this application provide an imperfect demonstration imitation learning method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an imperfect demonstration of the imitation learning method in Implementation Example 1 of this application.

[0032] In this embodiment, the imperfect demonstration imitation learning method includes steps S11 to S14: Step S11: Obtain the sparse expert dataset and the suboptimal dataset in the imitation learning scenario; It should be noted that the imitation learning scenario refers to offline policy learning scenarios that rely on expert demonstrations, such as robot control and robotic arm operation; the sparse expert dataset refers to a small set of high-quality trajectory samples generated by human experts or optimal control algorithms; and the suboptimal dataset refers to an imperfect set of trajectories consisting of operation records from non-experts, trajectories generated by early-stage checkpoint strategies, or historical logs containing noisy sensor data. The system acquires both the sparse expert dataset and the suboptimal dataset to construct a learning condition of "a small number of experts + a large number of suboptimal datasets," addressing the real-world pain points of scarce expert data and high acquisition costs. Simultaneously, it utilizes both real suboptimal data and noise-simulated suboptimal data as dual sources to maximize the extraction of usable information.

[0033] In addition, sparse expert datasets are used to provide standard expert distributions, and suboptimal datasets are used to provide a large number of low-cost samples that can be corrected. This setup can form a learning condition of a single expert trajectory paired with a large number of suboptimal trajectories, which significantly reduces the cost of manual teaching.

[0034] Specifically, the system first acquires expert trajectory data generated by expert operations and constructs a sparse expert dataset. Then, it simultaneously acquires real suboptimal trajectory data collected from the actual environment and simulated suboptimal trajectory data generated by noise injection. Finally, it unifies the two types of suboptimal trajectories into a single suboptimal dataset. For example, in a robotic arm operation scenario, the system acquires one expert control trajectory as a sparse expert dataset, simultaneously collects ordinary personnel operation records as real suboptimal trajectories, and adds Gaussian noise to the expert actions to generate simulated suboptimal trajectories, integrating them to form a suboptimal dataset.

[0035] Step S12: A diffusion model is trained based on the sparse expert dataset. The diffusion model is used to perform distribution-level correction on the suboptimal dataset to obtain a corrected dataset. The distribution-level correction is performed by using the diffusion model to perform forward noise addition and reverse noise removal on the suboptimal samples in the suboptimal dataset, gradually correcting the distribution of the suboptimal data to the expert manifold to obtain the corrected dataset. It should be noted that the diffusion model refers to a generative model that employs forward noise addition and inverse denoising; distribution-level correction refers to the process of migrating suboptimal data distributions to expert distributions; and the corrected dataset refers to the set of trajectories that conform to the expert manifold after diffusion purification. The system trains the diffusion model and performs correction to achieve the core idea of ​​constructing a correction function to correct suboptimal distributions to expert distributions. By leveraging the powerful distribution fitting capability of the diffusion model, imperfect trajectories are purified into candidate trajectories that approximate expert levels.

[0036] Specifically, the system constructs a correction function from suboptimal distribution to expert distribution based on sparse expert dataset. It trains a diffusion model using state-action pairs as samples, then adds noise to the suboptimal samples and reverses the noise to gradually correct the samples to the expert manifold. Finally, it summarizes the data to obtain the corrected dataset. The correction function refers to the mapping relationship between suboptimal distribution and expert distribution. The role of the diffusion model is to implement the correction function, which can form a unified mechanism that combines distribution mapping and model implementation, avoiding dangling steps.

[0037] Step S13: A discriminator is trained based on the sparse expert dataset and the calibration dataset. The discriminator is used to perform dynamic consistency screening on the calibration dataset to obtain a high-quality augmented dataset. The dynamic consistency screening is performed by the discriminator to distinguish expert behavior and perform dynamic verification on the calibration samples in the calibration dataset, and to filter out invalid samples that violate physical dynamic constraints in order to obtain a high-quality augmented dataset. It should be noted that the discriminator refers to a binary classification network used to distinguish between expert and non-expert samples; dynamic consistency screening refers to the filtering operation that verifies whether the trajectory conforms to physical dynamic constraints; and high-quality augmented dataset refers to the set of high-quality trajectories after removing hallucination and toxic samples.

[0038] The system trains and filters discriminators to address the problem of diffusion models generating illusory trajectories that do not conform to dynamics. The second stage of oversight achieves a closed loop of "generation-discrimination," forming a stable and reliable data augmentation mechanism.

[0039] Specifically, the system trains a discriminator using expert data as positive samples and calibration data as negative samples. The discriminator then calculates a proximity index for the calibration samples, filtering out invalid samples that have failed calibration, deviated from the expert manifold, or violated dynamics. Finally, these invalid samples are integrated to obtain a high-quality augmented dataset. For example, the system uses expert robotic arm trajectories to train the discriminator, performs proximity scoring and dynamics verification on the calibrated trajectories, eliminates unreasonable trajectories, and retains high-quality samples to form the augmented dataset.

[0040] Step S14: Merge the sparse expert dataset with the high-quality augmented dataset, complete the policy network training through the behavior cloning algorithm, and output a target policy network suitable for sparse expert data scenarios.

[0041] It should be noted that fusion refers to merging and expanding expert data with augmented data; the behavior cloning algorithm refers to an imitation learning method under a supervised learning framework that aims to minimize the difference between the policy output action and the expert demonstration action. This is achieved by iteratively updating the policy network parameters to gradually approximate the expert behavior distribution. The specific implementation process is as follows: First, a training dataset containing expert state-action samples is constructed, with the state as input and the corresponding expert action as the label. Then, the difference between the policy network output action and the expert labeled action is calculated, using mean squared error or cross-entropy loss as the optimization objective. The policy network parameters are iteratively updated using the backpropagation algorithm. In each training round, the network weights are adjusted based on the gradient information of the loss function, gradually converging the error between the policy output action and the expert action to a preset threshold, ultimately obtaining a policy network capable of reproducing expert behavior. The policy network refers to the network that outputs decision actions; sparse expert data scenario refers to a scenario where only a small number of expert demonstrations are available; the target policy network refers to the final deployable stable policy model.

[0042] The system integrates data and trains policies with the aim of using augmented data to train more robust policies, thus overcoming the shortcomings of traditional behavior cloning, such as large cumulative errors, poor generalization, and inability to recover biases under sparse data.

[0043] Specifically, the system merges sparse expert datasets with high-quality augmented datasets into a target training dataset. With the goal of learning expert-level policies, it iteratively trains the policy network through behavior cloning, ultimately outputting a target policy network suitable for sparse expert data scenarios. For example, the system trains a behavior cloning policy by merging expert trajectories and augmented trajectories; the output target policy network can stably complete robotic arm operations with only a small number of expert demonstrations.

[0044] This embodiment establishes basic data support by acquiring sparse expert datasets and diverse suboptimal datasets. It relies on a diffusion model to accurately correct the distribution of suboptimal data to expert data, and then uses a discriminator to perform dynamic consistency screening to remove invalid samples. Finally, it integrates high-quality data and completes policy learning through a behavior cloning algorithm. This allows for the full utilization of massive, low-cost, imperfect data to complete high-quality training sample augmentation, relying only on a small number of expert demonstrations. It effectively solves the problems of large cumulative error, weak generalization ability, insufficient policy stability, and deviation from safe state in traditional offline imitation learning when expert data is scarce. It significantly improves the control accuracy, robustness, and environmental adaptability of the policy in high-noise, low-sample, and offline environments, while greatly reducing the time and economic costs of expert teaching.

[0045] In one feasible implementation, obtaining the sparse expert dataset and the suboptimal dataset in the imitation learning scenario includes: Step S21: Obtain expert trajectory data generated by expert operations in the imitation learning scenario, and construct a sparse expert dataset based on the expert trajectory data; It should be noted that imitation learning scenarios refer to application scenarios in which intelligent agents learn from the behavior demonstrated by experts to achieve autonomous decision-making and control tasks; expert trajectory data refers to high-quality behavioral data containing continuous state and action sequences generated by professional operators or optimal control algorithms.

[0046] The system acquires expert trajectory data and constructs a sparse expert dataset, which provides a standard behavioral reference for subsequent model training, ensuring that policy learning always takes optimal behavior as the optimization guide. This makes it more practical in scenarios where expert data acquisition is costly. In one possible implementation, the system obtains expert trajectory data by collecting expert operation logs from real physical devices, and can also generate expert trajectory data by running the optimal policy in a simulation environment.

[0047] Specifically, in a target imitation learning scenario, the system collects time-series data corresponding to the states and actions generated by expert operations to obtain expert trajectory data. This expert trajectory data is then standardized and processed to remove abnormal segments and redundant information. The processed expert trajectory data is used as the standard optimal sample to construct a sparse expert dataset. , recorded as Each trajectory contains a series of state-action pairs (s, a). For example, in a multi-degree-of-freedom robotic arm operation scenario, the system collects joint angles, end-effector positions, and control action data generated by professional operators completing precise assembly tasks, forming expert trajectory data. After smoothing and normalizing the data, a sparse expert dataset containing only a small number of high-quality samples is constructed.

[0048] Step S22: Obtain real suboptimal trajectory data collected from non-expert operation records, intermediate results of strategy iteration, or noise sensor logs, and inject Gaussian noise into the expert actions in the expert trajectory data to generate simulated suboptimal trajectory data. It should be noted that non-expert operation records refer to equipment operation behavior records generated by ordinary operators or non-professionals; intermediate results of strategy iteration refer to transitional behavioral trajectories generated when strategy training has not converged; noise sensor logs refer to biased collected data caused by sensor interference and signal offset; real suboptimal trajectory data refers to trajectory data directly collected from the actual operating environment, which has behavioral deviations but contains valid information; Gaussian noise refers to random perturbation signals that follow a Gaussian distribution; and simulated suboptimal trajectory data refers to trajectory data generated by injecting noise into expert trajectory data, used to simulate imperfect behavior.

[0049] The system simultaneously acquires real suboptimal trajectory data and generates simulated suboptimal trajectory data, which can expand the diversity and coverage of suboptimal samples, making the subsequent calibration and screening process closer to complex real-world environments and improving the generalization ability of the method. In one possible implementation, the system injects Gaussian noise step by step according to a preset noise intensity level to generate simulated suboptimal trajectory data with different degrees of deviation. In addition, the system can also determine the parameters of the Gaussian noise based on the actual noise distribution of the sensor and complete the injection operation.

[0050] Specifically, the system extracts non-expert operations, intermediate iteration results of the strategy, and sensor information affected by noise from on-site operation records, strategy training logs, and sensor data acquisition files to form real suboptimal trajectory data. .

[0051] Furthermore, to verify the effectiveness of the method, Gaussian noise can be injected into the expert actions to simulate suboptimal behavior. Assume the expert action sequence in the sparse expert dataset is... The suboptimal action is generated according to the following formula. : ,

[0052] Where t is the time step, For each time step The expert's actions This represents the noise perturbation added at time step t. This leads to a suboptimal dataset. This contains a large number of imperfect but partially useful trajectories.

[0053] For example, the system extracts the operation trajectory of ordinary personnel and the non-optimal trajectory generated by the early training strategy from the robot arm's historical logs as real suboptimal trajectory data. At the same time, it superimposes Gaussian noise of different intensities on the expert action sequence to generate simulated suboptimal trajectory data with multiple deviation levels to cover different degrees of imperfect behavior scenarios.

[0054] Step S23: Construct a suboptimal dataset based on the real suboptimal trajectory data and the simulated suboptimal trajectory data.

[0055] It should be noted that the suboptimal dataset refers to an imperfect sample set composed of both real and simulated suboptimal trajectory data, used for distribution correction and quality screening. The system integrates real-source and simulated suboptimal trajectory data to construct the suboptimal dataset, balancing the authenticity of actual data with the controllability of simulated data. This provides sufficient and diverse input samples for subsequent diffusion correction, improving data augmentation effects and the stability of policy learning.

[0056] In one possible implementation, the system mixes real suboptimal trajectory data and simulated suboptimal trajectory data according to a preset ratio. In addition, the system can also perform time-series alignment and dimensional normalization on the two types of suboptimal trajectory data before integration.

[0057] Specifically, the system performs format unification, missing value completion, and outlier removal operations on both real and simulated suboptimal trajectory data. The processed data from the two types of trajectory data are then merged and randomly shuffled to form a suboptimal dataset containing diverse imperfect samples. For example, the system performs dimensional alignment and smoothing on the collected real robotic arm suboptimal trajectories and the noise-generated simulated suboptimal trajectories, mixing them in a 1:1 ratio and shuffling the sample order to ultimately construct a suboptimal dataset with a sufficient number of samples and a rich variety of bias types.

[0058] This embodiment establishes the optimal behavioral benchmark for policy learning by constructing a sparse expert dataset containing only a small number of high-quality samples. Then, it acquires diverse suboptimal trajectory data through a combination of real-world data collection and simulated generation, fully exploring the value of low-cost and easily accessible imperfect data. Finally, it constructs a suboptimal dataset with broad coverage and rich bias types, which can significantly expand the scale of available training samples without increasing the cost of expert teaching. This effectively alleviates the problems of insufficient samples and limited distribution coverage in sparse expert data scenarios, while improving the stability, robustness, and generalization ability of subsequent data processing and policy learning processes. This enables the agent to learn near-expert-level decision-making behavior even in offline environments and under high-noise conditions.

[0059] In one feasible implementation, the diffusion model trained based on the sparse expert dataset is used to perform distribution-level correction on the suboptimal dataset to obtain a corrected dataset, including: Step S31: Based on the distribution characteristics of the sparse expert dataset, construct a correction function to correct the suboptimal distribution to the expert distribution; It should be noted that distribution characteristics refer to the probability distribution and statistical regularity of expert data in the state-action space; the correction function refers to the mapping relationship used to map and transform the probability distribution of suboptimal data into the probability distribution of expert data. The system constructs a correction function based on the distribution characteristics of the sparse expert dataset, providing a unified target and transformation rules for correcting suboptimal data, ensuring that the learning direction of the subsequent diffusion model remains consistent with the optimal behavior distribution. In one possible implementation, the system constructs the correction function based on probability density matching; in another possible implementation, the system constructs the correction function based on minimizing the distribution migration loss.

[0060] Specifically, the system performs statistical analysis on the state-action pairs in the sparse expert dataset, extracting the distribution center, dispersion, and spatial manifold characteristics of expert behaviors. Based on these distribution characteristics, a mapping relationship is established from the suboptimal distribution to the expert distribution, resulting in a correction function that can achieve distribution transfer. For example, a correction function is constructed. Correcting the suboptimal distribution to an expert distribution:

[0061] in, Represents state-action pairs. This represents a suboptimal strategy. This indicates an optimal strategy. Represents the probability distribution of the policy. Representing suboptimal datasets After correction function The corrected dataset obtained after correction.

[0062] Step S32: Using the state-action pairs in the sparse expert dataset as training samples, a diffusion model matching the correction function is trained by forward noise addition and reverse noise removal. It should be noted that state-action pairs This refers to the combined data of the agent's environmental state and actions at the same moment; training samples refer to the labeled data used for model parameter learning; forward noise addition refers to the process of gradually adding noise to the original samples to make it approach a pure noise distribution; inverse denoising refers to the process of gradually restoring effective samples from noisy samples; diffusion model refers to a generative model that achieves distribution fitting and sample generation based on the noise addition and denoising process; matching with the correction function means that the output distribution of the diffusion model is consistent with the target distribution of the correction function.

[0063] The system trains a diffusion model using state-action pairs as samples, enabling the model to learn the inherent patterns of expert distribution and synergize with the correction function to achieve accurate distribution correction. In one possible implementation, the system uses Gaussian noise as the forward noise perturbation signal; in another possible implementation, the system uses mean squared error as the optimized loss function for inverse denoising.

[0064] Specifically, the system uses the state-action combinations in the sparse expert dataset as training samples, gradually adds noise during the forward process of the diffusion model, learns to predict and remove noise during the backward process, continuously optimizes the model parameters, and makes the output distribution of the diffusion model match the expert distribution pointed to by the correction function, and finally obtains the trained diffusion model.

[0065] In one embodiment, unlike other methods that employ differentiated processing strategies for trajectories of different quality, this application proposes a unified diffusion correction optimization mechanism that uniformly performs forward noise addition and reverse noise reduction operations on all trajectories of different quality during the correction process, without the need to explicitly distinguish the trajectory quality.

[0066] Specifically, in the forward pass of the diffusion model, the expert dataset is first input. State-action pairs Gaussian noise is gradually added to the expert data until it becomes a pure noise distribution. Let the noisy sample at step t be... The calculation is as follows: ，

[0067] in, Here are the noise scheduling parameters, where , The preset variance scheduling value; It is random noise sampled from a standard normal distribution; For noisy samples at time step t; This is the original expert sample.

[0068] In addition, after performing the forward denoising, a denoising neural network is trained during the inverse denoising process. To predict the added noise, the optimization objective is to minimize the mean square error between the predicted noise and the actual noise:

[0069] in, For model training loss, Here are the parameters for the denoising network (diffusion model), and t is the diffusion time step of random sampling. The denoising neural network to be trained (i.e., the diffusion model) has the following parameters: .

[0070] Step S33: Use the diffusion model to perform diffusion correction on the suboptimal dataset to obtain a corrected dataset that closely approximates the expert distribution.

[0071] It should be noted that diffusion correction refers to the process of purifying and migrating suboptimal samples towards the expert distribution using a diffusion model; the corrected dataset refers to a high-quality set of samples that, after diffusion correction, closely approximate the expert distribution and satisfy optimal behavioral characteristics. The system utilizes a diffusion model to perform diffusion correction, which can eliminate biases and noise in suboptimal data while preserving effective behavioral information, uniformly purifying imperfect samples into candidate samples that closely approximate expert standards.

[0072] Specifically, the system inputs each sample from the suboptimal dataset into the trained diffusion model. First, it adds appropriate noise to the samples to align their distribution with the expert distribution. Then, it progressively corrects the samples to the expert manifold through inverse denoising. Finally, it aggregates and standardizes all corrected samples to obtain a calibrated dataset that closely approximates the expert distribution. For example, the system inputs a suboptimal robotic arm trajectory with deviations into the diffusion model, performs correction through partial noise addition and inverse denoising, removes noise and deviations, and aggregates the data to obtain a calibrated dataset that meets expert control standards.

[0073] This embodiment constructs a correction function based on expert distribution to clarify the correction target, trains a diffusion model matching the correction function to achieve distribution fitting, and then uses the diffusion model to complete the purification and transfer of suboptimal data. It can transform a large number of low-cost imperfect suboptimal samples into high-quality, highly consistent correction samples with only a small amount of expert data, effectively expanding the number and coverage of expert equivalent samples, improving the data quality and training stability of subsequent policy learning, and avoiding the problems of over-smoothing, dynamic violation and information loss that are prone to occur in traditional correction methods.

[0074] In one feasible implementation, the step of performing diffusion correction on the suboptimal dataset using the diffusion model to obtain a corrected dataset that closely approximates the expert distribution includes: Step S41: Perform noise addition processing on the suboptimal samples in the suboptimal dataset to make the noise pattern of the sample distribution overlap with that of the expert distribution. It should be noted that a suboptimal sample refers to a single trajectory unit in a suboptimal dataset that contains both state and action information; noise addition refers to injecting random perturbation signals that conform to a preset distribution rule into the suboptimal sample; sample distribution refers to the probability distribution pattern of the suboptimal sample in the state-action space; expert distribution refers to the optimal behavior probability distribution represented by a sparse expert dataset; and noise pattern overlap refers to the fact that the distribution characteristics of the suboptimal sample and the expert sample tend to be consistent in the noise space after noise addition.

[0075] The system adds noise to suboptimal samples, which unifies the distribution representation of suboptimal samples and expert samples, providing an alignment basis for subsequent reverse denoising correction and avoiding correction failure due to excessive differences in the original distribution. In one possible implementation, the system adaptively determines the noise intensity based on the noise characteristics of the expert distribution.

[0076] Specifically, the system selects the suboptimal dataset. Each suboptimal sample The noise is added according to the noise intensity and distribution form that matches the expert distribution, and the suboptimal samples are mapped to the intermediate noise space so that the distribution of the suboptimal samples after noise addition is consistent with the distribution form of the expert samples in the same noise space.

[0077] For example, for suboptimal samples Perform forward noise addition up to a certain intermediate step. This causes its distribution to overlap with the noise pattern of the expert distribution:

[0078] This is a suboptimal sample (to be corrected). Add noise to the intermediate steps (to make the distribution overlap with the expert's). Add noise to the suboptimal sample to the first step, It is random noise sampled from a standard normal distribution.

[0079] Step S42: Based on the diffusion model, perform reverse denoising processing on the noisy suboptimal samples to gradually correct each suboptimal sample to the expert manifold. It should be noted that reverse denoising refers to the process of gradually removing perturbation signals from noisy samples and restoring effective features; the suboptimal sample after adding noise refers to the imperfect trajectory unit after noise processing; and the expert manifold refers to the optimal decision surface and constraint space followed by expert behavior in the state-action space. The system utilizes a diffusion model to perform reverse denoising, which can gradually pull suboptimal samples back to the expert behavior manifold while maintaining dynamic continuity, achieving a precise conversion from biased behavior to optimal behavior. In one possible implementation, the system uses a multi-step iterative approach to complete reverse denoising; in another possible implementation, the system dynamically adjusts the correction step size according to the denoising progress.

[0080] Specifically, the system inputs the suboptimal samples with added noise into the trained diffusion model. The diffusion model then gradually predicts and removes the injected noise, continuously bringing the sample features closer to the expert distribution, and finally stabilizing each suboptimal sample within the range of the expert manifold.

[0081] In one embodiment, from The first step involves using a pre-trained expert denoising model. Perform reverse sampling to gradually remove noise and obtain the corrected sample. (Right now The recursive formula for the inverse denoising of the diffusion model (based on the output samples) is as follows:

[0082] in, Suboptimal sample In the reverse denoising process The state of the step is iterated until i=1, at which point the corrected clean sample is obtained. , The final corrected samples (Corrected Transition) result in the corrected dataset. ; Suboptimal sample The state after adding noise to the i-th step is the input for this denoising process; This is the cumulative noise figure; The noise figure is for a single step. For noise variance scheduling; For standard Gaussian noise, the random term used in sampling: when hour, This ensures the randomness of sampling; when hour, =0, thus obtaining the final noise-free sample. .

[0083] For example, the system uses a diffusion model to perform multi-step reverse denoising on the suboptimal samples of the noisy robotic arm, gradually eliminating the bias and correcting the samples to a manifold space that conforms to the expert control law.

[0084] Step S43: Summarize and organize all the suboptimal samples that have been calibrated to obtain the calibrated dataset.

[0085] It should be noted that the suboptimal samples after calibration refer to the high-quality samples that conform to the expert distribution and dynamic constraints after noise addition and reverse denoising; the summary and sorting refers to the operation of uniformly collecting, format standardizing and removing redundancy of all calibration samples; the calibration dataset refers to a high-quality sample set composed of effective calibration samples that can be directly used for discrimination screening and strategy training.

[0086] The system summarizes and organizes the calibrated samples to form a unified, standardized, and high-quality candidate dataset, ensuring consistency and reliability in subsequent data use.

[0087] Specifically, the system collects all valid suboptimal samples after diffusion correction, performs format standardization, redundancy cleanup and anomaly verification on the samples, and completes regularization and combination according to time sequence and state sequence to finally form a complete and high-quality correction dataset.

[0088] This embodiment first performs noise processing on suboptimal samples to align them with the noise pattern of the expert distribution. Then, it uses a diffusion model to perform reverse denoising to accurately correct the samples to the expert manifold. Finally, it summarizes and organizes all valid corrected samples to form a standardized corrected dataset. It can efficiently and stably transform a large number of imperfect suboptimal samples into high-quality data that closely matches expert standards, relying only on a small amount of expert data. This significantly expands the scale and coverage of available training samples, improves the consistency and rationality of data distribution, and effectively avoids problems such as trajectory breakage, dynamic violation, or feature distortion during the correction process. It provides solid and reliable data support for subsequent discrimination, screening, and strategy learning.

[0089] In one feasible implementation, the discriminator trained based on the sparse expert dataset and the calibration dataset, and the discriminator used to perform dynamic consistency screening on the calibration dataset to obtain a high-quality augmented dataset, includes: Step S51: Construct discriminator training data using the sparse expert dataset as positive samples and the calibration dataset as negative samples; It should be noted that positive samples refer to training samples used to represent standard expert behavior; negative samples refer to training samples used to represent non-expert behavior to be distinguished; and discriminator training data refers to the complete dataset composed of both positive and negative samples used to train the discriminator network. The system uses the sparse expert dataset as positive samples and the calibration dataset as negative samples to construct the discriminator training data. This provides the discriminator network with clear criteria for behavioral distinction, enabling the discriminator to learn the essential differences between expert and non-expert behavior, ensuring the accuracy and reliability of subsequent discrimination processes. In one possible implementation, the system adaptively adjusts the ratio of positive to negative samples based on sample quality and distribution characteristics.

[0090] Specifically, the system extracts state-action pairs as positive samples from the sparse expert dataset and corresponding samples as negative samples from the calibration dataset. It then performs standardized formatting and labeling on both types of samples. The labeled positive and negative samples are combined and randomly shuffled to obtain balanced and standardized discriminator training data. For example, the system labels expert trajectory samples in a robotic arm control scenario as positive samples and diffusion-corrected trajectory samples as negative samples. By labeling and shuffling the order of the two types of samples, it constructs the discriminator training data.

[0091] Step S52: Based on the discriminator training data, a discriminator network is trained using a binary classification learning method to obtain a discriminator that can distinguish between expert behavior and non-expert behavior; It should be noted that binary classification learning refers to supervised learning aimed at distinguishing between two categories; discriminator network refers to a neural network model used to classify and score the confidence of input samples; expert behavior refers to high-quality behavior that meets optimal control criteria and physical and dynamic constraints; and non-expert behavior refers to non-optimal behavior that has biases, noise, or unreasonable dynamics.

[0092] Specifically, the system inputs the completed discriminator training data into the discriminator network, with the goal of correctly distinguishing between positive and negative samples. The system iteratively updates the parameters of the discriminator network until the classification accuracy of the discriminator network reaches a stable convergence state, thus obtaining a discriminator that can accurately identify expert behavior and non-expert behavior.

[0093] In one embodiment, the input expert dataset (Positive samples) and calibration dataset (Negative samples) Train a binary discriminator to distinguish between expert transitions and imperfect transitions, with the optimization objective being:

[0094] in, For the discriminator network, the output is the probability that the sample belongs to the expert distribution. To maximize the classification ability of the discriminator.

[0095] Furthermore, unlike the discriminators used in traditional adversarial training, the discriminator employed in this application is not directly optimized dynamically through adversarial training, but rather derived based on the analytical form of the optimal discriminator in GAN (Generative Adversarial Network). Accordingly, the optimal discriminator defined in this application... The expression is as follows:

[0096] This is the optimal discriminator (analytical form). and They are and The empirical density function.

[0097] Step S53: Use the trained discriminator to perform dynamic consistency detection on the corrected dataset and select high-quality augmented datasets.

[0098] It should be noted that dynamic consistency detection refers to the detection operation that verifies whether the behavior of samples conforms to the rules of physical dynamics, temporal continuity, and the rationality of state transitions. The system uses a discriminator to perform dynamic consistency detection on the calibration dataset, which can effectively eliminate illusory samples, calibration failure samples, and invalid samples that violate physical constraints generated during the calibration process, retaining only high-quality and reliable samples.

[0099] Specifically, since diffusion models may generate "illusionary" samples that conform to statistical distributions but violate physical dynamics, this stage requires filtering the corrected data. The system sequentially inputs each sample in the corrected dataset into the trained optimal discriminator, which then performs dynamic consistency determination and proximity scoring on the samples. Low-quality samples are filtered out based on a preset proximity threshold, while high-confidence, dynamically consistent, and effective samples are retained. These effective samples are then integrated to obtain a high-quality augmented dataset.

[0100] This embodiment constructs discriminator training data with expert samples as positive and correction samples as negative, and uses binary classification learning to train a high-precision discriminator. The discriminator is then used to perform dynamic consistency detection and high-quality sample screening, which can effectively remove unreasonable and toxic samples generated in the diffusion correction stage. This ensures that the data entering the policy learning stage meets the dual constraints of expert distribution and physical dynamics, blocking error propagation from the source and significantly improving the reliability, stability and purity of the training data. This provides high-quality data support for subsequent policy network learning, enabling the finally learned policy to have both expert-level accuracy and strong robustness.

[0101] In one feasible implementation, the step of using the trained discriminator to perform dynamic consistency detection on the corrected dataset to filter and obtain a high-quality augmented dataset includes: Step S61: Calculate the proximity index between the correction samples and expert behavior in the correction dataset using a discriminator to quantify the degree of matching between the correction samples and the expert distribution. It should be noted that the discriminator refers to a binary classification network model trained with expert samples and calibration samples; the calibration samples refer to the state-action pair trajectory units after distribution correction; expert behavior refers to the optimal decision-making behavior represented by the sparse expert dataset; the proximity index refers to a quantitative value used to measure the degree to which the calibration samples and expert behavior are indistinguishable; and the expert distribution refers to the optimal probability distribution formed by the sparse expert data in the state-action space. The system calculates the proximity index through the discriminator, which can objectively reflect the degree to which the calibration samples approximate the expert distribution in numerical form, providing a quantifiable basis for subsequent accurate screening.

[0102] Specifically, the system sequentially inputs each calibration sample in the calibration dataset into the trained discriminator, which outputs the confidence probability that the sample belongs to the expert distribution. Based on this probability, the system calculates the proximity index of the corresponding calibration sample, thereby achieving accurate quantification of the degree of matching between each calibration sample and the expert distribution.

[0103] In one embodiment, a proximity function based on an optimal discriminator is constructed. ;

[0104] The higher the value of the proximity function, the closer the sample is to indistinguishable expert behavior.

[0105] Step S62: Filter out invalid samples that have failed correction, deviated from the expert manifold, or violated dynamic constraints according to the proximity index; It should be noted that calibration failure refers to the calibration sample failing to complete an effective migration to the expert distribution; deviation from the expert manifold refers to a statistical distribution of the calibration sample that differs significantly from the expert behavior; violation of dynamic constraints refers to the calibration sample not conforming to physical rules, temporal continuity, or environmental motion laws; invalid samples refer to low-quality samples with calibration defects, distribution bias, or unreasonable dynamics.

[0106] The system filters invalid samples based on proximity metrics, eliminating hallucinatory and toxic samples generated during diffusion correction and preventing invalid data from interfering with subsequent policy learning. In one possible implementation, the system uses a preset fixed threshold for sample filtering; in another possible implementation, the system adaptively adjusts the filtering threshold based on the overall sample distribution.

[0107] Specifically, the system compares the proximity index of each calibrated sample with a preset screening threshold, and removes invalid samples that have a proximity index below the threshold, have not met the calibration standard, are far from the expert manifold, or do not conform to the dynamic continuity, while retaining only the high-quality samples that meet the requirements.

[0108] For example, setting a threshold We use this proximity function to select reliable suboptimal transitions that are statistically close to expert behavior based on the discriminator output;

[0109] The screening threshold is used to filter out samples that have failed correction or are far from the expert manifold; This is the final, reliable dataset selected.

[0110] Step S63: Retain and integrate the valid samples to obtain a high-quality augmented dataset that meets the requirements of dynamic continuity.

[0111] It should be noted that effective samples refer to high-quality calibration samples that are selected through proximity index, conform to expert distribution, and meet dynamic constraints; dynamic continuity requirements refer to constraints that ensure smooth trajectory state transitions, action changes conform to physical rules, and there are no abrupt jumps; high-quality augmentation datasets refer to a set of high-quality training samples composed of effective samples that can be used to expand the expert distribution.

[0112] The system retains and integrates valid samples, forming clean, reliable, and highly representative augmented data that closely resembles expert behavior, significantly improving the scale and quality of the training samples. In one possible implementation, the system randomly shuffles the valid samples to improve the uniformity of data distribution.

[0113] Specifically, the system summarizes, standardizes, cleans up redundancy and smooths the selected valid samples, and completes the regularization and integration according to the trajectory sequence, finally obtaining a high-quality augmented dataset that simultaneously satisfies expert distribution matching and dynamic continuity.

[0114] This embodiment quantifies the matching degree between samples and experts by calculating a proximity index based on a discriminator. Based on the index, it accurately filters out various invalid samples and standardizes and integrates the valid samples to obtain a high-quality augmented dataset. It can accurately extract high-quality data with both statistical consistency and dynamic rationality from the calibration samples, completely block the error propagation path, avoid the negative impact of unreasonable samples on policy learning, effectively expand the equivalent distribution of experts without increasing the cost of expert data, and significantly improve the stability, robustness and generalization ability of subsequent policy learning.

[0115] In one feasible implementation, the process of fusing the sparse expert dataset and the high-quality augmented dataset, training the policy network using a behavior cloning algorithm, and outputting a target policy network suitable for sparse expert data scenarios includes: Step S71: Merge the sparse expert dataset with the high-quality augmented dataset to construct the expanded target training dataset; It should be noted that the expanded target training dataset refers to a larger, more comprehensive, and more complete set of final samples that can be directly used for training policy networks. The system merges sparse expert datasets with high-quality augmented datasets, which can significantly expand the diversity and state space coverage of training data without increasing the cost of expert teaching, effectively making up for the data shortage problem caused by the scarcity of expert samples.

[0116] Specifically, the system unifies the format, aligns the temporal sequence, and normalizes the dimensions of the sparse expert dataset and the high-quality augmented dataset. It then directly merges the two datasets and randomly shuffles them to remove duplicate samples and outliers, ultimately constructing an expanded target training dataset with a larger sample size and more complete distribution. For example, the system merges a small number of expert trajectories from robotic arm control scenarios with selected high-quality augmented trajectories to form a more comprehensive target training dataset with richer movements.

[0117] Step S72: With the goal of learning expert-level behavioral policies, the policy network is iteratively trained using the behavior cloning algorithm based on the target training dataset. It should be noted that expert-level behavioral strategy refers to a decision-making strategy that can achieve the same accuracy, stability, and rationality as expert demonstrations; optimization objective refers to the optimal behavioral standard that the policy network needs to approximate during the learning process; policy network refers to a neural network model used to output decision actions based on environmental states; iterative training refers to the training process that continuously approximates expert behavior through multiple rounds of parameter updates.

[0118] The system trains behavior clones to learn expert-level behavioral policies, enabling the policy network to fully learn expert features from the expanded data, thus improving the policy's accuracy, robustness, and generalization ability. In one possible implementation, the system uses mean squared error as the loss function for behavior cloning; in another possible implementation, the system employs an early stopping strategy to prevent overfitting.

[0119] Specifically, the system inputs the target training dataset into the policy network, guided by minimizing the difference between the agent's actions and the expert's actions. It iteratively updates the network parameters through forward inference and backpropagation using the behavior cloning algorithm until the output actions of the policy network stably converge to the level of expert behavior.

[0120] In one embodiment, the final policy network is trained using the Behavior Cloning (BC) algorithm. :

[0121] This formula is the objective loss function of the behavior cloning algorithm, used to train the policy network through supervised learning. Its goal is to minimize the cross-entropy loss between the action distribution output by the policy network and the action distribution of experts in the merged training set, so that the policy network gradually approximates the expert behavior distribution.

[0122] Specifically, the meanings of each parameter in the formula are as follows: : Represents a policy network Trainable parameters; : Represents the conditional probability distribution of the policy network outputting expert demonstration action a under the environmental state s of the agent; : Represents a state-action pair in the training samples, where s is the environmental state of the agent and a is the corresponding expert demonstration action; : Represents a sparse expert dataset containing a small number of high-quality expert state-action samples; : Indicates a high-quality augmented dataset, containing valid samples obtained through diffusion correction and kinetic screening; : Represents the target training dataset after merging, which serves as the training data source for the behavior cloning algorithm; : Indicates taking the expectation of the samples on the target training dataset; : Represents the cross-entropy loss term. By minimizing this loss, the action distribution output by the policy network in state s is made as consistent as possible with the expert action distribution.

[0123] The final policy network This strategy combines expert-level precision with robustness to environmental disturbances through enhanced data.

[0124] Step S73: Output the trained policy network as the target policy network suitable for sparse expert data scenarios.

[0125] It should be noted that the trained policy network refers to a neural network model that has undergone sufficient iterative learning, has stable action outputs, and conforms to expert behavioral characteristics; sparse expert data scenarios refer to application environments where only a small number of expert demonstrations can be obtained, and a large number of expert samples cannot be collected; the target policy network refers to the final policy model that can be directly deployed to robots or control systems to perform actual tasks. The system outputs the trained policy network, which can provide high-precision, high-reliability, and high-robust decision-making capabilities for sparse expert data scenarios, enabling the agent to still achieve expert-level performance under limited expert sample conditions. In one possible implementation, the system saves the trained model in a standardized deployment format; alternatively, the system quantizes and compresses the policy network to improve deployment efficiency.

[0126] Specifically, the system exports and standardizes the policy network that converges stably and meets the accuracy requirements, and defines it as a target policy network specifically suitable for sparse expert data scenarios, which can be directly loaded into the control system to complete reasoning and decision-making.

[0127] This embodiment constructs an expanded training set by merging scarce expert data with high-quality augmented data. It uses behavior cloning to complete policy learning with expert-level behavior as the target and finally outputs a target policy network suitable for sparse expert scenarios. It can make full use of low-cost imperfect data to achieve high-quality policy learning under conditions with very few expert samples. It effectively alleviates the problems of scarce samples, large cumulative error, weak generalization ability and poor performance in offline environment in traditional imitation learning. It significantly improves the control accuracy, operation stability and environmental adaptability of the policy in high noise, few samples and complex dynamic environment.

[0128] In one feasible implementation, the method further includes: Step S81: Load the trained target policy network into the robot control system to execute the corresponding robot control task; It should be noted that the trained target policy network refers to a policy model with expert-level behavioral decision-making capabilities, trained on a sparse expert dataset and a high-quality augmented dataset; the robot control system refers to the overall hardware and software used to receive instructions, collect status data, and drive the robot to perform actions; and the robot control task refers to the specific tasks the robot needs to complete, such as trajectory tracking, precise operation, and mobile work. By loading the target policy network into the robot control system, the system can deploy the expert-level decision-making capabilities learned offline to the actual actuator, achieving a complete closed loop from model training to practical application. In one feasible implementation, the system loads the target policy network in a lightweight model format; alternatively, the system integrates the target policy network into the robot's real-time control node.

[0129] Specifically, the system performs model serialization and environment adaptation processing on the trained, converged, and validated target policy network, loads it into the decision-making and inference unit of the robot control system, completes model initialization and parameter configuration, and enables the target policy network to receive states and output control decisions to support the execution of corresponding robot control tasks. For example, the system loads the trained robotic arm operation target policy network into a six-axis robotic arm control system to perform robot control tasks such as precision assembly and object handling.

[0130] Step S82: During the task execution process, collect the robot's real-time status information and action execution information, and input them into the target policy network for inference calculation; It should be noted that real-time status information refers to the environmental and body states, such as joint angles, positions, postures, and torques, collected in real time during the robot's operation; action execution information refers to the actual control output and execution feedback of the robot at the previous moment; and reasoning calculation refers to the forward calculation process by which the target policy network outputs the optimal action decision based on the input information. By collecting real-time information and inputting it into the target policy network for reasoning, the system enables the robot to dynamically adjust its actions according to the current working conditions, ensuring the continuity, stability, and accuracy of the operation. In one feasible implementation, the system collects real-time status information through the robot's built-in sensors and supplements it with external status information through a visual feedback module.

[0131] Specifically, during the robot's task execution, the system uses sensing modules to collect real-time robot state information and action execution feedback information. After data preprocessing and format normalization, this information is input into the target policy network, which then performs forward inference calculations to obtain the optimal action decision output. For example, during the robotic arm's operation, the system collects information such as joint angles and end effector positions in real time, inputs it into the target policy network, and quickly completes decision inference.

[0132] Step S83: Control the robot to complete the specified operation according to the decision result output by the target strategy network, so as to achieve stable operation in sparse expert data scenario.

[0133] It should be noted that the decision result refers to the optimal control quantity, action sequence, or operation instruction output by the target policy network; the specified operation refers to the target action that the robot completes according to the task requirements; stable operation refers to the robot continuously, smoothly, and reliably completing the task without real-time expert intervention or online correction; and sparse expert data scenario refers to an application scenario in which policy learning is completed only by relying on a small number of expert demonstrations. The system controls the robot to execute specified operations based on the decision result, transforming offline learned expert behavior into real operational capabilities, enabling the robot to achieve expert-level operational performance even under conditions of scarce expert data.

[0134] Specifically, the system converts the decision results output by the target policy network into motion commands that the robot servo control system can recognize, driving the robot to complete specified operations such as trajectory tracking and precise manipulation. This achieves stable, efficient, and reliable operation without expert intervention or online interaction. For example, the system controls the robotic arm to perform gripping, placement, and assembly operations based on the motion decisions output by the target policy network, achieving stable operation throughout the process even with only a few expert demonstrations.

[0135] This embodiment deploys the target policy network to the robot control system, uses real-time state for online reasoning, and completes closed-loop control based on the decision results. This enables the offline-learned expert-level policies to be truly applied, allowing the robot to still have high-precision and high-robust autonomous operation capabilities under conditions of sparse expert data, high noise, and few samples. It effectively reduces the dependence on online expert intervention and large-scale teaching, and greatly improves the economy and practicality of robot deployment in complex scenarios.

[0136] To simulate the high cost of expert teaching and the difficulty of obtaining high-quality samples in industrial scenarios, this embodiment constructs an extremely sparse data test environment that combines a single expert trajectory with 1,000 suboptimal trajectories to verify whether core strategy features can be effectively extracted and high-quality distribution expansion can be achieved under the condition of insufficient expert sample coverage. As shown in Table 1 below, in 13 tasks involving complex robotic arm operations (such as Pen, Door, Hammer), our proposed method HEED (High-efficiency Expert-guided Enhanced Diffusion) is compared with several existing methods, including BCE (Behavior Cloning with Entropy Regularization), BCM (Behavior Cloning with Mixture), DWBC (Dual Weighted Behavior Cloning), DemoDICE (Demonstration Distribution Correction Estimation), ISWBC (Implicitly Self-Weighted Behavior Cloning), and ILID (Implicit Learning from Imperfect Demonstrations). Table 1

[0137] The results show that our proposed method achieves optimal performance across all tasks. For example, in the door opening task, which demands high trajectory continuity and motion accuracy, the existing best method, ISWBC, only achieves a score of 41.2, while our proposed method achieves a score of 100.2, representing an improvement of over 143%. In the walker2d continuous motion task, our proposed method achieves a score of 107.0, an improvement of approximately 83% compared to the suboptimal baseline ILID's score of 58.4. In challenging robotic arm operations such as hammer swinging, our proposed method achieves a score of 77.0, significantly higher than other baseline methods. This fully demonstrates the dual enhancement mechanism based on diffusion correction and discriminative screening, which effectively mines valuable information from suboptimal data and generates high-quality pseudo-expert data, even with only a small number of expert trajectory guides. This solves the problems of traditional methods struggling to converge and have weak generalization ability in sparse expert data scenarios, showcasing significant advantages in precision operation and complex control scenarios for industrial robots.

[0138] Meanwhile, this method can generate a large amount of high-quality equivalent expert data that conforms to dynamic constraints by using only a single expert trajectory through diffusion correction, which solves the problem that traditional methods are difficult to converge in sparse data scenarios. It is especially suitable for application scenarios such as precision manufacturing and special robots where it is difficult to collect expert data on a large scale.

[0139] To further verify the data efficiency in actual industrial deployments, this embodiment examines different numbers of expert trajectories. The performance of the strategies under different conditions is compared to reflect application scenarios ranging from single-teaching to repeated teaching, and the manual teaching cost required to achieve the target performance is evaluated. (For reference...) Figure 2 , Figure 2 The diagram illustrates the multi-task performance comparison results of the imitation learning method provided in this application, demonstrating that this embodiment further verifies HEED's overwhelming advantages in data economy and cold-start capability. Experimental results show that as the number of expert demonstration trajectories gradually increases from very few, the performance curve of HEED (red line) consistently significantly outperforms other comparative methods, including BCE, ISWBC, and ILID.

[0140] Specifically, in four challenging robotic arm operation tasks—Pen (pen spinning task), Hammer (hammer swinging task), Door (door opening task), and Relocate (moving task)—this method maintained optimal performance across various expert data volumes, significantly improving sample utilization. In the Hammer task, HEED achieved a high performance of approximately 80 points with only one expert trajectory; in contrast, state-of-the-art baseline methods (ISWBC, ILID) still hovered around 50 points even after obtaining 30 expert trajectories. This means that this invention achieved more than twice the performance of its competitors with only 1 / 30th the amount of data. In the most challenging Relocate task, existing methods almost failed to learn (scores approaching 0) with fewer than 10 expert trajectories, exhibiting a clear "cold start failure." This invention, however, began to converge rapidly with 5 data points (score of approximately 40) and achieved a high score of 60+ with 30 data points. This demonstrates that the "hierarchical enhancement mechanism" of this invention can extract effective information from a very small amount of data, breaking the dependence of complex tasks on large amounts of data. The above results show that this method can significantly reduce the number of expert teaching sessions required for industrial robot deployment, reducing the traditional requirement of dozens of teaching sessions to only 1-5 sessions to meet the usability standard. This significantly reduces the time and labor costs of manual teaching and has the value of low-cost and high-efficiency industrial applications.

[0141] Furthermore, to verify the computational overhead introduced by the two-stage architecture, this embodiment statistically analyzes the time required for each method to complete 1 million steps (1M steps) of training under the same hardware environment. (See reference...) Figure 3 .

[0142] The results show that although this method includes both a diffusion model and a discriminator module, thanks to its hierarchical decoupling architecture and efficient data filtering strategy, its runtime (approximately 45-50 minutes) is significantly shorter than existing mainstream baseline methods. Compared to DWBC (approximately 130 minutes), the training speed of this invention is nearly 3 times faster; compared to ILID (approximately 75 minutes), the speed is improved by approximately 40%. This indicates that this invention does not achieve performance gains by simply increasing computing power, but rather achieves "cost reduction and efficiency improvement" through algorithm-level optimization.

[0143] Behavioral Cloning (BC), as the most basic supervised learning method, has the simplest structure and represents the theoretical "lower limit of speed." In comparison, the runtime of this invention is only slightly longer than BC, meaning the additional computational overhead from the introduced "expert-guided correction and selection mechanism" is extremely low. Considering that HEED achieved a several-fold performance improvement over BC in the aforementioned experiments, this slight increase in computational cost is entirely acceptable and meets the industry's demand for "high-performance, cost-effective" algorithms.

[0144] In summary, the experimental results strongly demonstrate the inventiveness of this invention in terms of architectural design: this method successfully overcomes the "computationally intensive" defects that are usually associated with traditional diffusion models or two-layer optimization methods, and achieves expert-level policy performance while maintaining lightweight computational overhead, making it very suitable for deployment in embedded robot terminals with limited computing power or online learning systems that are sensitive to real-time requirements.

[0145] Furthermore, to verify the anti-interference capability under poor data conditions, this embodiment uses suboptimal trajectories contaminated with different noise intensities to simulate extreme scenarios such as sensor failure and unprofessional operation.

[0146] Specifically, this embodiment introduces noise levels of varying intensities. The suboptimal trajectory is contaminated, among which This represents a high-noise environment that significantly deviates from the true distribution, simulating a large amount of erroneous data generated by sensor malfunction or operation by unqualified personnel. As shown in Table 2, the differences in performance of different methods in noise enhancement reveal the key role of the core module (hierarchical selection mechanism) of this invention: Table 2

[0147] As can be seen, in the Ant task, as the noise increases from 0.25 to 0.6, the performance of the baseline method BCM drops drastically from 11.2 to -3.2 (complete failure / collapse), indicating its inability to handle heavily contaminated data. In contrast, the method of this invention... It still maintained a positive return of 18.1 even under extreme noise, demonstrating extremely strong system resilience.

[0148] High noise in the HalfCheetah mission ( Under the given settings, HEED (14.2) outperforms the baseline ILID (4.9) by approximately three times. This significant difference directly demonstrates that the "discriminator-based selection module" in this invention is not a simple addition, but rather plays a crucial "firewall" role—it successfully identifies and eliminates the illusionary trajectories that the diffusion model may generate, as well as toxic samples in the original data, preventing the collapse of policy learning.

[0149] The experimental results above demonstrate that the dual filtering and enhancement mechanism of this invention, consisting of "diffusion-corrected noise conversion" and "expert-guided discriminator selection," not only expands the available expert distribution but also constructs a robust safety boundary under conditions of high noise and low-quality data.

[0150] To further verify the synergistic effectiveness of the two-stage layered architecture, this embodiment sets up an ablation experiment to remove the screening module (i.e., Figure 4The w / o selection comparison in [the text] (HEED) is used to compare the performance difference between single diffusion correction and the complete architecture; see reference [the text]. Figure 4 The experimental results show that: 1. Prove the limitations of the diffusion model working alone (solving the "illusion" problem): Visible in Hammer (swinging) tasks: at noise levels Below, the performance using diffusion correction alone (yellow bars) is extremely poor, while the performance jumps by nearly 9 times after introducing the selection strategy (blue bars). This demonstrates that when dealing with complex dynamics or high-noise data, a single diffusion model, while pushing the data toward the expert distribution, often introduces a "statistically close but physically infeasible" illusory trajectory. Without the rigorous filtering of a second-stage discriminator, these erroneous correction data will directly contaminate the policy training. The "selection module" of this invention acts as a crucial "dynamic filter."

[0151] 2. Verify the robustness of the cooperative mechanism under high noise conditions (overcoming "composite error"): In the Pen and Ant tasks, as noise from... Increase to The performance of the yellow bars (no selection) often stagnates or even declines. This is because the greater the noise, the longer the reverse denoising process of the diffusion model, and the more severe the accumulated composite error. The complete architecture of this invention (blue bars) maintains a significant advantage across all noise levels. This demonstrates that the discriminator is not simply an added bonus, but rather forms a complementary closed loop with the diffusion model—diffusion is responsible for "casting a wide net" (generating diverse candidates), while the discriminator is responsible for "refining the net" (removing high-error samples); neither is dispensable.

[0152] This ablation experiment strongly demonstrates that the present invention is not a simple superposition of existing technologies, but rather proposes a collaborative optimization mechanism of "tight coupling of generation and discrimination". This mechanism successfully solves the problem of strategy instability caused by the lack of post-processing verification of the quality of generated samples in existing technologies, constituting a non-obvious but substantial improvement.

[0153] As can be seen, this method achieves the migration of suboptimal samples to the expert distribution through diffusion correction, and realizes dynamic consistency screening through the discriminator, forming a collaborative optimization mechanism with tight coupling between the generative model and the discriminative model. This not only expands the effective expert distribution, but also builds a stable data security boundary under high noise, low quality and sparse sample conditions, so that the strategy is significantly improved in terms of accuracy, stability, generalization ability and running efficiency.

[0154] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0155] This application also provides an imperfect demonstration and imitation learning device; please refer to [reference needed]. Figure 5 The imperfect demonstration imitation learning device includes: Module 51 is used to acquire sparse expert datasets and suboptimal datasets in imitation learning scenarios. The correction module 52 is used to train a diffusion model based on the sparse expert dataset, and to perform distribution-level correction on the suboptimal dataset through the diffusion model to obtain a corrected dataset. The distribution-level correction is performed by using the diffusion model to perform forward noise addition and reverse noise removal on the suboptimal samples in the suboptimal dataset, and gradually correcting the distribution of the suboptimal data to the expert manifold to obtain the corrected dataset. The filtering module 53 is used to train a discriminator based on the sparse expert dataset and the calibration dataset, and use the discriminator to perform dynamic consistency filtering on the calibration dataset to obtain a high-quality augmented dataset. The dynamic consistency filtering is performed by the discriminator to distinguish expert behavior and perform dynamic verification processing on the calibration samples in the calibration dataset, and to filter out invalid samples that violate physical dynamic constraints in order to obtain a high-quality augmented dataset. Learning module 54 is used to fuse the sparse expert dataset with the high-quality augmented dataset, complete the policy network training through the behavior cloning algorithm, and output a target policy network suitable for sparse expert data scenarios.

[0156] The imperfect demonstration imitation learning device provided in this application, employing the imperfect demonstration imitation learning method in the above embodiments, can solve the technical problems in the background art. Compared with the prior art, the beneficial effects of the imperfect demonstration imitation learning device provided in this application are the same as the beneficial effects of the imperfect demonstration imitation learning method provided in the above embodiments, and other technical features in the imperfect demonstration imitation learning device are the same as the features disclosed in the methods of the above embodiments, and will not be repeated here.

[0157] This application provides an imperfect demonstration imitation learning device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the imperfect demonstration imitation learning method in Embodiment 1 above.

[0158] The following is for reference. Figure 6The diagram illustrates a structural schematic suitable for implementing an imperfect demonstration and imitation learning device according to embodiments of this application. The imperfect demonstration and imitation learning device in embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The imperfect demonstration of the learning device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0159] like Figure 6 As shown, the imperfect demonstration and imitation learning device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the imperfect demonstration and imitation learning device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the imperfect demonstration and imitation learning device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show imperfect demonstration and imitation learning devices with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0160] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0161] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0162] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the imperfect demonstration imitation learning methods provided by the above methods.

[0163] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0164] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0165] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. An imperfect demonstration-imitation learning method, characterized in that, include: Obtain sparse expert datasets and suboptimal datasets in imitation learning scenarios; A diffusion model is trained based on the sparse expert dataset. The diffusion model is then used to perform distribution-level correction on the suboptimal dataset to obtain a corrected dataset. The distribution-level correction is achieved by using the diffusion model to perform forward noise addition and reverse noise removal on the suboptimal samples in the suboptimal dataset, gradually correcting the distribution of the suboptimal data to the expert manifold to obtain the corrected dataset. A discriminator is trained based on the sparse expert dataset and the calibration dataset. The discriminator is then used to perform dynamic consistency screening on the calibration dataset to obtain a high-quality augmented dataset. The dynamic consistency screening involves the discriminator performing expert behavior differentiation and dynamic verification on the calibration samples in the calibration dataset, and filtering out invalid samples that violate physical and dynamic constraints to obtain a high-quality augmented dataset. By fusing the sparse expert dataset with the high-quality augmented dataset, the policy network is trained using the behavior cloning algorithm, and a target policy network suitable for sparse expert data scenarios is output.

2. The imperfect demonstration imitation learning method as described in claim 1, characterized in that, The acquisition of sparse expert datasets and suboptimal datasets in the imitation learning scenario includes: Acquire expert trajectory data generated by expert operations in an imitation learning scenario, and construct a sparse expert dataset based on the expert trajectory data; Acquire real suboptimal trajectory data obtained from non-expert operation records, intermediate results of strategy iteration, or noise sensor logs, and inject Gaussian noise into the expert actions in the expert trajectory data to generate simulated suboptimal trajectory data. A suboptimal dataset is constructed based on the real suboptimal trajectory data and the simulated suboptimal trajectory data.

3. The imperfect demonstration imitation learning method as described in claim 1, characterized in that, The diffusion model trained based on the sparse expert dataset is used to perform distribution-level correction on the suboptimal dataset to obtain the corrected dataset, which includes: Based on the distribution characteristics of the sparse expert dataset, a correction function is constructed to correct the suboptimal distribution to the expert distribution; Using the state-action pairs in the sparse expert dataset as training samples, a diffusion model matching the correction function is trained by forward noise addition and reverse noise removal. The diffusion model is used to perform diffusion correction on the suboptimal dataset to obtain a corrected dataset that closely approximates the expert distribution.

4. The imperfect demonstration imitation learning method as described in claim 3, characterized in that, The process of applying diffusion correction to the suboptimal dataset using the diffusion model to obtain a corrected dataset that closely approximates the expert distribution includes: Noise is added to the suboptimal samples in the suboptimal dataset to make the noise pattern of the sample distribution overlap with that of the expert distribution. Based on the diffusion model, reverse denoising is performed on the noisy suboptimal samples to gradually correct each suboptimal sample to the expert manifold. The corrected dataset is obtained by summarizing and organizing all the suboptimal samples that have been corrected.

5. The imperfect demonstration imitation learning method as described in claim 1, characterized in that, The discriminator, trained based on the sparse expert dataset and the calibration dataset, is then used to perform dynamic consistency screening on the calibration dataset to obtain a high-quality augmented dataset, including: The sparse expert dataset is used as positive samples and the calibration dataset is used as negative samples to construct the discriminator training data. Based on the discriminator training data, a discriminator network is trained using a binary classification learning method to obtain a discriminator that can distinguish between expert behavior and non-expert behavior. The trained discriminator is used to perform dynamic consistency detection on the corrected dataset to select high-quality augmented datasets.

6. The imperfect demonstration imitation learning method as described in claim 5, characterized in that, The trained discriminator is used to perform dynamic consistency detection on the corrected dataset to filter out high-quality augmented datasets, including: The discriminator calculates the proximity index between the corrected samples and expert behavior in the corrected dataset, thereby quantifying the degree of matching between the corrected samples and the expert distribution. The proximity index is used to filter out invalid samples that have failed correction, deviated from the expert manifold, or violated dynamic constraints. By retaining and integrating valid samples, a high-quality augmented dataset that meets the requirements of dynamic continuity is obtained.

7. The imperfect demonstration imitation learning method as described in claim 1, characterized in that, The process involves fusing the sparse expert dataset and the high-quality augmented dataset, training the policy network using a behavior cloning algorithm, and outputting a target policy network suitable for sparse expert data scenarios, including: The sparse expert dataset and the high-quality augmented dataset are merged to construct the expanded target training dataset; With the goal of learning expert-level behavioral policies, the policy network is iteratively trained using a behavior cloning algorithm based on the target training dataset. The output of the trained policy network is used as the target policy network suitable for sparse expert data scenarios.

8. The imperfect demonstration imitation learning method as described in claim 1, characterized in that, The method further includes: The trained target policy network is loaded into the robot control system to execute the corresponding robot control tasks; During task execution, the robot's real-time status information and action execution information are collected and input into the target policy network for inference and calculation. Based on the decision results output by the target strategy network, the robot is controlled to complete the specified operation, achieving stable operation in sparse expert data scenarios.

9. An imperfect demonstration and imitation learning device, characterized in that, The imperfect demonstration imitation learning device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the imperfect demonstration imitation learning method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the imperfect demonstration imitation learning method as described in any one of claims 1 to 8.

Images