Data augmentation method for online reinforcement learning based on diffusion model

Abstract

The present invention discloses an online reinforcement learning data augmentation method based on a diffusion model, including processing the original trajectory data to generate enhanced feature data; extracting the global structure representation and the local detail representation; adopting the hierarchical integration method to fuse and obtain a unified hierarchical representation data; generating constraint coding data based on pre-stored physical constraint parameters and task constraint parameters, and fusing to generate constraint perception representation data; generating an evaluation result based on the current policy parameters; adaptively adjusting the constraint perception representation data to obtain adjusted representation data; performing modal analysis to generate diversified trajectory data to obtain a candidate trajectory set; performing quality assessment, and screening trajectories through a multi-objective optimization method, updating the policy network parameters, and obtaining updated policy parameters. The present invention can automatically generate high-quality training data and improve the training efficiency and performance of the reinforcement learning system.

Description

Technical Field

[0001] The present invention belongs to the field of data enhancement, and in particular to an online reinforcement learning data augmentation method based on a diffusion model. Background Art

[0002] In intelligent control systems in complex dynamic environments, obtaining high-quality training data is a key factor in improving the performance of reinforcement learning. Especially in tasks with high-dimensional state spaces such as robot control and autonomous driving, the system requires a large amount of trajectory data that meets physical constraints and task requirements to train the decision model. Traditional data acquisition methods often rely on repeated experiments of actual physical systems, which is not only time-consuming and labor-intensive, but may also bring safety risks and equipment losses. Therefore, the development of methods that can automatically generate high-quality training data has important theoretical value and practical significance. Through data augmentation technology, the need for physical experiments can be significantly reduced, the training process of reinforcement learning algorithms can be accelerated, and the physical rationality and task relevance of the generated data can be guaranteed.

[0003] At present, data augmentation research in the field of reinforcement learning mainly focuses on basic data transformation and simple generative models. Traditional methods mainly use simple data augmentation techniques such as random noise injection and trajectory segment reorganization. Although these methods can increase the amount of data to a certain extent, it is difficult to ensure the physical rationality of the generated data. Some researchers have tried to use generative adversarial networks (GANs) to generate training data, but due to problems such as unstable GAN training and mode collapse, the quality of the generated data is often difficult to guarantee. In addition, most existing methods use offline learning methods and lack the ability to adapt to dynamic changes in the environment. In terms of constraint processing, existing methods mainly rely on simple penalty functions or hard constraints, which are difficult to effectively handle complex physical constraints and task requirements.

[0004] The existing technologies have the following specific problems in practical applications: First, in the feature extraction stage, it is difficult for existing methods to effectively capture the long-term dependencies and multi-scale features in trajectory data, resulting in the lack of temporal coherence and structural integrity in the generated data. Second, in terms of constraint processing, there is a lack of effective constraint representation and fusion mechanisms, making it difficult to simultaneously meet multiple physical constraints, especially when dealing with nonlinear constraints and time-varying constraints. Third, the existing diffusion model mainly focuses on the generation of global structures and ignores the precise modeling of local details, resulting in discontinuity or non-smoothness in the generated trajectories at the micro level. In addition, in the data screening link, there is a lack of a systematic multi-objective evaluation mechanism, making it difficult to accurately measure the physical rationality and task relevance of the generated data. Finally, the existing methods lack an effective online adaptation mechanism and cannot dynamically adjust the data generation process according to the policy execution effect and environmental changes, affecting the performance of the system in a dynamic environment. These technical problems seriously restrict the effect and promotion of reinforcement learning systems in practical applications. Summary of the invention

[0005] The purpose of the invention is to provide an online reinforcement learning data augmentation method based on a diffusion model, in order to solve at least one technical problem existing in the prior art.

[0006] The technical solution is an online reinforcement learning data augmentation method based on a diffusion model, comprising the following steps:

[0007] S1, read the original trajectory data, perform dynamic feature extraction, physical consistency check and feature enhancement processing respectively, and generate enhanced feature data; wherein the original trajectory data contains state, action, reward and next state information;

[0008] S2, inputting the enhanced feature data into the pre-configured global diffusion model and the local diffusion model respectively to generate a global structure representation and a local detail representation; adopting a hierarchical integration method to fuse the global structure representation and the local detail representation to obtain a unified hierarchical representation data;

[0009] S3, based on the pre-stored physical constraint parameters and task constraint parameters, perform constraint encoding to obtain constraint encoding data; fuse the constraint encoding data with the hierarchical representation data to generate constraint perception representation data;

[0010] S4, obtaining current strategy parameters, performing strategy evaluation based on the current strategy parameters and pre-stored environmental state data, and obtaining evaluation results; adaptively adjusting the constraint perception representation data according to the evaluation results, and obtaining adjusted representation data;

[0011] S5. Based on the adjusted representation data, perform modal analysis to obtain analysis results; based on the analysis results, generate diversified trajectory data to obtain a candidate trajectory set;

[0012] S6. Based on the candidate trajectory set, quality assessment is performed to obtain a quality assessment result; based on the quality assessment result, trajectory screening is performed through a multi-objective optimization method to obtain a screened trajectory data set;

[0013] S7: storing the filtered trajectory data set into a data pool and updating the policy network parameters to obtain updated policy parameters.

[0014] Beneficial effects: The present invention can adaptively process complex dynamic environments, generate diversified trajectory data that meet physical constraints and task requirements, and improve the training efficiency and performance of the reinforcement learning system; ensure the physical rationality and task relevance of the generated data, provide rich training samples for the reinforcement learning system, and accelerate the convergence process of strategy learning; can be flexibly applied to different types of control tasks and scenarios, and realize the automatic generation and optimization of high-quality trajectory data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the present invention.

[0016] Figure 2 This is a flow chart of step S1 of the present invention.

[0017] Figure 3 This is a flow chart of step S2 of the present invention.

[0018] Figure 4 This is a flow chart of step S3 of the present invention.

[0019] Figure 5 This is a flow chart of step S4 of the present invention.

[0020] Figure 6 This is a flow chart of step S5 of the present invention.

[0021] Figure 7 This is a flow chart of step S6 of the present invention.

[0022] Figure 8 This is a flow chart of step S7 of the present invention. DETAILED DESCRIPTION

[0023] like Figure 1 As shown, this application proposes an online reinforcement learning data augmentation method based on a diffusion model, comprising the following steps:

[0024] S1, read the original trajectory data, perform dynamic feature extraction, physical consistency check and feature enhancement processing respectively, and generate enhanced feature data; wherein the original trajectory data contains state, action, reward and next state information;

[0025] S2, inputting the enhanced feature data into the pre-configured global diffusion model and the local diffusion model respectively to generate a global structure representation and a local detail representation; adopting a hierarchical integration method to fuse the global structure representation and the local detail representation to obtain a unified hierarchical representation data;

[0026] S3, based on the pre-stored physical constraint parameters and task constraint parameters, perform constraint encoding to obtain constraint encoding data; fuse the constraint encoding data with the hierarchical representation data to generate constraint perception representation data;

[0027] S4, obtaining current strategy parameters, performing strategy evaluation based on the current strategy parameters and pre-stored environmental state data, and obtaining evaluation results; adaptively adjusting the constraint perception representation data according to the evaluation results, and obtaining adjusted representation data;

[0028] S5. Based on the adjusted representation data, perform modal analysis to obtain analysis results; based on the analysis results, generate diversified trajectory data to obtain a candidate trajectory set;

[0029] S6. Based on the candidate trajectory set, quality assessment is performed to obtain a quality assessment result; based on the quality assessment result, trajectory screening is performed through a multi-objective optimization method to obtain a screened trajectory data set;

[0030] S7: storing the filtered trajectory data set into a data pool and updating the policy network parameters to obtain updated policy parameters.

[0031] like Figure 2 As shown, according to one aspect of the present application, step S1 is further:

[0032] S11, reading original trajectory data from a pre-stored data storage device; inputting the original trajectory data into a trajectory preprocessing module, performing data standardization processing, and obtaining standardized trajectory data; wherein the original trajectory data includes state, action, reward, and next state information;

[0033] S12, inputting the standardized trajectory data into a dynamic feature extractor, using a self-attention mechanism to calculate the correlation in the time dimension, and obtaining a temporal correlation matrix; performing feature extraction operations based on the temporal correlation matrix and the standardized trajectory data, and generating a dynamic feature vector;

[0034] S13, inputting the dynamic characteristic vector and the pre-stored physical constraint parameters into the physical constraint verification module, performing physical rationality verification calculation using the Lagrange equation, and outputting effective dynamic characteristics;

[0035] S14, performing wavelet transform processing on the effective dynamic features to obtain multi-scale feature data; inputting the multi-scale feature data into a feature fusion module to extract and combine local features and global features to generate enhanced feature data.

[0036] In one embodiment of the present application, the temporal attention calculation method is: the attention weight matrix A(Q, K) = softmax(QK T / sqrt(d) + M)·V; where Q = W_q·X is the query matrix, K = W_k·X is the key matrix, and V = W_v·X is the value matrix; M = mask(ΔT) is the temporal position encoding matrix, mask(ΔT) = -∞ if ΔT > τelse ln(1 / ΔT); W_q, W_k, W_v are learnable projection matrices; X∈R n×d is the input feature sequence; d is the feature dimension; n is the sequence length; τ is the time window threshold; ΔT is the time interval matrix; softmax(·) is the normalization function,T By introducing temporal position encoding and adaptive threshold, the ability to model long-term dependencies is enhanced.

[0037] Multi-scale feature fusion algorithm: Multi-feature fusion function F(X) = ∑(α_i·W_i·DWT_i(X)); where DWT_i(X) is the discrete wavelet transform result of the i-th scale; W_i is the transformation matrix of the i-th layer feature; α_i = sigmoid(MLP(h_i)) is the adaptive weight coefficient; h_i is the hidden layer state of the i-th layer; MLP(·) is a multi-layer perceptron; X∈R n×d is the input feature sequence; n is the sequence length; d is the feature dimension; i∈[1, L] is the scale level; L is the total number of scales. The dynamic fusion of multi-scale features is achieved through the adaptive weight mechanism.

[0038] This embodiment achieves high-quality preprocessing and feature extraction of the original trajectory data through multi-level feature extraction and enhancement processing. First, the self-attention mechanism is used to calculate the correlation in the time dimension, which enables the system to adaptively capture long-term and short-term dependencies in the trajectory data, avoiding the information loss problem in the traditional time series processing method. Secondly, the physical rationality is verified by the Lagrange equation to ensure that the generated features meet the basic physical laws. The introduction of this physical constraint improves the authenticity and availability of the subsequent generated data. Finally, through wavelet transform and feature fusion, the effective extraction and combination of multi-scale features are achieved, which not only retains the global structural characteristics of the trajectory, but also maintains the local detail information, thereby providing a richer and more accurate feature representation for the subsequent diffusion model. This embodiment improves the expressiveness of the features, so that the subsequent data enhancement process can better maintain the essential characteristics and physical constraints of the original data.

[0039] According to one aspect of the present application, step S11 is further:

[0040] S111, reading original trajectory data from a pre-stored data storage device; segmenting the original trajectory data by sliding a fixed-size time window along the time axis step by step to generate segmented trajectory data;

[0041] S112, input the segmented trajectory data into the trajectory preprocessing module, calculate the local density and relative distance between each data point and its neighboring points; judge and remove abnormal points according to the preset thresholds of the local density and relative distance, and generate preprocessed trajectory data;

[0042] S113, inputting the preprocessed trajectory data into a time series feature extraction module, decomposing the preprocessed trajectory data into a predetermined number of recursive subsequences, extracting time series features from each recursive subsequence and combining them to generate time series feature data;

[0043] S114, input the time series feature data into the data standardization module, calculate the mean and standard deviation of the time series feature data; based on the mean and standard deviation, perform zero mean and variance normalization processing on the time series feature data to generate standardized trajectory data.

[0044] This embodiment achieves high-quality initialization processing of the original trajectory data through data preprocessing and standardization processes. First, the use of a fixed-size time window for trajectory segmentation can effectively process trajectory sequences of different lengths and maintain the continuity of timing information. Secondly, by calculating local density and relative distance to identify and remove outliers, it is particularly suitable for processing noisy trajectory data and can effectively improve data quality. By decomposing the trajectory data into recursive subsequences and extracting timing features, the periodic patterns and long-term and short-term dependencies in the trajectory can be captured. Finally, through zero mean and variance normalization processing, the scale consistency of features of different dimensions is ensured, which not only improves the training effect of subsequent models, but also enhances the system's adaptability to different types of trajectory data. This embodiment forms a complete data cleaning and standardization system, which lays a solid foundation for subsequent feature extraction and model training.

[0045] According to one aspect of the present application, step S12 is further:

[0046] S121, input the standardized trajectory data into the dynamic feature extractor, calculate the position code according to the time step, add the position information to the data of each time position, and generate the position code data.

[0047] S122. Input the position encoding data into the query-key value calculation module, calculate the query vector, key vector and value vector respectively through linear transformation, and generate an attention calculation vector.

[0048] S123. Input the attention calculation vector into the relevance calculation module, calculate the dot product of the query vector and the key vector, and perform scaling and normalization to generate a temporal relevance matrix.

[0049] S124, input the time series correlation matrix and the standardized trajectory data into the feature synthesis module, perform weighted summation on the value vector through the correlation matrix and perform nonlinear transformation to generate a dynamic feature vector.

[0050] This embodiment realizes the deep temporal feature representation of trajectory data through the dynamic feature extraction method. First, the temporal information is added to each time step by means of position encoding, so that the model can perceive the relative position relationship of the data points in the time series. Secondly, the query-key value mechanism is used to calculate the attention vector, which can adaptively capture the dependencies between different time steps, and is particularly suitable for processing long sequence data. Through the calculation and normalization of the association matrix, the interaction strength between different time steps can be accurately measured, and the expressive power of the feature representation is improved. Finally, through the nonlinear transformation of the feature synthesis module, the temporal association information is fused with the original features to generate a dynamic feature vector with rich temporal semantics. This embodiment can not only effectively capture the long-term dependencies in the trajectory data, but also maintain the integrity of the local temporal structure.

[0051] According to one aspect of the present application, step S13 is further:

[0052] S131. Input the dynamic characteristic vector into the energy constraint verification module, calculate the rate of change of the system kinetic energy and potential energy and verify the law of conservation of energy, and generate energy constraint data.

[0053] S132, input the energy constraint data and pre-stored physical constraint parameters into the kinematic constraint module, calculate the continuity of the joint angle, velocity and acceleration and verify the kinematic restrictions, and generate motion constraint data.

[0054] S133, input the motion constraint data into the dynamic equation verification module, calculate the balance of generalized force and inertial force through Lagrange equation and check the dynamic consistency, and generate dynamic verification data.

[0055] S134, input the dynamic verification data into the constraint integration module, calculate the comprehensive satisfaction of multiple constraints and screen the compliance features to generate effective dynamic features.

[0056] This embodiment achieves high-quality screening of dynamic features through the physical constraint verification mechanism of the system. First, the changes in kinetic energy and potential energy of the system are calculated through the energy constraint verification module to ensure that the features satisfy the law of conservation of energy, and can effectively exclude features that violate physical laws. Secondly, the continuity of joint angles, velocities, and accelerations is verified through the kinematic constraint module to ensure the rationality of the features at the kinematic level. Through the dynamic verification of the Lagrange equation, the equilibrium relationship between generalized forces and inertial forces can be accurately evaluated, which improves the physical reliability of the features. Finally, the degree of satisfaction of multiple constraints is comprehensively evaluated through the constraint integration module, thereby achieving a comprehensive screening of dynamic features. This embodiment not only ensures the physical rationality of the features, but also provides an explainable feature selection criterion.

[0057] According to one aspect of the present application, step S14 is further:

[0058] S141. Input the effective dynamic characteristics into the wavelet decomposition module, decompose the signal into subsequences of different frequency bands through orthogonal wavelet basis functions, and generate wavelet coefficient data.

[0059] S142, input the wavelet coefficient data into the scale analysis module, perform threshold processing on the coefficients of each frequency band and reconstruct the subsequence to generate multi-scale feature data.

[0060] S143, input the multi-scale feature data into the local feature extraction module, calculate the local statistics and extract the local change pattern through the sliding window, and generate local feature data.

[0061] S144, inputting the local feature data and the multi-scale feature data into the feature fusion module, calculating the importance of features of different scales through an adaptive weight network and weighted combining them to generate enhanced feature data.

[0062] This embodiment achieves effective feature enhancement through multi-scale feature analysis and fusion methods. First, wavelet decomposition is used to decompose the signal into subsequences of different frequency bands, which can capture the global and local features of the signal at the same time. Secondly, the coefficients of different frequency bands are adaptively thresholded through the scale analysis module, which can effectively remove noise while retaining key information. By calculating local statistics through a sliding window, the local change pattern of the feature can be accurately described, which is particularly suitable for processing non-stationary time series data. Finally, the dynamic fusion of features of different scales is achieved through an adaptive weight network, which improves the expressive power of the features. This embodiment forms a complete multi-scale feature processing system, which provides rich feature representation for subsequent data generation.

[0063] like Figure 3 As shown, according to one aspect of the present application, step S2 is further:

[0064] S21, inputting the enhanced feature data into the global diffusion model, using a coarse-grained diffusion processing method to model and analyze the overall structure of the trajectory, and generating global structure representation data;

[0065] S22. Based on the global structure representation data, a fine-grained diffusion model is constructed; based on the fine-grained diffusion model, the local action sequence and state transition of the trajectory are processed by the local temporal association analysis method to generate local detail representation data;

[0066] S23. Input the global structure representation data and the local detail representation data into the hierarchical integration module, and use an adaptive weighted fusion method to integrate the global structure representation data and the local detail representation data to generate unified hierarchical representation data.

[0067] In one embodiment of the present application, the coarse-grained diffusion processing method is specifically as follows: diffusion process q(x_t|x_{t-1}) = N(x_t; sqrt(1-β_t)·x_{t-1}, β_t·I); reverse diffusion process p_θ(x_{t-1}|x_t) = N(x_{t-1}; μ_θ(x_t, t), σ_θ(x_t, t)); where μ_θ(x_t, t) = (x_t - β_t·ε_θ(x_t, t)) / sqrt(1-β_t); σ_θ(x_t, t) = z·sqrt(β_t / (1-β_t)); ε_θ is the noise prediction network; β_t is the noise scheduling coefficient; θ is the model parameter; t is the diffusion step; z is the random noise; N(·) is the normal distribution, I is the unit matrix, x_t is the data point at time t, q(x_t|x_{t-1}) is the conditional probability distribution of the diffusion process, and p_θ(x_{t-1}|x_t) is the conditional probability distribution of the inverse diffusion process. The quality of the global structure generation is improved by the improved noise prediction network.

[0068] Local state transition probability algorithm: state transition probability P(s_t|s_{t-1}, a_{t-1}) = ∑w_k·N(μ_k(s_{t-1}, a_{t-1}), Σ_k(s_{t-1}, a_{t-1})); where μ_k = f_θ(s_{t-1}, a_{t-1}) is the mean of the kth Gaussian component; Σ_k = g_φ(s_{t-1}, a_{t-1}) is the covariance matrix; w_k = softmax(h_ψ(s_{t-1}, a_{t-1})) is the mixing weight; f_θ, g_φ, h_ψ are neural networks; s_t is the state at time t; a_t is the action at time t; k is the mixed component index. More accurate local dynamic modeling is achieved through the mixed Gaussian model.

[0069] The specific process of integration is as follows: a hierarchical feature integration algorithm is used, and the feature integration function is H(x_g, x_l) = λ·GRU(x_g) + (1-λ)·TCN(x_l); x_g is a global feature; x_l is a local feature; GRU(·) is a gated recurrent unit; TCN(·) is a temporal convolutional network; λ = sigmoid(v T ·[x_g;x_l]) is the adaptive weight; v is the learnable parameter vector; [·;·] represents the feature concatenation operation. The dynamic weight mechanism is used to achieve the adaptive fusion of global and local features.

[0070] This embodiment realizes multi-scale generation and fusion of trajectory data by combining the global diffusion model and the local diffusion model. The global diffusion model adopts a coarse-grained diffusion processing method, and captures the macroscopic characteristics and long-range dependencies of the data by modeling and analyzing the overall structure of the trajectory. At the same time, the local diffusion model focuses on the local action sequence and state transition of the trajectory through fine-grained processing, ensuring the accuracy and continuity of the generated data at the micro level. This embodiment effectively balances the consistency of the global structure and the richness of local details through an adaptive weighted fusion mechanism. In particular, when processing complex action sequences, it can simultaneously ensure the overall coherence and local smoothness of the trajectory, thereby improving the quality and diversity of the generated data. Through this hierarchical processing and fusion method, not only the efficiency of data generation is improved, but also the authenticity and availability of the generated data at different scales are ensured.

[0071] According to one aspect of the present application, step S21 is further:

[0072] S211, inputting the enhanced feature data into the structural decomposition module, calculating the main structural components and extracting key patterns through singular value decomposition, and generating structural component data.

[0073] S212: Input the structural component data into the diffusion sampling module, gradually inject random noise into the data according to the noise scheduling table, calculate the sampling trajectory, and generate noise trajectory data.

[0074] S213, inputting the noisy trajectory data into a trajectory reconstruction module, gradually removing the noise and restoring the structural features through a reverse diffusion process, and generating reconstructed trajectory data.

[0075] S214, inputting the reconstructed trajectory data into the structure optimization module, optimizing and adjusting the trajectory structure by maximizing the variational lower bound, and generating global structure representation data.

[0076] This embodiment achieves high-quality structural representation of trajectory data through the application of a global diffusion model. First, by performing singular value decomposition through the structural decomposition module, the main structural components of the trajectory data can be effectively extracted, which is particularly suitable for processing high-dimensional trajectory data. Secondly, the diffusion process is controlled by a noise scheduling table. By gradually injecting and removing random noise, the potential structure of the data distribution can be explored, which improves the diversity and authenticity of the generated data. Trajectory reconstruction through the reverse diffusion process can accurately restore the structural characteristics of the original data while introducing new change patterns. Finally, the trajectory structure is optimized by maximizing the variational lower bound, which not only ensures the quality of the generated data, but also provides a reliable quality evaluation indicator. This embodiment forms a complete structural representation learning system, which is particularly suitable for processing trajectory data with complex spatiotemporal dependencies.

[0077] According to one aspect of the present application, step S22 is further:

[0078] S221, inputting the global structure representation data into the time series segmentation module, adaptively segmenting the trajectory based on the rate of change of state transition, and generating segmented sequence data.

[0079] S222, input the segmented sequence data into the fine-grained diffusion module, calculate the state transition probability in each segment and construct a local Markov jump matrix to generate local transition probability data.

[0080] S223: Input the local transition probability data into the action association analysis module, calculate the mutual information and conditional entropy for the action sequence in each time window, and generate action association data.

[0081] S224. Input the action-related data into the sequence reconstruction module, reconstruct the local state transition sequence through the probabilistic graphical model and optimize the action continuity to generate local detail representation data.

[0082] This embodiment achieves accurate modeling of local details of the trajectory through the design of a fine-grained diffusion model. First, adaptive segmentation based on the rate of change of state transitions can accurately capture the key transition points in the trajectory and maintain the integrity of the local action sequence. Secondly, by constructing a local Markov jump matrix to describe the state transition probability, it is particularly suitable for processing action sequences with uncertainty. By calculating mutual information and conditional entropy to analyze the correlation of actions, the dependencies in the action sequence can be accurately identified, thereby improving the accuracy of local modeling. Finally, the local state transition sequence is reconstructed through a probabilistic graphical model to ensure the continuity and smoothness of the action. This embodiment not only improves the generation quality of local details, but also ensures the rationality of the generated trajectory at the micro level.

[0083] According to one aspect of the present application, step S23 is further:

[0084] S231, inputting the global structure representation data and the local detail representation data into a feature alignment module, calculating the temporal correspondence between the global structure representation data and the local detail representation data through a dynamic time warping algorithm, and generating alignment feature data;

[0085] S232, inputting the alignment feature data into the importance evaluation module, calculating the information contribution of features at different levels through information entropy, and generating feature weight data;

[0086] S233, inputting the feature weight data and the alignment feature data into the feature fusion module, calculating the interaction relationship between the features through the attention mechanism, and generating the interaction feature data;

[0087] S234, input the interactive feature data into the hierarchical combination module, adaptively combine the multi-level features through the residual connection network, and generate unified hierarchical representation data.

[0088] This embodiment achieves the effective fusion of global structure and local details through a hierarchical feature integration mechanism. First, the dynamic time warping algorithm is used for feature alignment, which can handle feature representations of different time scales and ensure the accuracy of feature fusion. Secondly, the importance of features at different levels is evaluated through information entropy calculation, and the contribution of different features can be adaptively adjusted. By establishing the interactive relationship between features through the attention mechanism, the complex dependencies between features at different levels can be captured, and the expressive power of the representation is improved. Finally, the adaptive combination of multi-level features is realized through the residual connection network, ensuring the effective transmission of information between different levels. This embodiment not only improves the integrity of feature representation, but also enhances the model's ability to process information of different scales.

[0089] like Figure 4 As shown, according to one aspect of the present application, step S3 is further:

[0090] S31, inputting the pre-stored physical constraint parameters and task constraint parameters into the constraint representation learning module, and generating constraint encoding data through a constraint embedding calculation method;

[0091] S32. Input the constrained coding data and the unified hierarchical representation data into the constraint fusion module, use the conditional normalization processing method to perform fusion operation on the constrained coding data and the hierarchical representation data, and generate constraint-aware representation data.

[0092] In one embodiment of the present application, the constraint representation learning module is specifically: constraint encoding function E(c) = GAT(MLP(c)); constraint decoding function D(z) = σ(W·z + b); constraint reconstruction loss L_c = ||c - D(E(c))|| 2 +KL(E(c)||N(0,I)); where c is the original constraint; z is the latent code; GAT(·) is the graph attention network; MLP(·) is the multi-layer perceptron; σ(·) is the activation function; W, b are learnable parameters; KL(·) is the KL divergence, and N(0,I) represents a normal distribution with a mean of 0 and a covariance matrix of the identity matrix. Efficient encoding of constraints is achieved through variational autoencoders and graph neural networks.

[0093] The specific process of fusion operation is as follows: constrain fusion target J(θ) = ∑w_i·L_i(θ) + λ·R(θ); constrain update Δθ = -α·▽_θ·J(θ)·max(0, 1-Δ / ||▽_θ·J(θ)||); where L_i is the loss function of the ith constraint; w_i is the constraint weight; R(θ) is the regularization term; λ is the regularization coefficient; α is the learning rate; Δ is the trust region radius; θ is the model parameter, and ▽_θ is the gradient of the model parameter. Stable multi-constraint optimization is achieved through trust region constraints.

[0094] This embodiment realizes the effective integration of physical constraints and task constraints through constraint encoding and fusion mechanism. First, through the constraint embedding calculation method, the discrete constraint conditions are converted into continuous vector representation. This conversion enables the system to better understand and handle various types of constraint relationships. Secondly, the conditional normalization processing method is used for constraint fusion, which not only ensures the effective application of constraint conditions, but also maintains the stability of feature distribution. Through this constraint-aware representation learning method, it is possible to ensure that the generated trajectory data strictly complies with predefined physical laws and task requirements while maintaining data diversity. This embodiment improves the physical rationality and task relevance of the generated data, and also provides an explainable constraint processing mechanism, so that the system can better control and adjust the data generation process.

[0095] According to one aspect of the present application, step S31 is further:

[0096] S311. Input the pre-stored physical constraint parameters and task constraint parameters into the parameter decomposition unit of the constraint representation learning module, classify and decompose the constraints through the constraint type identifier, and generate constraint decomposition data.

[0097] S312. Input the constraint decomposition data into the semantic encoding unit of the constraint representation learning module, encode the logical relationship between the constraints through the graph neural network, and generate constraint relationship data.

[0098] S313, input the constraint relationship data into the embedding calculation unit of the constraint representation learning module, map the constraints to the continuous vector space through the deep encoder, and generate constraint embedding data.

[0099] S314, input the constraint embedding data into the coding optimization unit of the constraint representation learning module, optimize the distinguishability of the constraint representation by a comparative learning method, and generate constraint coding data.

[0100] This embodiment achieves efficient encoding of physical and task constraints through the design of constraint representation learning. First, the constraints are classified and decomposed through the constraint type identifier, which can handle complex mixed constraints. Secondly, the logical relationship between constraints is encoded by a graph neural network, which is particularly suitable for processing constraint systems with complex dependencies. By mapping the constraints to a continuous vector space through a deep encoder, a constraint representation with good mathematical properties can be generated, which improves the flexibility of constraint processing. Finally, the distinguishability of the constraint representation is optimized through a comparative learning method, ensuring the separability of different constraints in the representation space. This embodiment not only improves the efficiency of constraint processing, but also provides a reliable evaluation basis for subsequent constraint satisfaction.

[0101] According to one aspect of the present application, step S32 is further:

[0102] S321. Input the constraint coding data and the unified hierarchical representation data into the constraint matching module, calculate the constraint similarity matrix in the representation space and identify the key constraint points, and generate constraint matching data.

[0103] S322: Input the constraint matching data into the normalization processing module, calculate the conditional distribution parameters by batch statistics and perform adaptive scaling to generate normalized constraint data.

[0104] S323: Input the normalized constraint data into the hierarchical fusion module, perform weighted combination on constraint features at different levels and adjust feature distribution to generate fused feature data.

[0105] S324. Input the fused feature data into the constraint verification module, verify the constraint satisfaction through the forward propagation network and adjust the feature representation to generate constraint-aware representation data.

[0106] This embodiment realizes the unified processing and integration of multiple constraints through the application of constraint fusion mechanism. First, the importance weights of different constraints are calculated through the constraint priority evaluation module, which can accurately reflect the actual impact of the constraint conditions. Secondly, the soft constraint conversion technology is used to convert hard constraints into soft constraints with penalty terms, which improves the robustness and convergence of constraint solving. By constructing a constraint graph network to describe the interaction relationship between constraints, the coupling effect between constraints can be effectively processed, which is particularly suitable for processing constraint systems with complex dependencies. Finally, the dynamic adjustment and balance of constraints are realized through the Lagrange multiplier method, ensuring the stability of the constraint satisfaction process. This embodiment not only improves the efficiency of constraint processing, but also provides a reliable constraint guarantee for generating high-quality trajectory data.

[0107] like Figure 5 As shown, according to one aspect of the present application, step S4 is further:

[0108] S41, obtaining current strategy parameters, inputting the current strategy parameters and pre-stored environmental status data into a strategy status evaluation module, performing calculations through a strategy performance analysis method and an environmental dynamic feature analysis method, and generating adaptive parameter data;

[0109] S42, input the adaptive parameter data and the constraint-aware representation data into the condition adjustment module, and use a dynamic update method to adaptively adjust the constraint-aware representation data to generate adjusted representation data.

[0110] In one embodiment of the present application, the parameter adaptive adjustment algorithm: parameter update equation Δθ = clip(α·▽_θJ(π_θ)·A(s,a), -ε,ε); advantage function A(s,a) = Q_w(s,a) - V_v(s); where π_θ is the policy function; Q_w is the action value function; V_v is the state value function; α is the learning rate; ε is the clipping threshold; ▽_θJ is the policy gradient; clip(·) is the clipping function; w, v are the value network parameters. Stable parameter optimization is achieved through advantage estimation and gradient clipping.

[0111] This embodiment introduces an adaptive adjustment mechanism based on policy evaluation to achieve dynamic optimization of representation data. Through the dual evaluation method of policy performance analysis and environmental dynamic feature analysis, it is possible to capture the policy execution effect and environmental change characteristics in real time and generate high-quality adaptive parameter data. Especially in a dynamic and highly uncertain environment, this adaptive adjustment mechanism can timely adjust the feature distribution of the representation data according to the execution effect of the current policy and the changes in the environmental state, ensuring that the generated data can better adapt to the current task requirements and environmental constraints. The adaptive adjustment of the representation data by the dynamic update method not only improves the environmental adaptability and task relevance of the generated data, but also enhances the robustness and generalization ability of the entire data enhancement process. This embodiment improves the effect and practicality of data enhancement, especially when dealing with complex reinforcement learning tasks, it can better balance the relationship between exploration and utilization.

[0112] According to one aspect of the present application, step S42 is further:

[0113] S421. Input the adaptive parameter data and the constraint perception representation data into the state prediction unit of the condition adjustment module, calculate the state transition probability through Bayesian reasoning, and generate state prediction data.

[0114] S422: input the state prediction data into the parameter adjustment unit of the condition adjustment module, calculate the optimal adjustment strategy through dynamic programming, and generate adjustment strategy data.

[0115] S423. Input the adjustment strategy data and the constraint-aware representation data into the representation update unit of the condition adjustment module, update the representation features through a recursive neural network, and generate updated feature data.

[0116] S424, input the updated feature data into the consistency verification unit of the condition adjustment module, verify the adjusted consistency through the constraint propagation network, and generate adjusted representation data.

[0117] This embodiment realizes dynamic optimization of representation data through the design of an adaptive parameter adjustment mechanism. First, by calculating the direction and amplitude of parameter adjustment through policy gradient estimation, the impact of parameter changes on policy performance can be effectively captured. Secondly, the step size of parameter update is controlled by trust region constraints, which effectively prevents performance fluctuations caused by excessive parameter changes. By constructing a value function network to evaluate the effect of parameter adjustment, the impact of parameter changes on future returns can be accurately predicted, thereby improving the efficiency of parameter optimization. Finally, by accumulating and utilizing historical adjustment experience through the experience replay mechanism, the stability and convergence of the parameter optimization process are ensured. This embodiment not only improves the quality of representation data, but also enhances the system's ability to adapt to environmental changes.

[0118] like Figure 6 As shown, according to one aspect of the present application, step S5 is further:

[0119] S51, inputting the adjusted representation data into a modal analysis module, identifying and processing the behavior pattern through a cluster analysis method, and generating modal description data;

[0120] S52, inputting the modal description data into a trajectory generation module, using a variational reasoning method to generate diversified trajectory data, and obtaining a candidate trajectory set.

[0121] This embodiment achieves the generation of a high-quality candidate trajectory set through modal analysis and diversified trajectory generation methods. Through the modal analysis module and cluster analysis method, it is possible to accurately identify and process behavioral patterns, generate detailed modal description data, and help better understand and analyze the operating status and behavioral characteristics of the system. Diversified trajectory data is generated through variational inference methods, ensuring the diversity and representativeness of the generated trajectories. This embodiment not only improves the quality and diversity of the generated data, but also ensures that the generated trajectories can cover the key behavioral patterns in the task space, providing rich training samples for the subsequent reinforcement learning process.

[0122] According to one aspect of the present application, step S51 is further:

[0123] S511, input the adjusted representation data into the feature embedding module, map the high-dimensional representation to the low-dimensional space through the manifold learning method and maintain the local structure to generate embedded feature data.

[0124] S512: Input the embedded feature data into the density estimation module, calculate the local density distribution of the sample points and identify the density peak points, and generate density distribution data.

[0125] S513: Input the density distribution data into the pattern grouping module, perform adaptive grouping on the sample points through density reachability analysis and merge similar clusters to generate behavior grouping data.

[0126] S514: Input the behavior grouping data into the pattern feature extraction module, calculate the statistical features for each behavior cluster and extract the typical behavior pattern to generate modal description data.

[0127] In one embodiment of the present application, the modal density estimation algorithm: kernel density estimation function p(x) = ∑K((x-x_i) / h) / (nh d ); adaptive bandwidth h_i = h·(f(x_i) / g) -β ; where K(·) is the kernel function; h is the basic bandwidth; n is the number of samples; d is the feature dimension; f(x_i) is the local density estimate; g is the geometric mean density; β is the adaptive coefficient; x_i is the sample point. A more accurate modal density estimation is achieved through adaptive bandwidth.

[0128] This embodiment achieves accurate identification and classification of trajectory modes through the application of density estimation and cluster analysis. First, the manifold learning method is used for feature embedding, which maintains the local structural characteristics of the high-dimensional representation in the low-dimensional space, which not only improves the computational efficiency, but also maintains the essential characteristics of the data. Secondly, through density estimation and reachability analysis, different behavior patterns can be automatically identified and classified, which is particularly suitable for processing continuous trajectory data. By calculating statistical features for each behavior cluster and extracting typical behavior patterns, the generated modal description data can more comprehensively reflect the characteristics of different behavior patterns and improve the accuracy of behavior pattern feature extraction. This embodiment can more accurately identify and analyze behavior patterns, improve the overall performance and decision-making ability of the system, and help make better decisions in complex environments.

[0129] like Figure 7 As shown, according to one aspect of the present application, step S6 is further:

[0130] S61, inputting the candidate trajectory set into the evaluation index calculation module, processing it through the physical feasibility calculation method and the task relevance calculation method, and generating quality score data;

[0131] S62: input the quality score data and the candidate trajectory set into the trajectory screening module, select the optimal trajectory sample using a multi-objective optimization method, and generate a screened trajectory data set.

[0132] In one embodiment of the present application, a multi-objective evaluation algorithm: evaluation function Score(x) = ω_1·f_dyn(x) + ω_2·f_task(x) + ω_3·f_smooth(x); dynamic constraint f_dyn(x) = exp(-||x"-g(x, x')|| 2 / σ 2 ); Task completion f_task(x) = exp(-d(x, x_goal) / τ); Smoothness f_smooth(x) = exp(-∑||x"_t-x"_{t-1}|| 2 / γ 2 );where: g(·) is the dynamic model; d(·) is the distance metric; ω_i is the weight coefficient; σ, τ, γ are scaling parameters; x_goal is the target state; x is the state; x' is the first-order derivative of the state, indicating the velocity; x" is the second-order derivative of the state, indicating the acceleration. Comprehensive trajectory evaluation is achieved through multi-objective combination.

[0133] This embodiment achieves effective screening of high-quality trajectory data through a multi-objective optimized trajectory screening mechanism. First, the candidate trajectory set is evaluated through a physical feasibility calculation method and a task relevance calculation method to generate quality score data, ensuring that the trajectory data is physically feasible and highly relevant to the task, and improving the quality and reliability of the trajectory data. Then, a multi-objective optimization method is used to screen the quality score data and the candidate trajectory set, and select the optimal trajectory sample, so that multiple evaluation indicators can be comprehensively considered to ensure that the selected trajectory sample performs well in all aspects, and the trajectory selection process is optimized. Finally, by generating a screened trajectory data set, decisions can be made more quickly and accurately, improving the decision-making ability and efficiency of the system, which is of great significance for application scenarios with high real-time requirements, such as intelligent transportation and robot navigation. This embodiment can better cope with various complex environments and emergencies, enhance the adaptability and robustness of the system, which enables the system to maintain stable and efficient operation in different scenarios.

[0134] According to one aspect of the present application, step S61 is further:

[0135] S611, input the candidate trajectory data set into the physical constraint verification module, calculate the residual of the trajectory dynamics equation and verify energy conservation, and generate physical constraint scoring data.

[0136] S612: Input the physical constraint scoring data and its corresponding trajectory features into the task consistency module, calculate the deviation between the trajectory and the task target, evaluate the degree of completion, and generate task relevance data.

[0137] S613: Input the task relevance data into the smoothness analysis module, calculate the acceleration change rate and jitter index of the trajectory, and generate smoothness score data.

[0138] S614: Input the smoothness score data and task relevance data into the comprehensive evaluation module, calculate the overall performance score through multi-indicator weighting, and generate quality score data.

[0139] This embodiment achieves high-quality screening of trajectory data. The physical constraint verification module calculates the residual of the dynamic equation of the trajectory and verifies the conservation of energy, and the generated physical constraint scoring data ensures the physical feasibility and rationality of the trajectory data. The task consistency module calculates the deviation between the trajectory and the task target and evaluates the completion degree, and the generated task relevance data ensures the high consistency of the trajectory data and the task target, and improves the accuracy and efficiency of task completion. The smoothness analysis module calculates the acceleration change rate and jitter index of the trajectory, and the generated smoothness scoring data ensures the smoothness of the trajectory, reduces the mutation and jitter in the trajectory, and improves the stability and controllability of the trajectory. The comprehensive evaluation module performs multi-index weighted calculation on the smoothness scoring data and the task relevance data, and the generated quality scoring data can comprehensively evaluate the overall performance of the trajectory, ensuring that the selected trajectory performs well in all aspects. This embodiment can evaluate and select the optimal trajectory more quickly and accurately, improve the decision-making ability and efficiency of the system, which is of great significance for application scenarios with high real-time requirements, such as intelligent transportation and robot navigation.

[0140] like Figure 8 As shown, according to one aspect of the present application, step S7 is further:

[0141] S71, storing the screened trajectory data set into an enhanced data pool, combining the data in the enhanced data pool with the data in the pre-stored real data pool to generate a combined data set;

[0142] S72: input the combined data set into the policy update module, use the trust domain policy optimization method to update the policy network, and generate updated policy parameters.

[0143] This embodiment realizes the effective combination of data enhancement and policy optimization through data pool management and policy update mechanism. First, by combining the screened trajectory data with the data in the real data pool, a training data set with good balance and diversity is constructed. Secondly, the trust domain policy optimization method is used to update the policy network, which can effectively control the step size of the policy update and avoid instability caused by excessive policy changes. This embodiment not only improves the utilization efficiency of the generated data, but also ensures the stability and convergence of the policy learning process. Especially when dealing with long-term planning tasks, it can better balance the use of historical experience and newly generated data, thereby improving the training effect of reinforcement learning.

[0144] This application aims at the problem of long-term dependency and multi-scale feature extraction, uses a fixed-size time window to segment the trajectory, and retains the temporal dependency information through recursive subsequence decomposition; introduces position encoding and self-attention mechanism, effectively captures long-term dependency through query-key mechanism; realizes multi-scale feature analysis, captures feature information of different frequency bands through wavelet decomposition and adaptive threshold processing; realizes dynamic fusion of features of different scales through adaptive weight network to ensure the integrity of feature representation. For the problem of constraint representation and fusion, the semantic decomposition and graph neural network encoding of constraints are realized through constraint representation learning module; the constraints are mapped to continuous vector space by deep encoder to facilitate the processing of nonlinear constraints; the hard constraints are processed by soft constraint conversion technology to improve the robustness of constraint solution; the constraint graph network is constructed to process the coupling effect between constraints, and dynamic adjustment is realized through Lagrange multiplier method. For the problem of local detail modeling, the main structural features of the trajectory are captured through the global diffusion model; the local details are described through state transition modeling and Markov jump matrix; the effective fusion of global structure and local details is realized through the hierarchical feature integration mechanism; the dynamic time warping algorithm is used to ensure the alignment of features of different time scales. For multi-objective evaluation problems, the physical feasibility score is calculated through the dynamic equation, and the task completion evaluation module is constructed to measure the task relevance; smoothness and continuity indicators are introduced to evaluate the motion quality; and the Pareto optimization method is used to balance multiple evaluation indicators. For the problem of online adaptation mechanism, the adaptive adjustment mechanism of policy evaluation is used to achieve dynamic optimization of the representation data; parameter updates are controlled based on policy gradient estimation and trust region constraints; data pool management and policy update mechanisms are used to achieve dynamic combination of data enhancement and policy optimization; and the trust region policy optimization method is used to ensure the stability of policy updates.

[0145] In another embodiment of the present application, a method for online reinforcement learning data augmentation based on a diffusion model comprises the following steps:

[0146] Step 1: Basic model construction and initialization.

[0147] First, a data augmentation module based on the conditional diffusion probability model is constructed. This module includes the denoising process and the denoising process, which is used to perform high-quality augmentation of trajectory data. Then, combined with the online reinforcement learning framework, the main part of reinforcement learning is constructed, including the actor network and the critic network. Finally, the above network structure is initialized. The specific steps include the following:

[0148] a) Construct the conditional diffusion model of the denoising process and the denoising process. The denoising process mainly uses the linear projection layer and the temporal U-Net architecture. The temporal U-Net architecture is mainly composed of residual blocks, which is used to efficiently process time series data. Specifically: First, the trajectory data τ is input, and the conditional diffusion model gradually introduces noise into the data through the denoising process. The denoising process follows the forward diffusion process q(x_(k+1) (τ)|x_k (τ)). Through this process, the original data loses details and expands the data space. Then, the denoising network of the temporal U-Net architecture is used for reverse denoising. The temporal U-Net architecture is mainly stacked with multiple convolutional residual blocks (depth: 6 layers). This structure can improve the network's expression ability and training stability, and gradually recover high-quality samples close to the real distribution from the noisy data. Construct an Actor network based on a convolutional neural network, a maximum pooling layer, and a fully connected layer, as well as a Critic network based on a fully connected layer. By introducing the reward guidance mechanism of the Critic network, that is, adding reward condition constraints to the denoising process, the generated trajectory data is ensured to be more in line with the direction of current strategy optimization in action selection. During the network update process, the Critic network is trained and updated based on virtual trajectory samples, while the Actor network is updated based on real trajectory samples.

[0149] b) Construct an Actor network mainly based on a convolutional neural network (CNN), a maximum pooling layer, and a fully connected layer, and a Critic network mainly based on a fully connected layer. The Critic network is used to evaluate the value of the strategy and generate reward guidance conditions for the denoising process, and the Actor network is used to optimize the behavior strategy.

[0150] Step 2: Data collection.

[0151] Use the current strategy to interact with the real environment, collect N real trajectory data, and store the data in the real experience playback pool B. A trajectory data {s t , a t , r t ,s t+1} t=0 T Mainly contains status t 、Action a t , Reward tand the next state s t+1 These data serve as experience samples for online reinforcement learning and form the basis for subsequent expansion and optimization. The specific steps include:

[0152] a) Reset the environment, initialize the environment state, and obtain the initial state s0.

[0153] b) Based on the current strategy π, from the current state s t Start by sampling an action a t .

[0154] c) The sampled action a t Act on the real environment and get the next state s t+1 and reward r t .

[0155] d) Repeat steps b to c until the predetermined trajectory length (i.e., the time step of trajectory expansion) is reached.

[0156] e) The sampled trajectory data samples {s t , a t , r t ,s t+1} t=0 T Stored in the experience replay pool B as experience samples for online reinforcement learning.

[0157] f) Repeat the above steps until the preset N trajectories are collected.

[0158] Step 3: Data augmentation.

[0159] Sample trajectory data τ = {s t , a t , r t ,s t+1} t=0 T, and input it into the conditional diffusion model for trajectory data augmentation. In image processing, the diffusion process is applied to all pixel values ​​in the image. Based on this intuitive idea, a similar diffusion process can naturally be applied to state and action modeling in trajectories in reinforcement learning. However, in the scenario of reinforcement learning, direct diffusion modeling of actions will face some practical problems: First, although the state in reinforcement learning is usually continuous, the expression of actions is more diverse and is discrete in many scenarios. Second, action sequences (for example, usually represented as joint torques) often have high-frequency characteristics, change more drastically and lack smoothness, which makes it more difficult to predict and model them. Based on these practical considerations, high-quality generation of trajectory data is achieved through the denoising process and the denoising process, and the optimization direction and effectiveness of the data are improved by combining the Critic network reward guidance mechanism. Specifically, only the state and reward are selected for diffusion modeling, which are defined as follows: x k (τ)=(s0, r0, s1, r1..., s T , r T ) k ; where k represents the time step in the forward process; x k (τ) represents the data after noise addition at the kth step. The diffusion process adds noise to the state-reward pair at each time step, and then gradually restores the original state-reward sequence from the Gaussian noise through the denoising process. In addition, in order to reasonably deduce actions in the trajectory generated by the diffusion model, an action generation method based on the inverse dynamics model is proposed. The diffusion model mainly models the state-reward pairs in the trajectory rather than directly diffusing the actions. However, relying solely on the states and rewards generated by the diffusion model is not enough to define the strategy. To this end, the inverse dynamics model is introduced. According to the two consecutive states (s t ,s t+1 ) deduce action a t , which is formally expressed as: a t =f ϕ (s t ,s t+1 );where f ϕ It is an inverse dynamics model obtained through neural network training, which can effectively predict the state s t Transfer to state s t+1 Required action a t This embodiment mainly uses online reinforcement learning trajectory data to update the inverse dynamics model f ϕ .

[0160] The part of constructing the conditional diffusion model based on the noisy data x(τ) mainly includes the following steps:

[0161] a) The noise adding process gradually introduces noise to gradually transform the distribution of the original trajectory data τ into a standard Gaussian distribution, thus achieving the expansion and modeling of the data space. The specific definition is as follows: Given a data sample x0(τ), the noise adding process maps the data sample x0(τ) into a noisy data x0(τ) by a predefined noise adding method. k (τ).x k (τ)=sqrt(a k ) x0(τ)+sqrt(1-a k ) Ξ,Ξ~N(0,1); where x k (τ) represents the data after the k-th step of noise addition, Ξ~N(0,1) is random Gaussian noise, a k is the noise weight that changes with the diffusion step k (usually using a predefined noise schedule). This process gradually introduces noise into the data, making the trajectory data distribution tend to the standard Gaussian distribution, so as to model in a wider data space.

[0162] b) The denoising process is done by using the trained diffusion model P θ (x k-1 (τ)|x k (τ)) gradually removes the noise in the data and generates high-quality and diverse trajectory data. Specifically, the denoising process is a reverse process defined as: k-1 (τ)=1 / sqrt(a k ){x k (τ)+(1-a k ) / sqrt(1-a k ) θ (x k (τ), k)}+µ k z, z~N(0,1); where x k-1 (τ) represents the trajectory data of the k-1th step denoising, Ξ θ (x k (τ), k) is a noise fraction estimation model parameterized by a neural network, which uses the trained denoising network to estimate the noise. k Control the randomness in the denoising process, z~N(0,1) is the additional random Gaussian noise. This process gradually restores high-quality trajectory data from Gaussian noise, thereby improving the sample efficiency of the model and the generalization of the Critic network.

[0163] c) To ensure that the generated data conforms to the optimization direction of the current strategy, the reward guidance condition of the Critic network is introduced in the denoising process. The Critic network is used to evaluate the state and action in the initial trajectory data, calculate the corresponding reward y(τ), and use it as the denoising process of the conditional guidance diffusion model. Then a conditional diffusion model P needs to be trained.θ (x k-1 (τ)|x k (τ), y) and adjust the formula in the diffusion model to adapt to the reward condition. This embodiment uses a classifier-free guidance mechanism to achieve conditional guidance, which is a guidance mechanism for conditional diffusion models that aims to enhance the relevance of generated data and conditional information while maintaining the diversity of generated data. Unlike traditional classifier guidance methods, classifier-free guidance does not require additional training of an independent classifier, but instead learns unconditional and conditional generated noise through the same model to achieve conditional guidance. The formula for adjusting noise is as follows: θ =Ξ θ (x k (τ), k)+γ(Ξ θ (x_ k (τ), y(τ), k)-Ξ θ (x_ k (τ), k)); where Ξ θ (x k (τ), k) represents the unconditional noise prediction, that is, the noise estimate without the condition y(τ). θ (x k (τ), y(τ), k) represents the noise estimate when there is conditional y(τ). γ is a control parameter, called the guidance scale, which is used to balance the contribution of unconditional noise and conditional noise. This guidance mechanism is used to extract high-likelihood trajectories in the trajectory dataset so that the generated trajectories are more in line with the conditional requirement y(τ) in terms of quality. Finally, the augmented data is placed in the virtual experience replay pool B' to provide additional data support for online reinforcement learning. These augmented data and actual environment interaction data are used for training and updating of the Critic network and strategy optimization of the Actor network, respectively, effectively improving the sample utilization efficiency, generalization ability and decision-making performance of the model, thereby achieving better performance in complex dynamic environments.

[0164] Step 4: Update the Actor-Critic network.

[0165] The augmented data generated in step 3 and the real environment interaction data are used for training and updating the Critic network and the Actor network respectively. The Critic network is mainly responsible for evaluating the value of the state-action pair under the current strategy and providing reward condition guidance for the denoising process of the diffusion model. The goal is to improve the accuracy and generalization ability of value evaluation and generate high-quality augmented data. The Actor network uses the feedback optimization strategy of the Critic network to enable it to continuously improve its decision-making performance in the actual data distribution. This part mainly includes the following steps:

[0166] a) Sampling and augmenting data to update the Critic network.

[0167] The update process of the critic network is achieved by minimizing the temporal difference (TD) error, mainly using the augmented data in step 3. The main update formula is as follows: L critic =E π [∑ t=0 H-1 (r t +bQ(s t+1 , a t+1 ))-Q(s t , a t )) 2 ]; where Q(st, at) is the state action value function output by the Critic network, rt represents the immediate reward at the current time step t, and b is the discount factor used to balance the contribution of current rewards and future rewards. Guided by augmented data, the Critic network can improve the fitting effect of the value function under scarce data or unseen conditions, ensuring that the generated augmented data provides effective feedback for network updates.

[0168] b) Sampling real experience data to update the Actor network.

[0169] The Actor network optimizes the strategy π at each time step through the policy gradient method, with the goal of maximizing the cumulative reward. Its main update formula is as follows: π =-E π [∑ t=0 H-1 log(π(a t |s t) )A t ]; where A t =Q(s t , a t )-V(s t ) is the advantage function, which indicates the degree of advantage of the current action. The Actor network uses the value evaluation results provided by the Critic network as the optimization basis, so that the generated strategy can continuously improve the performance on the real environment data. By optimizing the advantage function, the Actor network can focus on strengthening high-return actions and suppress inefficient decisions, thereby improving the performance of the strategy on the real data distribution. During the training process, the Actor network relies on the value evaluation output by the Critic network as a reference, and iterates and optimizes the strategy through actual experience data. Combined with the accurate valuation of the Critic network, the Actor network can gradually converge to the optimal strategy on the actual interaction data, ensuring good generalization and stability in a dynamic environment.

[0170] Step 5: Closed-loop process of strategy and data update.

[0171] After completing the update of the Critic network and the Actor network, the current strategy is used to interact with the environment again, and new experience trajectory data is collected and stored in the replay pool B, forming a closed loop of data augmentation and strategy optimization. The augmented data is sampled from the replay pool B' to strengthen the value evaluation of the Critic network, and the real data is sampled from the replay pool B for the strategy learning of the Actor network, and it is iterated and updated repeatedly until the training is completed.

[0172] This embodiment first interacts with the real environment to collect real trajectory data containing elements such as state, action and reward, and stores it in the experience replay pool as an empirical basic data set for online reinforcement learning. At the same time, the state and reward in the trajectory data are modeled by the conditional diffusion model. The diffusion model gradually transforms the data through the noise process, extending from the original data distribution to the standard Gaussian distribution; then, the high-quality state and reward data close to the real distribution are gradually restored from the Gaussian noise through the denoising process, thereby generating rich trajectory samples. In order to further improve the quality of the generated trajectory and the policy optimization effect, the reward guidance mechanism of the Critic network is introduced. By providing conditional constraints for the denoising process of the diffusion model, the generated trajectory data is more in line with the optimization goal of the current strategy. In addition, in order to solve the high-frequency characteristics and discontinuity problems of directly modeling actions, an action generation method based on an inverse dynamics model is constructed. After generating the state and reward, the diffusion model derives the corresponding action according to two consecutive states through the inverse dynamics model, thereby generating complete trajectory data. The inverse dynamics model is trained and updated through the trajectory data of online reinforcement learning to ensure that the generated action has practical feasibility and policy consistency. Finally, the generated high-quality augmented data is stored in the virtual experience replay pool and used together with the real trajectory data to update the Actor-Critic network: the Critic network optimizes the state-action value function based on the augmented data to improve the accuracy and generalization of value assessment; the Actor network updates the strategy based on the real data, relying on the value signal provided by the Critic network to optimize the strategy performance. Through the synergy of data augmentation, inverse dynamics action generation and strategy optimization, this embodiment realizes the closed-loop process of data augmentation and strategy update, fully utilizing the augmented data generated by the diffusion model and the real environment interaction data to form an efficient data replay and strategy optimization cycle, effectively improving the model generalization ability and decision-making performance of online reinforcement learning in complex dynamic environments.

[0173] This application can be applied to DMC scenarios, and DMC provides a series of simulation environments for continuous control tasks. The environment covers a variety of dynamic tasks and objects, such as robot balance, walker standing and walking, quadruped motion control, and cheetah running and other high-dimensional tasks. For these tasks, this application can complete control decisions through dynamic interaction in a simulation environment, and evaluate the adaptability and performance of the model under different tasks. In addition, this application can be widely used in drone racing scenarios, aiming to improve the performance of drones in complex tracks and high-speed dynamic environments through task decision-making and flight path planning. Specific tasks include: path planning: optimizing the flight trajectory of the drone on the racing track to ensure rapid obstacle avoidance and efficient completion of tasks; attitude control: dynamically adjusting the attitude and stability of the drone during high-speed flight. This application is also applicable to other dynamic control and decision-making task scenarios, including robot task control, environmental simulation and behavior decision optimization, providing reliable task execution and performance evaluation support.

[0174] The present invention realizes the automatic generation and optimization of high-quality trajectory data by combining diffusion model and reinforcement learning. A complete data enhancement framework is constructed, including key links such as feature extraction, diffusion model processing, constraint fusion, parameter optimization, trajectory generation and screening, which can adaptively handle complex dynamic environments, generate diversified trajectory data that meet physical constraints and task requirements, and improve the training efficiency and performance of the reinforcement learning system. In particular, when dealing with high-dimensional and strongly constrained robot control tasks, the present invention exhibits excellent performance and stability. Through multi-level feature processing and constraint fusion, the physical rationality and task relevance of the generated data are ensured, providing a wealth of training samples for the reinforcement learning system and accelerating the convergence process of strategy learning. At the same time, the modular design of the present invention makes it have good scalability and adaptability, and can be flexibly applied to different types of control tasks and scenarios.

[0175] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. An online reinforcement learning data augmentation method based on a diffusion model, characterized in that: The steps include: S1, read the original trajectory data, perform dynamic feature extraction, physical consistency inspection and feature enhancement processing respectively, and generate enhanced feature data; The original trajectory data contains state, action, reward and next state information; The original trajectory data is applied to the drone racing scene. The specific tasks include: path planning: optimizing the flight trajectory of the drone on the racing track to ensure rapid obstacle avoidance and efficient completion of the task; attitude control: dynamically adjusting the attitude and stability of the drone during high-speed flight; S2, inputting the enhanced feature data into the pre-configured global diffusion model and the local diffusion model respectively to generate a global structure representation and a local detail representation; adopting a hierarchical integration method to fuse the global structure representation and the local detail representation to obtain a unified hierarchical representation data; S3, based on the pre-stored physical constraint parameters and task constraint parameters, perform constraint encoding to obtain constraint encoding data; fuse the constraint encoding data with the hierarchical representation data to generate constraint perception representation data; S4, obtaining current strategy parameters, performing strategy evaluation based on the current strategy parameters and pre-stored environmental state data, and obtaining evaluation results; adaptively adjusting the constraint perception representation data according to the evaluation results, and obtaining adjusted representation data; S5. Based on the adjusted representation data, perform modal analysis to obtain analysis results; based on the analysis results, generate diversified trajectory data to obtain a candidate trajectory set; S6. Based on the candidate trajectory set, quality assessment is performed to obtain a quality assessment result; based on the quality assessment result, trajectory screening is performed through a multi-objective optimization method to obtain a screened trajectory data set; S7: storing the filtered trajectory data set into a data pool and updating the strategy parameters to obtain updated strategy parameters.

2. The online reinforcement learning data augmentation method based on the diffusion model according to claim 1, characterized in that: Step S1 is further as follows: S11, reading original trajectory data from a pre-stored data storage device; performing data standardization processing based on the original trajectory data to obtain standardized trajectory data; wherein the original trajectory data includes state, action, reward and next state information; S12. Based on the standardized trajectory data, the self-attention mechanism is used to calculate the correlation in the time dimension to obtain a temporal correlation matrix; based on the temporal correlation matrix and the standardized trajectory data, a feature extraction operation is performed to generate a dynamic feature vector; S13, based on the dynamic characteristic vector and the pre-stored physical constraint parameters, the Lagrange equation is used to perform physical rationality verification calculations and output effective dynamic characteristics; S14, performing wavelet transform processing on the effective dynamic features to obtain multi-scale feature data; based on the multi-scale feature data, extracting and combining local features and global features to generate enhanced feature data.

3. The online reinforcement learning data augmentation method based on the diffusion model according to claim 2, characterized in that: Step S2 is further as follows: S21, based on the enhanced feature data, a coarse-grained diffusion processing method is used to model and analyze the overall structure of the trajectory to generate global structure representation data; S22. Based on the global structure representation data, a fine-grained diffusion model is constructed; based on the fine-grained diffusion model, the local action sequence and state transition of the trajectory are processed by the local temporal association analysis method to generate local detail representation data; S23. Adopt an adaptive weighted fusion method to integrate the global structure representation data and the local detail representation data to generate unified hierarchical representation data.

4. The online reinforcement learning data augmentation method based on the diffusion model according to claim 3 is characterized in that: Step S3 is further as follows: S31, generating constraint coding data by a constraint embedding calculation method based on pre-stored physical constraint parameters and task constraint parameters; S32. Use a conditional normalization processing method to perform a fusion operation on the constrained coded data and the hierarchical representation data to generate constrained perceived representation data.

5. The online reinforcement learning data augmentation method based on the diffusion model according to claim 4, characterized in that: Step S4 is further as follows: S41, obtaining current strategy parameters, and generating adaptive parameter data by calculating through a strategy performance analysis method and an environment dynamic feature analysis method based on the current strategy parameters and pre-stored environment state data; S42. Based on the adaptive parameter data, a dynamic update method is used to adaptively adjust the constraint-aware representation data to generate adjusted representation data.

6. The online reinforcement learning data augmentation method based on the diffusion model according to claim 5, characterized in that: Step S5 is further as follows: S51, based on the adjusted representation data, identifying and processing the behavior pattern through a cluster analysis method to generate modal description data; S52. Based on the modal description data, a variational inference method is used to generate diversified trajectory data to obtain a set of candidate trajectories.

7. The online reinforcement learning data augmentation method based on the diffusion model according to claim 6, characterized in that: Step S6 is further as follows: S61, based on the candidate trajectory set, processing is performed by a physical feasibility calculation method and a task relevance calculation method to generate quality score data; S62. Based on the quality score data and the candidate trajectory set, a multi-objective optimization method is used to select the optimal trajectory sample to generate a screened trajectory data set.

8. The online reinforcement learning data augmentation method based on the diffusion model according to claim 7, characterized in that: Step S7 is further as follows: S71, storing the screened trajectory data set into an enhanced data pool, combining the data in the enhanced data pool with the data in the pre-stored real data pool to generate a combined data set; S72. Based on the combined data set, the trust domain policy optimization method is used to update the policy network and generate updated policy parameters.

9. The online reinforcement learning data augmentation method based on the diffusion model according to claim 8, characterized in that: Step S11 is further as follows: S111, reading original trajectory data from a pre-stored data storage device; segmenting the original trajectory data by sliding a fixed-size time window along the time axis step by step to generate segmented trajectory data; S112, based on the segmented trajectory data, calculating the local density and relative distance between each data point and its neighboring points; judging and removing abnormal points according to preset thresholds of the local density and relative distance, and generating preprocessed trajectory data; S113, decomposing the preprocessed trajectory data into a predetermined number of recursive subsequences, extracting and combining time series features from each recursive subsequence to generate time series feature data; S114, calculating the mean and standard deviation of the time series feature data; based on the mean and standard deviation, performing zero mean conversion and variance normalization processing on the time series feature data to generate standardized trajectory data.

10. The online reinforcement learning data augmentation method based on the diffusion model according to claim 8, characterized in that: Step S23 is further as follows: S231, calculating the temporal correspondence between the global structure representation data and the local detail representation data by a dynamic time warping algorithm, and generating alignment feature data; S232, based on the aligned feature data, calculating the information contribution of features at different levels through information entropy, and generating feature weight data; S233, based on the feature weight data and the aligned feature data, calculating the interaction relationship between the features through the attention mechanism to generate interaction feature data; S234. Based on the interactive feature data, multi-level features are adaptively combined through a residual connection network to generate unified hierarchical representation data.

Images