Adaptive memory-augmented imitation learning approach for robotic control tasks

Abstract

The present invention discloses an adaptive memory-enhanced imitation learning method for robot control tasks, including standardizing original multimodal data and extracting temporal features to construct hierarchical feature representation; establishing a hierarchical structure including short-term memory, working memory and long-term memory, evaluating memory importance based on usage frequency, task relevance and novelty, and dynamically updating memory content; retrieving relevant memory according to the current state and performing multimodal feature fusion; calculating shaping rewards based on neighbor sample search; and optimizing strategy network parameters through multi-loss function combination. The present invention improves the learning efficiency and execution performance of the robot control system through adaptive memory management and multimodal feature fusion mechanism.

Description

Technical Field

[0001] The invention belongs to the field of robot control and imitation learning, in particular to an adaptive memory enhancement imitation learning method for robot control tasks. Background Art

[0002] Robot control has a wide range of applications in manufacturing, medical care, services and other fields. Imitation learning, as an important robot control method, enables robots to learn to complete complex tasks by observing and imitating the actions of human experts. In actual application scenarios, robots often need to process heterogeneous data from multiple sensors such as vision, force, touch and position, and are required to make accurate control decisions in real time in a dynamically changing environment. Therefore, developing adaptive and robust robot control algorithms is of great significance to improving the robot's ability to perform tasks in complex environments.

[0003] At present, researchers have proposed a variety of robot control methods based on imitation learning. Behavior Cloning trains control strategies by directly learning the state-action mapping of expert demonstrations, but it is prone to error accumulation problems. Inverse Reinforcement Learning attempts to infer the reward function from expert demonstrations and then optimize the control strategy through reinforcement learning, but it has high computational complexity and is difficult to process multimodal data. Memory-based Methods assist decision-making by storing and retrieving historical experience, but existing methods often use fixed memory structures and lack dynamic management mechanisms for memory content. In addition, some studies have attempted to combine attention mechanisms and multimodal fusion techniques to improve control performance, but these methods still face challenges in dealing with long temporal dependencies and cross-modal interactions.

[0004] The existing technical solutions have the following specific problems: First, the inconsistent sampling frequency of multimodal data makes it impossible to accurately align features, affecting the accuracy of state representation; second, the memory module uses a fixed storage strategy and cannot dynamically adjust the importance of memory according to task requirements, resulting in inefficient use of memory resources; third, the traditional memory retrieval method does not consider the temporal association between task context and memory items, and the retrieval results are not sufficiently relevant to the current task; fourth, the existing reward shaping method is mainly based on a single distance metric, which makes it difficult to accurately evaluate the similarity of state-action pairs; finally, the weights between various learning objectives in the strategy optimization process are fixed, and the optimization direction cannot be adaptively adjusted according to the learning progress. These technical problems limit the performance and adaptability of robot control systems and require the development of more advanced solutions. Summary of the invention

[0005] The purpose of the invention is to provide an adaptive memory-enhanced imitation learning method for robot control tasks, in order to solve at least one technical problem existing in the prior art.

[0006] The technical solution is an adaptive memory-enhanced imitation learning method for robot control tasks, comprising the following steps:

[0007] S1. Obtaining original multimodal data, standardizing the original multimodal data to obtain standardized data; based on the standardized data, sampling and feature extraction are performed using a preset time window to obtain a time-series alignment feature; based on the time-series alignment feature, an adaptive weighted hierarchical method is used to construct a feature hierarchy to obtain a hierarchical feature representation; wherein the original multimodal data includes visual data, force feedback data, and position data;

[0008] S2. Based on the hierarchical feature representation, a hierarchical structure including short-term memory, working memory and long-term memory is constructed to obtain an initialized memory structure; based on the hierarchical feature representation, a memory importance index is calculated to obtain a memory importance score; based on the memory importance score, the initialized memory structure is dynamically updated to obtain an updated memory structure;

[0009] S3, obtaining the original current state, normalizing the original current state to obtain the standardized current state; calculating the correlation score between the standardized current state and the updated memory structure; performing memory retrieval based on the correlation score to obtain a multimodal retrieval memory; adaptively fusing the multimodal retrieval memory to obtain an integrated memory representation;

[0010] S4, based on the standardized current state, the integrated memory representation and the pre-stored current action, perform a neighbor sample search on the current state-action pair to obtain a neighbor sample set; based on the neighbor sample set and the pre-stored original reward, calculate the shaping reward value to obtain the shaping reward;

[0011] S5. Based on the shaping reward and integrated memory representation, a predetermined number of loss function components are constructed and combined to obtain a total loss function; based on the total loss function and the preset initial strategy parameters, the strategy network parameters are updated to obtain updated strategy parameters.

[0012] Beneficial effects: The present invention provides accurate memory support through multi-level data processing, adaptive memory management, intelligent memory retrieval, optimized reward calculation and efficient strategy learning, and realizes efficient memory management and efficient learning and precise execution of robot control tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of the present invention.

[0014] Figure 2This is a flow chart of step S1 of the present invention.

[0015] Figure 3 This is a flow chart of step S2 of the present invention.

[0016] Figure 4 This is a flow chart of step S3 of the present invention.

[0017] Figure 5 This is a flow chart of step S4 of the present invention.

[0018] Figure 6 This is a flow chart of step S5 of the present invention. DETAILED DESCRIPTION

[0019] like Figure 1 As shown, the present application proposes an adaptive memory-enhanced imitation learning method for robot control tasks, comprising the following steps:

[0020] S1. Obtaining original multimodal data, standardizing the original multimodal data to obtain standardized data; based on the standardized data, sampling and feature extraction are performed using a preset time window to obtain a time-series alignment feature; based on the time-series alignment feature, an adaptive weighted hierarchical method is used to construct a feature hierarchy to obtain a hierarchical feature representation; wherein the original multimodal data includes visual data, force feedback data, and position data;

[0021] S2. Based on the hierarchical feature representation, a hierarchical structure including short-term memory, working memory and long-term memory is constructed to obtain an initialized memory structure; based on the hierarchical feature representation, a memory importance index is calculated to obtain a memory importance score; based on the memory importance score, the initialized memory structure is dynamically updated to obtain an updated memory structure;

[0022] S3, obtaining the original current state, normalizing the original current state to obtain the standardized current state; calculating the correlation score between the standardized current state and the updated memory structure; performing memory retrieval based on the correlation score to obtain a multimodal retrieval memory; adaptively fusing the multimodal retrieval memory to obtain an integrated memory representation;

[0023] S4, based on the standardized current state, the integrated memory representation and the pre-stored current action, perform a neighbor sample search on the current state-action pair to obtain a neighbor sample set; based on the neighbor sample set and the pre-stored original reward, calculate the shaping reward value to obtain the shaping reward;

[0024] S5. Based on the shaping reward and integrated memory representation, a predetermined number of loss function components are constructed and combined to obtain a total loss function; based on the total loss function and the preset initial strategy parameters, the strategy network parameters are updated to obtain updated strategy parameters.

[0025] like Figure 2 As shown, according to one aspect of the present application, step S1 is further:

[0026] S11. Obtain original multimodal data, calculate the mean and standard deviation of each type of data in the original multimodal data; subtract the corresponding mean from each type of data in the original multimodal data and divide the result by the corresponding standard deviation to obtain standardized data; wherein the original multimodal data includes visual data, force feedback data and position data, and the standardized data includes standardized visual data, force feedback data and position data.

[0027] S12. Based on a preset time window, sliding sampling is performed on the standardized data to obtain sampled data; wavelet coefficients are calculated for the sampled data in each window to extract time-frequency features at different scales; the time-frequency features extracted from each window are aligned in the time dimension to obtain time series alignment features; wherein the time series alignment features include visual features, force feedback features and position features.

[0028] S13. Based on the time-series alignment features, calculate the task complexity and determine the hierarchical depth; construct a feature clustering distance threshold for each layer of the hierarchical depth; based on the feature clustering distance threshold, construct a feature hierarchy for the time-series alignment features in a bottom-up order; update the cluster center at each layer of the feature hierarchy using adaptive weights to obtain a hierarchical feature representation. The hierarchical feature representation includes features of d levels from the lowest feature representation H1 to the highest feature representation Hd.

[0029] In one embodiment of the present application, the original multimodal data M = {Mv, Mf, Mp} is obtained, where Mv represents visual data, Mf represents force feedback data, and Mp represents position data; each modal data is normalized: standardized data M' = {M'v, M'f, M'p}; M'i = (Mi - μi) / σi, where i∈{v, f, p}, μi is the mean, and σi is the standard deviation. Based on the standardized data M', a time window W is used for sliding sampling, with a window size of τ; a wavelet transform is applied to each time window for feature extraction, and a time alignment feature F = {Fv, Ff, Fp} is output. Based on the time alignment feature F, an adaptive weighted hierarchical clustering (AWHC) algorithm is used to construct a feature hierarchy, and the hierarchical depth d is automatically adjusted according to the complexity of the task; the hierarchical feature representation H = {H1, H2, ..., Hd} is output.

[0030] In another embodiment of the present application, the time-frequency feature extraction method is as follows: wavelet coefficient C(a, b) = (1 / sqrt(a))·∫x(t)·ψ((tb) / a)dt; time-frequency feature F(τ) = Σ(wi·|C(ai, τ)| 2); where a is the scale parameter, b is the translation parameter, x(t) is the input signal, ψ(t) is the wavelet basis function; τ is the time window position, wi is the feature weight, satisfying Σwi = 1; feature alignment loss L = Σ||F(τ) - F(τ+1)|| 2 + λ·Σ||ΞF(τ) / Ξτ|| 2 ; λ is the smoothing term coefficient, Ξ is the partial derivative.

[0031] Feature hierarchy construction method: hierarchy depth d = ⌈log2(Cc·Se / Mi)⌉; clustering threshold sequence T(l) =T0·exp(-β·l); hierarchy node stability Si = (1 / ni)·Σexp(-||xi - μi|| 2 / σi 2 );Cc is the conditional complexity, Se is the sample entropy, Mi is the mutual information; T0 is the initial threshold, β is the attenuation coefficient, l is the hierarchical index; ni is the number of samples contained in node i, xi is the sample in the node, μi is the node center, σi is the node variance; representative index Ri = Si·exp(-λ·Di); Di is the minimum distance to the adjacent node.

[0032] This embodiment realizes the effective integration and hierarchical representation of multimodal data through the standardized processing of multimodal data and the extraction of time-series alignment features, combined with the adaptive weighted hierarchical method. Specifically, the original visual, force feedback and position data are first standardized to eliminate the dimensional differences between different modal data, so that data of different modalities can be compared and fused at the same scale; then, the time window is used for sampling and feature extraction, and the time-frequency features at different scales are obtained by calculating the wavelet coefficients, which ensures the alignment of multimodal data in the time dimension and avoids the problem of feature misalignment caused by inconsistent sampling frequency; finally, the adaptive weighted hierarchical method is used to construct the feature hierarchy, and the importance of features at different levels is dynamically adjusted through the adaptive weight update mechanism, so that the feature representation can adaptively reflect the complexity of the task and the intrinsic structure of the data, thereby providing a high-quality feature representation basis for subsequent memory storage and strategy learning.

[0033] According to one aspect of the present application, step S11 is further:

[0034] S111, obtaining original multimodal data, and using an adaptive sampling frequency to perform time-series sampling on visual data, force feedback data, and position data in the original multimodal data to obtain sampled data; performing median filtering on the sampled data to remove outliers to obtain filtered multimodal data;

[0035] S112, constructing a sliding time window based on the filtered multimodal data; calculating the local mean, local standard deviation, skewness and kurtosis of the data in each window in the sliding time window to obtain local statistical features; calculating the global mean, global standard deviation, global skewness and global kurtosis based on the filtered multimodal data to obtain global statistical features; using an exponentially weighted average method to fuse local statistical features with global statistical features to obtain fused statistical features;

[0036] S113. Based on the fused statistical features, an adaptive scaling factor is calculated; the local standard deviation is weightedly combined with the global standard deviation to obtain a weighted standard deviation; based on the adaptive scaling factor, the filtered multimodal data is subtracted from the local mean and divided by the weighted standard deviation, and nonlinear mapping is performed through a hyperbolic tangent function to obtain standardized data.

[0037] In one embodiment of the present application, an adaptive sampling frequency and data normalization method: adaptive sampling frequency f(t) = f0·exp(-k·σ(t)); standardized data M'(t) = tanh((M(t)-μl) / (γ·σw)); wherein f0 is the basic sampling frequency, k is the adjustment coefficient, and σ(t) is the real-time signal variance; M(t) is the original data, μl is the local mean, σw is the weighted standard deviation, and γ is the adaptive scaling factor; weighted standard deviation σw = w1·σl + w2·σg; σl is the local standard deviation, σg is the global standard deviation, w1 and w2 are weight coefficients and satisfy w1 + w2 = 1; adaptive scaling factor γ = 1 + λ·|sk|·|ku|; sk is the skewness, ku is the kurtosis, and λ is the scaling factor.

[0038] This embodiment achieves high-quality standardized processing of multimodal data through adaptive sampling and statistical feature fusion. First, adaptive sampling frequency is used to perform time-series sampling of visual, force feedback and position data, and the sampling frequency is dynamically adjusted according to the characteristics of different modal data to ensure that the sampled data can fully reflect the characteristic changes of each modality; then median filtering is applied to the sampled data to remove outliers, thereby improving the reliability of the data; then local statistical features are calculated through a sliding time window, including local mean, standard deviation, skewness and kurtosis, and local statistical features are fused with global statistical features using an exponential weighted average method to obtain a more comprehensive statistical feature representation; finally, an adaptive scaling factor is calculated based on the fused statistical features, and the data is standardized in combination with the local mean and weighted standard deviation. The nonlinear mapping of the hyperbolic tangent function further enhances the data expression ability, so that the standardized data not only retains the important features of the original data, but also has good numerical stability and comparability.

[0039] According to one aspect of the present application, step S13 is further:

[0040] S131, obtaining a time alignment feature F, calculating the sample entropy Se, mutual information Mi and conditional complexity Cc of the time alignment feature, and performing a weighted combination of these three indicators using an adaptive weight network to obtain a task complexity indicator D; wherein the sample entropy Se represents the uncertainty of the data, the mutual information Mi represents the correlation between the features, and the conditional complexity Cc represents the conditional dependency of the features;

[0041] S132, obtaining the task complexity index D and the timing alignment feature F, constructing a feature distance matrix M, applying the spectral analysis method to the feature distance matrix to calculate the eigenvalue distribution, determining the optimal clustering threshold sequence according to the attenuation curve of the eigenvalue, and obtaining an adaptive clustering threshold vector T; wherein the feature distance matrix uses the Mahalanobis distance to measure the similarity between features;

[0042] S133, obtaining a time series alignment feature F and an adaptive clustering threshold vector T, performing feature clustering on the time series alignment feature using a hierarchical bidirectional propagation method, constructing a hierarchical structure from bottom to top during forward propagation, and optimizing clustering boundaries from top to bottom during reverse propagation to obtain an initial hierarchical structure L;

[0043] S134. Obtain an initial hierarchical structure L, calculate the stability index Si and representative index Ri of each node, split or merge the nodes in the initial hierarchical structure whose stability index is lower than a threshold, update the feature representation of the nodes with a low representative index, and obtain a hierarchical feature representation H; wherein the stability index Si is calculated based on the consistency of samples within the node, and the representative index Ri is calculated based on the distinguishability of the node and its neighboring nodes.

[0044] This embodiment realizes the construction of adaptive hierarchical features based on task complexity. First, the complexity of the task is evaluated by calculating the three indicators of sample entropy, mutual information and conditional complexity, and these indicators are weighted and combined using an adaptive weight network to obtain an accurate task complexity indicator; then, a feature distance matrix is ​​constructed and a spectral analysis method is applied to determine the optimal clustering threshold sequence by analyzing the decay curve of the eigenvalue, thereby ensuring the rationality of feature clustering; then, a hierarchical bidirectional propagation method is used for feature clustering, and a hierarchical structure is constructed from the bottom up during the forward propagation process, and the clustering boundaries are optimized from the top down during the reverse propagation process, thereby realizing the effective organization of features; finally, by calculating the stability index and representativeness index of the node, unstable nodes are split or merged, and the feature representation of nodes with insufficient representation is updated, thereby constructing a stable and representative hierarchical feature representation structure, which provides a reliable feature basis for subsequent memory storage and retrieval.

[0045] like Figure 3 As shown, according to one aspect of the present application, step S2 is further:

[0046] S21, distribute the hierarchical feature representation to short-term memory, working memory and long-term memory according to storage capacity and time attributes, and obtain an initialized memory structure; wherein the capacity of the short-term memory STM is cs and the storage time is ts, the capacity of the working memory WM is cw and the storage time is tw, and the capacity of the long-term memory LTM is cl and the storage time is tl;

[0047] S22. Take each feature layer in the hierarchical feature representation as a memory item to be evaluated; based on the initialized memory structure, calculate the usage frequency, task relevance and novelty score of each memory item to be evaluated; perform weighted summation on the usage frequency, task relevance and novelty score to obtain a memory importance score; wherein the usage frequency, task relevance and novelty score are weighted summation according to weights α, β, γ, and the calculation formula of the memory importance score I is I = α·U + β·R +γ·N, where U is the usage frequency, R is the task relevance, and N is the novelty score;

[0048] S23. Based on the initialized memory structure, features whose memory importance scores are greater than a preset high threshold are upgraded, features whose memory importance scores are less than a preset low threshold are downgraded or forgotten, and features whose memory importance scores are between a preset low threshold and a preset high threshold are kept at the current level to obtain an updated memory structure.

[0049] In one embodiment of the present application, based on the hierarchical feature representation H, a three-layer memory structure is constructed: short-term memory STM: capacity cs, storage time ts; working memory WM: capacity cw, storage time tw; long-term memory LTM: capacity cl, storage time tl; output initialization memory structure M = {STM, WM, LTM}. Based on the new memory item m and the current memory state M, a memory importance index I(m) is constructed: I(m) = α·U(m) + β·R(m) + γ·N(m), where U(m) is the frequency of use; R(m) is the task relevance, and N(m) is the novelty score; output memory importance score I, α, β, γ are weight coefficients. Obtain the memory importance score I and the current memory state M, and decide whether to keep the memory item based on the importance threshold θ: I >θh: upgrade to a higher level memory; θl< I <θh: keep the current layer; I < θl: downgrade or forget; output the updated memory structure M'. Where θl is the preset low threshold, and θh is the preset high threshold.

[0050] In another embodiment of the present application, a memory importance score calculation method is as follows: memory importance score I = α·U + β·R + γ·N; wherein usage frequency index U = exp(-λt·Δt)·log(1+v); λt is the time decay coefficient, Δt is the time interval, and v is the number of visits; task relevance index R = Σ(ai·Mi) / (||a||·||M||); ai is the attention weight, and Mi is the mutual information matrix element; novelty score N = 1 - exp(-W(p1, p2) / σ); W(p1, p2) is the Wasserstein distance between feature distributions p1 and p2, and σ is a smoothing factor; α, β, and γ are adaptive weight coefficients, which are dynamically generated by the weight network based on historical data to satisfy α+β+γ=1.

[0051] This embodiment establishes a hierarchical memory structure with short-term memory, working memory and long-term memory, and realizes efficient management and optimized storage of memory content through memory importance evaluation and dynamic update mechanism. First, the features are allocated to different memory levels according to storage capacity and time attributes, ensuring that the memory structure can maintain both rapid response and long-term memory capabilities; then, by calculating the indicators of three dimensions, namely, frequency of use, task relevance and novelty, the importance of each memory item is comprehensively evaluated, so that the memory system can retain the most valuable information; finally, based on the memory importance score, dynamic updates are performed, high-importance features are upgraded and saved, and low-importance features are downgraded or forgotten, thus realizing the dynamic allocation and optimal utilization of memory resources and improving the storage efficiency and information value density of the memory system.

[0052] According to one aspect of the present application, step S22 is further:

[0053] S221, based on the hierarchical feature representation, calculate the time decay coefficient; based on the initialized memory structure, calculate the access frequency vector; exponentially combine the time decay coefficient and the access frequency vector to obtain a usage frequency index; wherein the time decay coefficient decays exponentially with the time interval, and the access frequency vector records the number of historical accesses to each memory item;

[0054] S222. Based on the hierarchical feature representation and the initialized memory structure, a task context vector is constructed; the mutual information matrix between the hierarchical feature representation and the task context vector is calculated; the mutual information matrix is ​​weightedly aggregated using an attention mechanism to obtain a task relevance index; wherein the task context vector is obtained by encoding the current task state, and the mutual information matrix represents the relevance between the feature and the task;

[0055] S223, calculating a first probability density distribution of hierarchical feature representation; calculating a second probability density distribution of features stored in the initialized memory structure; using Wasserstein distance to measure the difference between the first probability density distribution and the second probability density distribution to obtain a novelty score; wherein the probability density distribution is obtained by a kernel density estimation method, and the Wasserstein distance takes into account the geometric structure of the feature space;

[0056] S224. Based on the usage frequency index, task relevance index and novelty score, an adaptive weight network is constructed to obtain a dynamic weight coefficient; the usage frequency index, task relevance index and novelty score are weighted and combined with the corresponding dynamic weight coefficient to obtain a memory importance score. The input of the adaptive weight network is the historical change trend of the three indicators, and the adaptive weight network updates the weight coefficient in real time through the gradient descent method.

[0057] This embodiment constructs a memory importance calculation mechanism based on multi-dimensional evaluation. First, the usage frequency index is calculated through the exponential combination of the time decay coefficient and the access frequency vector, realizing the dynamic tracking of the usage of memory items; then the task context vector is constructed, the mutual information matrix between the feature and the task context is calculated, and weighted aggregation is performed through the attention mechanism to obtain an accurate task relevance index; then the kernel density estimation method is used to calculate the probability density distribution of the feature, and the difference between the new feature and the stored feature is measured using the Wasserstein distance to obtain an objective novelty score; finally, the weight coefficients of the three indicators are dynamically adjusted through the adaptive weight network to realize the comprehensive evaluation of memory importance, ensure the retention of important memory and the timely removal of useless memory, and improve the storage efficiency and information quality of the memory system.

[0058] like Figure 4 As shown, according to one aspect of the present application, step S3 is further:

[0059] S31, obtaining the original current state collected in real time, standardizing the original current state to obtain the standardized current state; calculating the state distance between the standardized current state and each memory item in the updated memory structure; substituting the state distance into the preset distance scaling temperature parameter for exponential transformation to obtain the correlation score; wherein the calculation formula of the correlation score C is C = exp(-D / T), exp( ) is an exponential function, D is the state distance, and T is the distance scaling temperature parameter;

[0060] S32, based on the preset attention temperature parameter, normalize the relevance score to generate the attention weight of memory retrieval; based on the attention weight, retrieve the relevant memory items from the updated memory structure to obtain the multimodal retrieval memory; wherein the calculation formula of the attention weight is A = softmax(C / η), η is the attention temperature parameter, and C is the relevance score;

[0061] S33. Based on the multimodal retrieval memory and the standardized current state, the fusion weights of the visual features, force feedback features and position features are calculated using a fully connected neural network; based on the fusion weights, the memory features in the multimodal retrieval memory are weighted and summed to obtain an integrated memory representation, wherein the integrated memory representation includes the fusion result of the multimodal memory information.

[0062] In one embodiment of the present application, based on the current state s and the memory structure M', the context relevance score C(m, s) is calculated: C(m, s) = exp(-D(m, s) / T); where D(m, s) is the state distance metric; T is the temperature parameter that controls the relevance sensitivity; the relevance score C is output. Based on the relevance score C and the memory structure M', the attention mechanism is used for memory retrieval: A(m) = softmax(C(m, s) / η), where η is the attention temperature parameter, and the retrieval memory R is output. Based on the retrieval memory R, ​​adaptive weight fusion is used: w = FCN(R, s), where FCN is a fully connected neural network; the integrated memory representation R' = Σ(wi·Ri).

[0063] In another embodiment of the present application, the state correlation calculation method is: state distance D(s, m) = sqrt(Σ(λi·(si - mi) 2 )); correlation score C = exp(-D / T)·(1 + α·cos(θ)); where s is the current state, m is the memory term, λi is the feature dimension weight; T is the temperature parameter, which is obtained by gradient descent optimization as T = T0·exp(-η·ΞL / ΞT); θ is the state vector angle, α is the angle modulation coefficient; L is the retrieval loss function, and T0 is the initial temperature parameter.

[0064] Attention weight calculation method: Attention weight A = softmax(Q K T / sqrt(d) + M); memory retrieval probability P(m|s) = Σ(Ai·exp(-Di / η)); where Q is the query matrix, K is the key matrix, d is the feature dimension, and M is the mask matrix; Di is the memory item distance, η is the attention temperature parameter; consistency verification score V = min(cos(mt, mt+1)); mt is the temporal memory item.

[0065] Multimodal feature fusion method: modal fusion weight W(m) = σ(FC(concat(Ev, Ef, Ep))); complementary matrix R(i, j) = (1 - |ρij|)·exp(-||Ei - Ej|| 2 / σ 2 ); integrated features R' = Σ(G(m)·W(m)·E(m)); where Ev, Ef, Ep are the enhanced representations of vision, force feedback and position features respectively, FC is the fully connected layer, σ is the sigmoid function; ρij is the correlation coefficient of features i and j, σ is the Gaussian kernel width; G(m) is the gating function G(m) = sigmoid(FC(E(m))); E(m) is the enhanced feature, m∈{v, f, p} represents the modality type.

[0066] This embodiment realizes intelligent memory retrieval and multimodal information fusion based on the current state, and improves the accuracy and efficiency of memory retrieval through correlation calculation and adaptive fusion. Specifically, the current state is first standardized to ensure that it is in the same feature space as the stored memory features; then the correlation score between the standardized state and each memory item in the memory structure is calculated, and the temperature parameter is used for exponential transformation to make the correlation distribution more reasonable; then memory retrieval is performed based on the correlation score, and important memory information is highlighted through the attention mechanism; finally, the retrieved multimodal memory information is adaptively fused, and the fusion weights of different modal features are calculated through a fully connected neural network, which realizes the effective integration of multimodal information and provides a more comprehensive and accurate memory representation.

[0067] According to one aspect of the present application, step S32 is further:

[0068] S321. Based on the correlation score and the updated memory structure, an adaptive temperature adjustment network is constructed; based on the adaptive temperature adjustment network, a temperature parameter is optimized by a gradient descent method to obtain an attention temperature parameter; based on the attention temperature parameter, a temperature scaling is performed on the correlation score to obtain a scaled correlation score; wherein the input of the adaptive temperature adjustment network is the distribution characteristics of the correlation score and the density characteristics of the memory structure; the value range of the attention temperature parameter is (0, 1], which is used to control the smoothness of the correlation distribution;

[0069] S322, dividing the scaled relevance score by the attention temperature parameter and inputting the result into the Softmax function to calculate the attention weight distribution; applying the adaptive gating mechanism to the attention weight distribution to filter and obtain the filtered attention weight; wherein, the calculation formula of the filtered attention weight is A = softmax(Cs / η), the gating threshold is dynamically adjusted according to the entropy value of the weight distribution, Cs is the scaled relevance score, and η is the attention temperature parameter;

[0070] S323. Based on the post-screening attention weights, weighted sampling is performed on the memory items in the updated memory structure to obtain sampled memory items; consistency verification is performed on the sampled memory items to obtain verified memory items; and importance is sorted based on the verified memory items to obtain multimodal retrieval memory. The consistency verification is performed based on the temporal relationship and semantic similarity between the memory items.

[0071] This embodiment implements an efficient memory retrieval mechanism based on adaptive temperature regulation. First, an adaptive temperature regulation network is constructed. The network analyzes the distribution characteristics of the correlation score and the density characteristics of the memory structure, and uses the gradient descent method to dynamically optimize the temperature parameters to achieve adaptive regulation of the correlation distribution; then the scaled correlation score is normalized by the temperature parameter, and the attention weight distribution is calculated in combination with the Softmax function. The attention weight is screened through the adaptive gating mechanism to ensure that the retrieval process focuses on the most relevant memory information; finally, the memory structure is weighted sampled according to the screened attention weight, and the consistency verification of the temporal relationship and semantic similarity is used to ensure the quality and reliability of the retrieved memory. This adaptive temperature regulation and multiple verification mechanism improves the accuracy and efficiency of memory retrieval, allowing the system to quickly locate and obtain the most relevant historical experience.

[0072] According to one aspect of the present application, step S33 is further:

[0073] S331. Obtain the retrieval memory R and the standardized current state s', construct a three-layer feature fusion network, pre-process the visual features, force feedback features and position features in the retrieval memory through the feature enhancement layer respectively, calculate the complementary matrix between the features, and obtain the enhanced feature representation E; wherein, the feature enhancement layer adopts a residual connection structure, and the complementary matrix represents the degree of information complementarity of different modal features.

[0074] S332. Obtain the standardized current state s', build a multi-head attention module, calculate the query vector, key vector and value vector, perform multi-head self-attention operation on the enhanced feature representation, fuse the output results of the multi-head attention, and obtain the modal weight W; wherein the number of attention heads is adaptively set according to the feature dimension, and each attention head independently captures the correlation between features.

[0075] S333. Obtain the modal weight W, perform dynamic gating operation on the enhanced feature representation and modal weight, use the gating signal to modulate the importance of the feature, integrate the modulated feature through the cross-modal fusion layer, and obtain the integrated memory representation R'; wherein the gating signal is calculated based on the temporal consistency and spatial correlation of the feature.

[0076] This embodiment establishes an intelligent multimodal memory fusion mechanism. First, the retrieved visual features, force feedback features and position features are preprocessed through a three-layer feature fusion network, and the residual connection structure is used to enhance the feature expression ability. At the same time, the complementary matrix between features is calculated to capture the information complementary relationship between different modal features; then a multi-head attention module is constructed to independently capture the feature correlation of different dimensions through multiple attention heads, and the output results of multi-head attention are fused to obtain accurate modal weight distribution; finally, the importance of features is modulated through dynamic gating operations, and the deep integration of features is achieved by combining the cross-modal fusion layer. This multi-level feature fusion and attention mechanism not only improves the fusion quality of multimodal information, but also ensures that the fusion results can make full use of the advantageous information of each modality.

[0077] like Figure 5 As shown, according to one aspect of the present application, step S4 is further:

[0078] S41, constructing a state-action pair based on the standardized current state and the pre-stored current action; calculating a weighted distance between the state-action pair and a memory sample in the integrated memory representation; selecting a neighbor sample based on the weighted distance to form a neighbor sample set;

[0079] S42. Calculate the quality evaluation value of the neighbor samples based on the neighbor sample set and the pre-stored original reward; substitute the weighted distance of the neighbor samples into the preset smoothing factor, perform an exponential transformation, and obtain an exponential transformation result; combine the exponential transformation result with the original reward and the neighbor sample quality evaluation value to obtain a shaping reward.

[0080] In one embodiment of the present application, based on the current state-action pair (s, a) and the integrated memory representation R', the weighted distance of the k nearest neighbor samples is calculated: d(s, a) = Σ(λi·||si - s|| 2 + μi·||ai - a|| 2 ), si is the historical state sample in memory, s is the current state, ai is the historical action sample in memory, a is the current action, λi is the adaptive weight coefficient of the state distance, μi is the adaptive weight coefficient of the action distance; neighbor sample set N. Based on the neighbor sample set N and the original reward r, calculate the shaping reward: r'(s, a) = r(s, a) + φ(N) · exp(-d(s, a) / σ), r(s, a) is the original reward value, φ(N) is the quality evaluation function value of the neighbor sample set, d(s, a) is the weighted distance value, and σ is the smoothing factor that controls the reward decay speed; output the shaping reward r'.

[0081] In another embodiment of the present application, the neighbor sample evaluation method: weighted distance d(s, a) = Σ(λi·||si - s|| 2+ μi·||ai - a|| 2 )·exp(-β·cos(θsa)); sample quality φ(N) = (1 / |N|)·Σ(St·Ss·Sa); shaping reward r'(s,a) = r(s,a) + φ(N)·exp(-d(s,a) / σ); where λi and μi are adaptive weight coefficients obtained by gradient update; θsa is the state-action vector angle, β is the angle weight; St, Ss, Sa are the temporal consistency score, spatial correlation score and action coherence score, respectively; σ is the adaptive smoothing factor σ=σ0·exp(-k·var(d)), σ0 is the initial smoothing factor, k is a constant, and var( ) is the variance function.

[0082] This embodiment achieves accurate evaluation and reward optimization of the current state-action pair through neighbor sample search and shaping reward calculation. First, neighbor sample search is performed based on the state-action pair, and the state distance and action distance are weighted respectively using adaptive weight coefficients to ensure that the searched neighbor samples have high correlation; then the quality evaluation value of the neighbor samples is calculated, considering multiple dimensions such as the temporal consistency, spatial correlation and action coherence of the samples, and the value of the neighbor samples is comprehensively evaluated; finally, the weighted distance of the neighbor samples is exponentially transformed through a smoothing factor, and combined with the original reward and sample quality evaluation value to generate a shaping reward, which achieves optimization and enhancement of the original reward, so that the reward signal can better guide the learning process of the strategy, and improve the learning efficiency and strategy performance.

[0083] According to one aspect of the present application, step S42 is further:

[0084] S421. Based on the nearest neighbor sample set, a sample quality assessment network is constructed to calculate the temporal consistency score, the spatial correlation score and the motion coherence score; the temporal consistency score, the spatial correlation score and the motion coherence score are combined by adaptive weights to obtain a sample quality assessment value; wherein the temporal consistency score is calculated based on the smoothness of the sample sequence, the spatial correlation score is calculated based on the spatial distribution characteristics of the sample, and the motion coherence score is calculated based on the continuity of the motion trajectory;

[0085] S422, calculating the Mahalanobis distance between the sample in the neighbor sample set and the current state-action pair, and constructing a distance covariance matrix; based on the distance covariance matrix, using the eigendecomposition method to calculate the distance metric matrix, and obtaining the weighted distance; wherein the distance covariance matrix represents the distribution structure of the sample in the feature space;

[0086] S423. Based on the sample quality evaluation value, the weighted distance and the pre-stored original reward, an adaptive smoothing factor network is constructed to obtain a dynamic smoothing factor; the weighted distance is substituted into the dynamic smoothing factor for exponential transformation to obtain an exponential transformation result; the exponential transformation result is combined with the original reward and the sample quality evaluation value to obtain a shaping reward. The input of the adaptive smoothing factor network is the statistical characteristics of the sample distribution, and the output is the dynamic smoothing factor.

[0087] This embodiment implements a high-precision neighbor sample evaluation and reward shaping mechanism. First, a sample quality evaluation network is constructed. By calculating the temporal consistency score, spatial correlation score, and action coherence score, these scores are combined using adaptive weights to achieve a comprehensive evaluation of the quality of neighbor samples; then the Mahalanobis distance between the neighbor sample and the current state-action pair is calculated, the distance covariance matrix is ​​constructed, and the distance metric matrix is ​​updated through eigendecomposition to ensure the accuracy of the distance metric; finally, the smoothing factor is dynamically adjusted through the adaptive smoothing factor network, and the weighted distance, original reward, and sample quality evaluation value are optimized and combined to generate a more instructive shaping reward. This reward shaping mechanism based on multi-dimensional evaluation improves the quality of the reward signal and provides more effective guidance for strategy learning.

[0088] like Figure 6 As shown, according to one aspect of the present application, step S5 is further:

[0089] S51. Based on the shaped reward and integrated memory representation, a policy network including at least three layers of fully connected neural networks is constructed to calculate the policy loss, reward prediction loss and memory consistency loss; the policy loss, reward prediction loss and memory consistency loss are weightedly combined according to the preset weight coefficient to obtain the total loss function;

[0090] S52. Based on the total loss function and the preset initial strategy parameters, calculate the initial learning rate and the learning rate decay coefficient; based on the initial learning rate and the learning rate decay coefficient, perform gradient update on the initial strategy parameters to obtain updated strategy parameters.

[0091] In one embodiment of the present application, based on the shaping reward r' and the integrated memory representation R', a combined loss function is constructed: L = α1·Lπ + α2·Lr + α3·Lm, where Lπ is the mean square error between the policy network output and the expert action, Lr is the mean square error between the predicted reward and the actual reward, Lm is the consistency error between the policy output and the memory representation, and α1, α2, and α3 are weight coefficients of each loss component; the total loss function L is output. Based on the total loss function L, an adaptive learning rate gradient update is adopted: the learning rate is updated to η* = η0·exp(-ρ·t), t is the current iteration step, η0 is the initial learning rate, ρ is the learning rate decay coefficient, and the policy parameter is updated to θ' = θ – η*·▽θL, θ is the current policy parameter, η* is the current learning rate, and ▽θL is the gradient of the loss function with respect to the policy parameter; the updated policy parameter θ' is output.

[0092] In another embodiment of the present application, the process of obtaining the total loss function is specifically as follows: strategy loss Lπ = Σ(H(π(s), πe(s)) + λ·DKL(π||πe)); reward prediction loss Lr = ||r'(s, a) – r*(s, a)|| 2 + α·||▽r'(s, a) - ▽r*(s, a)|| 2 ; Memory consistency loss Lm = -Σ(p(m|s)·log(q(m|s))); Total loss L= ω1·Lπ + ω2·Lr + ω3·Lm; where H is the cross entropy loss, DKL is the KL divergence, π and πe are the learning strategy and expert strategy respectively; r* is the predicted reward, ▽ represents the gradient operator; p(m|s) is the memory retrieval probability distribution, q(m|s) is the memory access distribution generated by the strategy; The weight coefficient ωi is generated by the adaptive regulator and satisfies Σωi = 1.

[0093] This embodiment constructs a policy network optimization mechanism with a combination of multiple loss functions, and realizes efficient learning and updating of policy parameters through adaptive weight adjustment. Specifically, first, a total loss function containing three components, namely, policy loss, reward prediction loss, and memory consistency loss, is constructed, so that policy learning simultaneously considers the three aspects of behavior imitation, reward prediction, and memory utilization; then, the weight coefficients of different loss components are dynamically adjusted through an adaptive weight network, so that the optimization process can adaptively adjust the importance of each loss term according to the learning progress and task requirements; finally, the policy parameters are updated based on the current learning rate and the gradient of the loss function, and the exponential decay mechanism of the learning rate is used to ensure the stability and convergence of the learning process, thereby realizing efficient optimization of the policy network parameters.

[0094] According to one aspect of the present application, step S51 is further:

[0095] S511. Obtain shaping reward r' and integrated memory representation R', build a multi-branch strategy network structure, the backbone network contains three fully connected layers, the number of neurons in each layer is n1, n2, and n3 respectively, add batch normalization layers and residual connections between each layer, transform and map the input features, and obtain the strategy network output P; among them, the strategy network adopts Swish activation function, and the residual connection dynamically adjusts the information flow through the gating mechanism.

[0096] S512. Obtain the policy network output P, ​​the shaped reward r' and the integrated memory representation R', and calculate three types of loss values ​​respectively: the deviation between the policy network output and the expert action obtains the policy loss Lπ, the difference between the predicted reward and the actual reward obtains the reward prediction loss Lr, the inconsistency between the policy output and the memory representation obtains the memory consistency loss Lm, and the loss component vector V is obtained; among them, the policy loss adopts the Huber loss function, the reward prediction loss adopts the mean square error, and the memory consistency loss adopts the KL divergence.

[0097] S513. Obtain the loss component vector V, construct an adaptive weight regulator, input the historical change trend of the loss component and the current task progress, calculate the contribution of each loss component to the overall performance, use the soft attention mechanism to generate a dynamic weight coefficient, and obtain an adaptive weight coefficient W; wherein the weight coefficient contains three components α1, α2, and α3, and satisfies the constraint condition of α1 + α2+ α3 = 1.

[0098] S514. Obtain the loss component vector V and the adaptive weight coefficient W, normalize the loss component, multiply the normalized loss component by the corresponding weight coefficient, construct a gradient scaling layer to adjust the optimization direction of different loss items, and perform weighted combination of the three losses according to the weights to obtain the total loss function L.

[0099] This embodiment constructs an adaptive multi-loss optimization framework. First, through a multi-branch policy network structure, batch normalization layers and residual connections are added between each layer, and the Swish activation function is used to improve the network's expressiveness and training stability; then the policy loss, reward prediction loss, and memory consistency loss are calculated separately, and different loss function forms are used to capture different aspects of learning objectives; then the weight coefficients of the loss components are dynamically adjusted through an adaptive weight regulator, and the optimization direction is adaptively allocated according to the historical change trend of the loss components and the current task progress; finally, the optimization direction of different loss items is adjusted through the gradient scaling layer to achieve a weighted combination of loss functions. This embodiment not only improves the efficiency of policy learning, but also ensures the stability and convergence of the learning process, thereby obtaining better policy parameters.

[0100] In one embodiment of the present application, an adaptive memory-enhanced imitation learning method for robot control tasks includes:

[0101] Step 1: Collect and process expert data and unlabeled data.

[0102] The dataset used in this example comes from the MuJoCo task in the D4RL (Datasets for Deep Data-Driven Reinforcement Learning) database. The D4RL dataset is widely used in the fields of reinforcement learning and imitation learning, especially for high-dimensional and complex control tasks. MuJoCo (Multi-Joint dynamics with Contact) is a physics engine specially designed to simulate complex scenarios such as multi-joint motion and contact mechanics in robot control tasks, providing a realistic virtual robot control environment for research. MuJoCo tasks cover a series of robot environments with different dynamic characteristics, including Ant (quadruped robot tasks, the robot needs to learn to move to the target position), Walker2d (bipedal robot walking tasks, the robot needs to learn to walk on a plane and maintain balance), Hopper (single-legged jumping tasks, simulating a single-legged robot jumping forward on a plane), etc. The task also provides data of different difficulty levels, including "expert", "medium-expert", "medium" and "random" data. Since the data of MuJoCo tasks have different numerical ranges, in order to ensure the stability of training and the generalization of the model, the state s and action a are first normalized. The data values ​​are mapped to the range of [-1,1] to avoid numerical instability. The specific formula is: s'=(s-smin) / (smax-smin)×2-1; a'=(a-amin) / (amax-amin)×2-1; where smin and smax are the minimum and maximum values ​​of state s, respectively, and amin and amax are the minimum and maximum values ​​of action a, respectively.

[0103] We use the “expert” data in the D4RL database, which contains high-quality expert demonstration data, as the expert demonstration dataset D E , select "random" data as the unlabeled dataset D U, used for the generation of shaping rewards and the training of imitation learning. Specific settings for the three tasks: Hopper: foot link size U[0.05, 0.07]×U[0.1755, 0.2145], leg link size U[0.03, 0.05]×U[0.23, 0.27], thigh link size U[0.04, 0.06]×U[0.2, 0.25], torso link size U[0.04, 0.06]×U[0.2, 0.25], thigh joint U[-2.79253, -2.44346]×U[0, 0].

[0104] Walker2d: Left foot link size U[0.05, 0.07]×U[0.09, 0.11], right foot link size U[0.05, 0.07]×U[0.09, 0.11], left leg link size U[0.03, 0.05]×U[0.23, 0.27], right leg link size U[0.03, 0.05]×U[0.23, 0.27], left thigh link size U[0.04, 0.06]×U[0.2, 0.25], right thigh link size U[0.04, 0.06]×U[0.2, 0.25], left thigh joint U[-2.79253, -2.44346]×U[0, 0].

[0105] Ant: Left leg link size U[0.05, 0.11], right leg link size U[0.05, 0.11], rear right leg link size U[0.05, 0.11], auxiliary link 1 size U[0.05, 0.11], auxiliary link 2 size U[0.05, 0.11], auxiliary link 3 size U[0.05, 0.11], auxiliary link 4 size U[0.05, 0.11], left ankle link size U[0.05, 0.11], right ankle link size U[0.05, 0.11], rear left ankle link size U[0.05, 0.11], rear right ankle link size U[0.05, 0.11], hip joint 1 U[-40, -20] × U[20, 40], hip joint 2 U[-40, -20] × U[20, 40], hip joint 3 U[-40, -20]×U[20, 40], hip 4 U[-40, -20]×U[20, 40], ankle 1 U[-80, -60]×U[-40, -20], ankle 2 U[-80, -60]×U[-40, -20], ankle 3 U[-80, -60]×U[-40, -20], ankle 4 U[-80, -60]×U[-40, -20]. Here U[a, b] represents a uniform distribution in the range of a to b, simulating the link size and joint range of different robots.

[0106] Step 2: Shaping reward generation and strategy update.

[0107] Step 21. Model initialization: input expert demonstration data D E , unlabeled data D U Initialize the policy parameters θ, learning rate α, noise standard deviation σ, weight coefficients α1, α2, α3, hyperparameters λ1, λ2, β1, β2, and cumulative reward AUCi=0.

[0108] Step 22: Memory neural network construction: based on offline data set D = {(si, ai)} i=1 N and memory capacity m to introduce memory learning. Use neural gas clustering to cluster the state space S to obtain a set of nodes N and a set of edges ε. For each node n∈N, find the nearest observation in the offline dataset D and label it as N′. At the same time, return the corresponding action D(N′) performed by the expert in the offline dataset D. Define a neural gas network g, where the nodes are N′ and the edges are ε. Use the neural gas network g to construct a memory codebook B such that B|S=g and B|A= D(N′). Pair the nodes with the corresponding actions and define the nearest neighbor function f B NN (x)=B(s′), the output of the memory neural network is: f θ，B MC (x) = f B NN (x)(e -λ1d(x，s′) )+L(1-e -λ1d(x，s′) ) σ(f θ (x)); where σ β1 (x) is the activation function, specifically: σ β1 (x)=2[LeakyReLU β1 ((x+1) / 2)-(1-β1)ReLU((x-1) / 2)]-1; β1=max(0,1-step / 100) is used to control the activation strength.

[0109] Step 23: Neighbor Search Shaping Reward Model Training: For the unlabeled dataset D U For each sample (s, a, s′), find the closest expert sample (se*, ae*, se*′). The nearest neighbor operation is implemented by the following formula: (se*, ae*, se*′) = argmin (se，ae，se′)∈DED((se,ae,se′),(s,a,s′)); where D((se,ae,se′),(s,a,s′)) is a distance metric function defined in the state and action space, which is used to measure the similarity between the current unlabeled sample and the expert sample. The Euclidean distance is used as the metric function. The specific distance calculation formula is: D((se,ae,se′),(s,a,s′))=sqrt((se-s) 2 +(ae-a) 2 +(se′-s′) 2 ). Then the similarity distance d is calculated: d = D((se*, ae*, se*′), (s, a, s′)), and the generated shaping reward is: r′(s, a) = r(s, a) + λ2 · exp(-β2×d / |A*|), where |A*| is the dimension of the action space.

[0110] Step 24: Strategy parameter optimization and cumulative reward shaping update: For each optimization step, add noise ξ~N(0, σ 2 ), generate the perturbed policy parameters θi=θ+σξi. Use the shaping reward to update the strategy: πij-RL Stepon reward r′; cumulative reward update AUCi-AUCi+E ζ~πij [∑ h=0 H-1 r′(s h , a h ,s h+1 )]. Loss function of memory neural network: L1=E (si,ai) ~DE [L(f θ，B MC (si), ai)]; Loss function for shaping reward design: L2=-∑ (s，a，s′) ∈DU r′(s, a); Loss function for cumulative reward optimization: L3=-1 / N∑ i=1 N AUCi; total loss function: L = α1·L1+α2·L2+α3·L3; update strategy parameters through the gradient of the total loss function: θ-θ-α▽ θ L. Among them, - means update to.

[0111] In this embodiment, the learning rate α is 0.01; the noise standard deviation σ is 0.01; the weighting coefficient α1 is 0.4; the weighting coefficient α2 is 0.3; the weighting coefficient α3 is 0.3; the memory capacity m is 100; the hyperparameter λ1 is 0.1; the hyperparameter λ2 is 0.1; the hyperparameter β1 is 0.5; and the hyperparameter β2 is 0.5.

[0112] The present invention realizes efficient learning and precise execution of robot control tasks by constructing an adaptive memory-enhanced imitation learning framework. First, the basis of data representation is established through standardized processing of multimodal data and hierarchical feature extraction; then a hierarchical memory structure with short-term memory, working memory and long-term memory is constructed, and efficient memory management is achieved through memory importance evaluation and dynamic update; then intelligent memory retrieval and multimodal information fusion are performed based on the current state to provide accurate memory support; at the same time, the reward signal is optimized through neighbor sample search and shaping reward calculation; finally, the efficient learning of strategy parameters is realized by adopting a combination of multiple loss functions and adaptive weight adjustment. The present invention improves the precise execution capability of robot control tasks through multi-level data processing, adaptive memory management, intelligent memory retrieval, optimized reward calculation and efficient strategy learning.

[0113] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. An adaptive memory-enhanced imitation learning method for robot control tasks, characterized in that The steps include: S1. Obtain original multimodal data, perform standardization on the original multimodal data, and obtain standardized data; based on the standardized data, perform sampling and feature extraction using a preset time window to obtain a time series alignment feature; Based on the time-series alignment features, an adaptive weighted hierarchical method is used to construct feature hierarchies and obtain hierarchical feature representations; The original multimodal data includes visual data, force feedback data and position data; S2. Based on the hierarchical feature representation, a hierarchical structure including short-term memory, working memory and long-term memory is constructed to obtain an initialized memory structure; based on the hierarchical feature representation, a memory importance index is calculated to obtain a memory importance score; based on the memory importance score, the initialized memory structure is dynamically updated to obtain an updated memory structure; S3, obtaining the original current state, normalizing the original current state to obtain a standardized current state; calculating a correlation score between the standardized current state and the updated memory structure; performing memory retrieval based on the correlation score to obtain a multimodal retrieval memory; Adaptively fuse multimodal retrieval memories to obtain integrated memory representation; S4, based on the standardized current state, the integrated memory representation and the pre-stored current action, perform a neighbor sample search on the current state-action pair to obtain a neighbor sample set; based on the neighbor sample set and the pre-stored original reward, calculate the shaping reward value to obtain the shaping reward; S5. Based on the shaping reward and integrated memory representation, a predetermined number of loss function components are constructed and combined to obtain a total loss function; based on the total loss function and the preset initial strategy parameters, the strategy network parameters are updated to obtain updated strategy parameters.

2. The adaptive memory-enhanced imitation learning method for robot control tasks according to claim 1, characterized in that: Step S1 is further as follows: S11, obtaining original multimodal data, calculating the mean and standard deviation of each type of data in the original multimodal data; subtracting the corresponding mean from each type of data in the original multimodal data and dividing the result by the corresponding standard deviation to obtain standardized data; S12, based on the preset time window, sliding sampling is performed on the standardized data to obtain sampled data; wavelet coefficients are calculated for the sampled data in each window to extract time-frequency features at different scales; Align the time-frequency features extracted from each window in the time dimension to obtain the time series alignment features; S13. Based on the time series alignment features, calculate the task complexity and determine the hierarchical depth; construct a feature clustering distance threshold for each layer of the hierarchical depth; Based on the feature clustering distance threshold, the feature hierarchy is constructed from bottom to top for the temporal alignment features; At each level of the feature hierarchy, the cluster centers are updated using adaptive weights to obtain a hierarchical feature representation.

3. The adaptive memory-enhanced imitation learning method for robot control tasks according to claim 2, characterized in that: Step S2 is further as follows: S21, allocating the hierarchical feature representation to short-term memory, working memory and long-term memory according to storage capacity and time attributes to obtain an initialized memory structure; S22, taking each feature layer in the hierarchical feature representation as a memory item to be evaluated; based on the initialized memory structure, calculating the usage frequency, task relevance and novelty score of each memory item to be evaluated; performing weighted summation of the usage frequency, task relevance and novelty score to obtain a memory importance score; S23. Based on the initialized memory structure, features whose memory importance scores are greater than a preset high threshold are upgraded, features whose memory importance scores are less than a preset low threshold are downgraded or forgotten, and features whose memory importance scores are between a preset low threshold and a preset high threshold are kept at the current level to obtain an updated memory structure.

4. The adaptive memory-enhanced imitation learning method for robot control tasks according to claim 3, characterized in that: Step S3 is further as follows: S31, obtaining the original current state collected in real time, standardizing the original current state to obtain a standardized current state; calculating the state distance between the standardized current state and each memory item in the updated memory structure; substituting the state distance into a preset distance scaling temperature parameter for exponential transformation to obtain a correlation score; S32, based on the preset attention temperature parameter, normalize the relevance score to generate the attention weight of memory retrieval; Based on the attention weights, relevant memory items are retrieved from the updated memory structure to obtain multimodal retrieval memory. S33, based on multimodal retrieval memory and standardized current state, a fully connected neural network is used to calculate the fusion weights of visual features, force feedback features and position features respectively; Based on the fusion weights, the memory features in the multimodal retrieval memory are weighted and summed to obtain the integrated memory representation.

5. The adaptive memory-enhanced imitation learning method for robot control tasks according to claim 4, characterized in that: Step S4 is further as follows: S41, constructing a state-action pair based on the standardized current state and the pre-stored current action; calculating a weighted distance between the state-action pair and a memory sample in the integrated memory representation; selecting a neighbor sample based on the weighted distance to form a neighbor sample set; S42. Calculate the quality evaluation value of the neighbor samples based on the neighbor sample set and the pre-stored original reward; substitute the weighted distance of the neighbor samples into the preset smoothing factor, perform an exponential transformation, and obtain an exponential transformation result; combine the exponential transformation result with the original reward and the neighbor sample quality evaluation value to obtain a shaping reward.

6. The adaptive memory-enhanced imitation learning method for robot control tasks according to claim 5, characterized in that: Step S5 is further as follows: S51. Based on the shaped reward and integrated memory representation, a policy network including at least three layers of fully connected neural networks is constructed to calculate the policy loss, reward prediction loss and memory consistency loss; the policy loss, reward prediction loss and memory consistency loss are weightedly combined according to the preset weight coefficient to obtain the total loss function; S52. Based on the total loss function and the preset initial strategy parameters, calculate the initial learning rate and the learning rate decay coefficient; based on the initial learning rate and the learning rate decay coefficient, perform gradient update on the initial strategy parameters to obtain updated strategy parameters.

7. The adaptive memory-enhanced imitation learning method for robot control tasks according to claim 6, characterized in that: Step S11 is further as follows: S111, acquiring original multimodal data, and performing time-series sampling on visual data, force feedback data, and position data in the original multimodal data respectively using an adaptive sampling frequency to obtain sampled data; Perform median filtering on the sampled data to remove outliers and obtain filtered multimodal data; S112, constructing a sliding time window based on the filtered multimodal data; Calculate the local mean, local standard deviation, skewness and kurtosis of the data in each window in the sliding time window to obtain local statistical characteristics; Based on the filtered multimodal data, the global mean, global standard deviation, global skewness and global kurtosis are calculated to obtain global statistical features; the local statistical features are fused with the global statistical features using the exponential weighted average method to obtain fused statistical features; S113, calculating an adaptive scaling factor based on the fused statistical features; performing a weighted combination of the local standard deviation and the global standard deviation to obtain a weighted standard deviation; Based on the adaptive scaling factor, the filtered multimodal data are subtracted from the local mean and divided by the weighted standard deviation, and then nonlinearly mapped through the hyperbolic tangent function to obtain the standardized data.

8. The adaptive memory-enhanced imitation learning method for robot control tasks according to claim 6, characterized in that: Step S22 is further as follows: S221, based on the hierarchical feature representation, calculate the time decay coefficient; based on the initialized memory structure, calculate the access frequency vector; exponentially combine the time decay coefficient and the access frequency vector to obtain a usage frequency index; S222. Based on the hierarchical feature representation and the initialized memory structure, construct a task context vector; calculate the mutual information matrix between the hierarchical feature representation and the task context vector; use the attention mechanism to perform weighted aggregation on the mutual information matrix to obtain a task relevance index; S223, calculating a first probability density distribution of hierarchical feature representation; calculating a second probability density distribution of features stored in the initialized memory structure; using Wasserstein distance to measure the difference between the first probability density distribution and the second probability density distribution to obtain a novelty score; S224. Based on the usage frequency index, task relevance index and novelty score, an adaptive weight network is constructed to obtain a dynamic weight coefficient; the usage frequency index, task relevance index and novelty score are weightedly combined with the corresponding dynamic weight coefficient to obtain a memory importance score.

9. The adaptive memory-enhanced imitation learning method for robot control tasks according to claim 6, characterized in that: Step S32 is further as follows: S321. Based on the correlation score and the updated memory structure, an adaptive temperature regulation network is constructed; based on the adaptive temperature regulation network, a temperature parameter is optimized by using a gradient descent method to obtain an attention temperature parameter; Based on the attention temperature parameter, the relevance score is temperature scaled to obtain the scaled relevance score; S322, dividing the scaled correlation score by the attention temperature parameter and inputting the result into the Softmax function to calculate the attention weight distribution; Apply an adaptive gating mechanism to filter the attention weight distribution to obtain the filtered attention weight; S323, based on the filtered attention weights, weighted sampling is performed on the memory items in the updated memory structure to obtain sampled memory items; Perform consistency verification on the sampled memory items to obtain verified memory items; Based on the verified memory items, the importance is sorted to obtain multimodal retrieval memory.

10. The adaptive memory-enhanced imitation learning method for robot control tasks according to claim 6, characterized in that: Step S42 is further as follows: S421, based on the nearest neighbor sample set, construct a sample quality assessment network, calculate the temporal consistency score, spatial correlation score and motion coherence score; use adaptive weights to combine the temporal consistency score, spatial correlation score and motion coherence score to obtain a sample quality assessment value; S422, calculating the Mahalanobis distance between the sample in the neighbor sample set and the current state-action pair, and constructing a distance covariance matrix; Based on the distance covariance matrix, the distance metric matrix is ​​calculated using the eigendecomposition method to obtain the weighted distance; S423. Based on the sample quality evaluation value, the weighted distance and the pre-stored original reward, an adaptive smoothing factor network is constructed to obtain a dynamic smoothing factor; the weighted distance is substituted into the dynamic smoothing factor for exponential transformation to obtain an exponential transformation result; the exponential transformation result is combined with the original reward and the sample quality evaluation value to obtain a shaping reward.

Images