A scenario memory based robot training system and method

Abstract

The application discloses a scenario memory-based robot training system and method in the technical field of artificial intelligence and robot learning, and solves problems of lack of effective memory scheduling system, non-intelligent experience value quantization mechanism and insufficient multi-modal information fusion capability in existing robot training methods.The system comprises a perception module, a multi-modal unified memory encoder, a progressive memory scheduling system, a multi-dimensional value quantization evaluation mechanism, an intelligent experience hierarchical scheduler, a multi-modal unified encoding retriever, a strategy generation module, an execution module and an anomaly detection module.The multi-modal unified memory encoder adopts a hierarchical dimension reduction Transformer architecture, and fuses an RGB image, a depth image, force sensor data and joint angle information into a 576-dimensional unified feature vector.The progressive memory scheduling system comprises a working memory, a short-term memory and a long-term memory three-layer structure.The multi-dimensional value quantization evaluation mechanism quantitatively scores experience based on reward evaluation, novelty evaluation and uncertainty evaluation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and robot learning, in particular to a robot training system and method based on episodic memory. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, robot learning has become an important research direction of robot technology. Existing robot training methods are mainly based on deep reinforcement learning, imitation learning and other technologies, which have improved the learning ability of robots to a certain extent.

[0003] In the prior art, CN114161419B discloses a robot operation skill learning method guided by episodic memory, which establishes an episodic memory model of the robot learning system, constructs a similarity measurement algorithm to realize the retrieval and matching of event and scene information, and adopts a two-stage episodic memory guided deep reinforcement learning algorithm. This technology improves the learning efficiency of robot operation skills to a certain extent, but its memory model adopts a single-layer structure, lacks effective management of experiences of different time scales, and mainly processes visual information, with limited multi-modal information fusion capability.

[0004] US2022410380A1 discloses a large-scale robot skill learning method, which pre-trains the network using offline positive sample robot demonstration data, and further trains the network based on online robot episodes, and uses various adaptation techniques to alleviate the forgetting problem in the training process. Although this technology has improved in terms of forgetting prevention, it lacks a systematic memory architecture design, and the experience selection mechanism is relatively simple, which cannot realize intelligent allocation of memory resources.

[0005] CN113657573A discloses a robot skill acquisition method based on meta-learning, which establishes an episodic memory model, constructs a similarity measurement algorithm, and adopts a meta-learning algorithm guided by episodic memory. This technology realizes a certain degree of knowledge transfer, but its memory architecture is relatively simple, lacks a hierarchical memory management mechanism, and the method for evaluating the importance of experience is not perfect.

[0006] The existing technologies have the following shortcomings: first, the memory architecture design is not fine enough, and most of them use single-layer memory structure, which cannot effectively manage experiences of different time scales and importance; second, the experience selection mechanism is relatively simple, lacks multi-dimensional comprehensive evaluation of the importance of experience, resulting in low utilization efficiency of memory resources; third, the multi-modal information fusion capability is insufficient, mainly processing visual information, lacking unified encoding of multi-source information such as tactile and proprioceptive information; finally, the forgetting mechanism design is not reasonable, lacking differentiated forgetting strategies based on the importance of experience, and lacking systematic exception handling and safety guarantee mechanism. SUMMARY

[0007] Technical problems

[0008] The technical problem to be solved by the present application is that the existing robot training method lacks an effective progressive memory scheduling system, the experience value quantification evaluation mechanism is not intelligent enough, the cross-sensory unified coding capability is insufficient, the progressive memory decay mechanism is not reasonable enough, and the system exception handling and safety guarantee mechanism is lacking, resulting in low training efficiency, low sample utilization rate, limited generalization ability, and insufficient system stability and safety.

[0009] Technical solutions

[0010] To solve the above technical problems, the present application provides a robot training system based on episodic memory, which constructs a complete multi-modal perception and memory learning architecture. The perception layer of the system adopts an RGB-D camera and a 6-axis force sensor to form a perception module, realizing comprehensive collection of the state information of the environment around the robot. On this basis, a multi-modal unified memory encoder uses a hierarchical dimension reduction Transformer architecture to uniformly encode and process multi-source perception data, wherein the RGB image is reduced to 256-dimensional visual features after extracting 2048-dimensional features through a ResNet-50 network, the depth image generates 128-dimensional depth features through a 3D convolution network, the 6-axis force sensor data is processed through a fully connected layer to obtain 128-dimensional tactile features, the joint angle information is extracted through a sine-cosine encoding method to obtain 64-dimensional proprioceptive features, and finally the above features are fused into a 576-dimensional unified feature vector through a cross-modal attention mechanism.

[0011] The core memory architecture of the system adopts a progressive memory scheduling system, which is composed of three hierarchical memory units. The working memory unit uses a multi-head self-attention mechanism to store and manage the state-action pairs of the current task, with a storage capacity of 64 state-action pairs, and realizes the dynamic ordering of experience importance through time step attention weights. The short-term memory unit stores important experience fragments based on the attention mechanism, with a storage capacity of 5000 experience fragments. The long-term memory unit uses variational autoencoder technology to compress and store historical experiences, with a storage capacity of 50000 compressed experiences, a compression ratio controlled within the range of 20:1 to 40:1, and a reconstruction error threshold dynamically adjusted according to the compression ratio, ensuring that the reconstruction error MSE is less than 0.005 when the compression ratio is 20:1, and the reconstruction error MSE is less than 0.018 when the compression ratio is 40:1.

[0012] The system designs a multi-dimensional value quantitative evaluation mechanism to realize the intelligent evaluation and distribution of experience. The mechanism includes a reward evaluation unit based on time difference error, a novelty evaluation unit based on Euclidean distance calculation, and an uncertainty evaluation unit based on the entropy value of the action probability distribution output by the strategy network. The three evaluation units work together to comprehensively quantify and score the experience. The comprehensive evaluation unit adopts a weighted fusion algorithm with configurable weights. The reward evaluation weight is set to 0.3-0.6, the novelty evaluation weight is set to 0.2-0.4, and the uncertainty evaluation weight is set to 0.1-0.3, satisfying the constraint condition . The comprehensive importance score is calculated by the formula .

[0013] The intelligent experience hierarchical scheduler realizes the automatic hierarchical storage of experience based on the comprehensive score of the multi-dimensional value quantitative evaluation mechanism. High-value experience with a score greater than 0.8 is directly stored in the long-term memory unit and an index pointer is established in the working memory unit. Medium-value experience with a score of 0.5-0.8 is stored in the short-term memory unit. Low-value experience with a score less than 0.5 is stored in the temporary buffer area of the working memory unit. The scheduler also integrates an adaptive forgetting processing unit, which uses a dynamic forgetting rate adjustment mechanism based on task performance and memory utilization. The base forgetting rate of the working memory unit is 0.05-0.12, the base forgetting rate of the short-term memory unit is 0.008-0.025, and the base forgetting rate of the long-term memory unit is 0.001-0.008.

[0014] The decision generation part of the system is composed of a multi-modal unified encoding retriever, a strategy generation module, and an execution module. The multi-modal unified encoding retriever uses a cross-modal attention mechanism to retrieve relevant historical experience from the progressive memory scheduling system, providing experience support for decision generation. The strategy generation module uses an Actor-Critic architecture, taking the retrieved historical experience as input features of the value function, and generates a robot action strategy through a policy gradient algorithm combined with the retrieved experience. The execution module is responsible for executing the action strategy generated by the strategy generation module, and is equipped with an abnormality detection module. When a sensor failure or memory network anomaly is detected, the system automatically switches to a safe mode and triggers an emergency stop mechanism.

[0015] The application also provides a robot training method based on episodic memory, which realizes a complete training process through seven steps. First, the state information of the robot environment is obtained through an RGB-D camera and a 6-axis force sensor, and a layered dimension reduction Transformer encoder is used to encode the multi-modal information into a 576-dimensional feature vector. Then the feature vector is input into the progressive memory scheduling system for similarity matching retrieval, and the cosine similarity calculation method is used, the basic retrieval threshold is set to 0.7, and the layered retrieval strategy is used, the similarity threshold is 0.9 when accurate matching is performed in the working memory, the similarity threshold is 0.7 when fuzzy matching is performed in the short-term memory, and the similarity threshold is 0.6 when semantic matching is performed in the long-term memory.

[0016] The core evaluation link of the method adopts a multi-dimensional value quantitative evaluation mechanism to quantitatively score the current experience, specifically including calculating the reward importance score based on the time difference error TD_error

[0017]

[0018] Wherein TD_error is the time difference error, and max_TD is the maximum possible value of the time difference error; the novelty score based on the Euclidean distance between the experience feature vector and the nearest neighbor experience in the memory bank is calculated

[0019]

[0020] Wherein min_distance is the Euclidean distance between the current experience and the nearest neighbor experience in the memory bank, and max_distance is the maximum Euclidean distance in the experience feature space; and the uncertainty score based on the entropy value of the action probability distribution output by the strategy network is calculated

[0021]

[0022] Wherein p(a) is the probability of action a output by the strategy network, and max_entropy is the maximum entropy value of the action probability distribution. The comprehensive importance score is obtained by weighted fusion of the configurable weights. After storing the experience into the corresponding memory level according to the comprehensive importance score, the action strategy is generated by using the policy gradient algorithm combined with the retrieved historical experience and the current state.

[0023] ​​​The execution and optimization phase of the method includes three key links of action execution, memory update and exception monitoring. Among them, the memory update adopts adaptive frequency control, and the forgetting update is performed every N training steps, and N is dynamically adjusted in the range of 10-50 according to the task complexity. After the system executes the action and obtains the environment feedback, the memory network is updated every N training steps, and N is dynamically adjusted in the range of 10-50 according to the task complexity. The adaptive forgetting update adopts the strategy of combining time decay and importance protection, and the forgetting probability calculation formula is

[0024] ;

[0025] Wherein base_rate is the basic forgetting rate; importance_score is the experience importance score and the range is 0-1; time_decay is an exponential decay function , wherein t is the time from the generation of the experience, τ is the time decay constant; task_factor is selected in the range of 0.5-2.0 according to the task complexity. Finally, the abnormal detection module continuously monitors the system state, and when the sensor failure or memory network anomaly is detected, the system automatically switches to the safe mode to ensure the stable operation of the system.

[0026] Beneficial effects

[0027] The present application has the following beneficial effects compared with the prior art:

[0028] 1. The training efficiency is significantly improved: through the progressive memory scheduling system and the intelligent experience selection mechanism, compared with the traditional reinforcement learning method, the training sample demand is reduced by 50-70%, from hundreds of thousands to tens of thousands, the training time is shortened by 40-60%, and the sample efficiency is improved from 1% of the traditional method to 3-8%.

[0029] 2. The generalization ability is obviously enhanced: through the multi-modal unified encoding and hierarchical memory management, the new task adaptation time is shortened from several hours to 1-2 hours, the generalization accuracy is improved by 15-30%, and the performance fluctuation is reduced by 25-40% when the environment changes.

[0030] 3. The memory management is more intelligent: through the multi-dimensional value quantitative judgment and the progressive memory decay mechanism, the optimal allocation of memory resources is realized, the important experience protection rate is improved by 35%, and the memory utilization efficiency is improved by 40%.

[0031] 4. The multi-modal information processing ability is enhanced: through the unified multi-modal encoding architecture, the effective fusion of multi-source information such as vision, touch and proprioception is realized, and the complex task processing capacity is improved by 20-35%.

[0032] 5. System security and stability improvement: through the abnormality detection module and security mechanism, the system fault detection accuracy reaches more than 95%, the emergency response time is less than 100ms, and the long-term running stability is significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is the overall architecture schematic diagram of the robot training system based on episodic memory of the present application;

[0034] Figure 2 is the progressive memory scheduling architecture schematic diagram of the present application;

[0035] Figure 3 is the robot training method flow schematic diagram of the present application. DETAILED DESCRIPTION

[0036] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0037] The present application also provides a robot training method based on episodic memory, which realizes a complete training process through seven steps. Figure 3 As shown in the figure, the entire training process includes key links such as perception acquisition, similarity matching retrieval, value quantification judgment, experience storage, strategy generation, action execution and abnormality monitoring, and forms a complete closed-loop control system between each step.

[0038] Example 1

[0039] The purpose of this embodiment is to verify the training effect and performance improvement of the progressive memory scheduling system in the mechanical arm grasping task.

[0040] As shown in the figure, this embodiment constructs a complete robot training system based on episodic memory. Figure 1 As shown in the figure, this embodiment constructs a complete robot training system based on episodic memory. The system hardware configuration includes: 6-DOF robot arm (UR5e), RGB-D camera (Intel RealSense D435i), 6-axis force sensor (ATI Nano17), high-performance computing platform (NVIDIA RTX 3080).

[0041] As shown in the figure, this embodiment constructs a complete robot training system based on episodic memory. Figure 2As shown, the specific configuration of the progressive memory scheduling system is as follows: the working memory unit capacity is set to 64 state-action pairs, a multi-head self-attention mechanism is used, the number of attention heads is 4, the hidden layer dimension is 256, and the importance ranking of experience is managed through time step attention weight; the short-term memory unit capacity is set to 5000 experience fragments, a multi-head attention mechanism is used, and the number of attention heads is 8; the long-term memory unit capacity is set to 50000 compressed experiences, a variational autoencoder is used, the encoder dimension is 512, the decoder dimension is 576, the compression ratio is set to 30:1, and the reconstruction error is controlled to be less than 0.008. Figure 2 The hierarchical structure relationship between the working memory unit, the short-term memory unit and the long-term memory unit, and the complete process of how the intelligent experience layering scheduler allocates experiences to the corresponding memory levels according to the comprehensive score are clearly shown.

[0042] The specific implementation of the multi-modal unified memory encoder: the RGB image is extracted through a pre-trained ResNet-50 to obtain a 2048-dimensional feature, and then reduced to 256-dimensional through a fully connected layer; the depth image is extracted through a 3-layer 3D convolutional network (convolution kernel size 3x3x3, step 1, padding 1) to obtain a 128-dimensional depth feature; the 6-axis force sensor data (Fx, Fy, Fz, Mx, My, Mz) is extracted through a 2-layer fully connected network (hidden layer dimension 256) to obtain a 128-dimensional tactile feature; the joint angle information is extracted through sine and cosine coding (sin(θ), cos(θ)) to obtain a 64-dimensional proprioceptive feature. The four modal features are fused through a cross-modal attention mechanism, and the attention weights are 0.4, 0.25, 0.25 and 0.1 respectively, and finally a 576-dimensional unified feature vector is formed.

[0043] The specific calculation method of the multi-dimensional value quantification evaluation mechanism is as follows: where TD_error is the time difference error, and max_TD is the historical maximum time difference error; the novelty evaluation score is where min_distance is the Euclidean distance between the current experience and the nearest neighbor experience in the memory bank, and max_distance is the historical maximum distance; the uncertainty evaluation score is where p(a) is the action probability distribution output by the policy network, and max_entropy is the historical maximum entropy value. The comprehensive importance score calculation formula is: Score = 0.5xR_score + 0.3xN_score + 0.2xU_score.

[0044] The allocation strategy of the intelligent experience hierarchical scheduler: experiences with a comprehensive score greater than 0.8 are directly stored in the long-term memory unit and an index pointer is established in the working memory unit; experiences with a score of 0.5-0.8 are stored in the short-term memory unit; experiences with a score less than 0.5 are stored in the temporary buffer area of the working memory unit.

[0045] The specific parameters of the adaptive forgetting mechanism: the basic forgetting rate of the working memory unit is set to 0.1, the basic forgetting rate of the short-term memory unit is set to 0.01, and the basic forgetting rate of the long-term memory unit is set to 0.001. The forgetting probability calculation formula is: where τ is the time decay constant (set to 1000), and task_factor is set to 1.0 according to the task complexity.

[0046] The configuration of the anomaly detection module: sensor fault detection is achieved through data integrity and reasonableness checks, with a detection period of 10ms; memory network anomaly detection is achieved through access delay and data consistency monitoring, with a detection period of 100ms; in safety mode, the system automatically stops learning and updating, maintains the current strategy, and limits the action amplitude.

[0047] The training process uses a standard robotic arm grasping task, including 8 different shapes and weights of objects (cubes, cylinders, spheres, etc., with a weight range of 50g-500g). The training environment is set to standard laboratory conditions (temperature 22±2℃, humidity 50±10%, illumination intensity 500lux).

[0048] The strategy generation module uses an Actor-Critic architecture, with the Actor network containing 3 fully connected layers (dimensions 576, 512, 256 respectively), outputting a 6-dimensional action vector; the Critic network contains 3 fully connected layers (dimensions 576, 512, 1 respectively), outputting a state value. The learning rate is set to 0.001, the batch size is 64, and the discount factor is 0.99.

[0049] After 78000 training samples, the system's average success rate in 8 types of grasping tasks reached 85.3%, with a cube grasping success rate of 89.2%, a cylinder grasping success rate of 83.7%, and a sphere grasping success rate of 81.5%. The average adaptation time for new tasks was 92 minutes, with the fastest being 67 minutes. In a 168-hour continuous operation stability test, the performance remained above 83.1%, with a performance decay of less than 3% and a standard deviation of 1.8%. The anomaly detection module successfully detected and handled 12 sensor data anomalies and 3 memory network access anomalies during the test period, with a detection accuracy of 96.7%.

[0050] The effectiveness of the progressive memory scheduling system is verified by this embodiment. Compared with the traditional reinforcement learning method, the training sample demand is reduced by 61%, and the training time is shortened by 45%. The hierarchical memory architecture has a significant advantage in improving training efficiency.

[0051] Embodiment 2

[0052] The purpose of this embodiment is to verify the performance of the multi-modal unified encoding architecture in complex assembly tasks and the effect of multi-modal information fusion.

[0053] This embodiment adopts a dual-arm collaborative robot system (ABB YuMi) equipped with a binocular stereo vision system (Basler acA1300-60gm), a multi-point tactile sensor array (16 tactile units), and a high-precision position encoder (resolution 0.001°).

[0054] Optimized configuration of the multi-modal unified encoding architecture: EfficientNet-B3 network is used for visual feature extraction, and the output is reduced to 256 dimensions after dimensionality reduction of 1536-dimensional features; deep features are extracted through an improved 3D ResNet network, and the output is reduced to 128 dimensions after dimensionality reduction of 160-dimensional features; tactile features are processed through a temporal convolution network (TCN) to extract 128-dimensional temporal tactile features from 16-channel tactile signals; proprioceptive features include 14 joint angles and angular velocity information, which are extracted through position encoding and velocity encoding to obtain 32-dimensional features, and then combined into 64-dimensional proprioceptive features.

[0055] The cross-modal attention mechanism adopts an improved Transformer structure, which contains 4 attention heads, each with a dimension of 144. The attention weights are obtained through adaptive learning, and the weight distribution after training convergence is: vision 0.35, depth 0.28, tactile 0.27, and proprioception 0.10.

[0056] The memory network capacity is adjusted according to the task complexity: the working memory unit is expanded to 80 state-action pairs, the short-term memory unit is expanded to 8000 experience fragments, and the long-term memory unit is expanded to 80000 compressed experiences. The compression ratio is optimized to 25:1, and the reconstruction error is controlled to be less than 0.006.

[0057] The weight configuration of the multi-dimensional value quantification evaluation is adjusted according to the characteristics of the assembly task: the reward evaluation weight w1=0.45, the novelty evaluation weight w2=0.35, and the uncertainty evaluation weight w3=0.20. The hierarchical scheduling threshold is adjusted accordingly: high importance experience (Score>0.75) is stored in long-term memory, medium importance experience (Score 0.4-0.75) is stored in short-term memory, and low importance experience (Score<0.4) is stored in working memory.

[0058] The test task is precision electronic component assembly, including 12 sub-tasks such as circuit board insertion, screw tightening, and cable connection. The total number of training samples is 95,000, and the training time is 36 hours.

[0059] The experimental results show that the average success rate of the system in 12 assembly tasks reaches 82.7%, among which the success rate of circuit board insertion is 86.4%, the success rate of screw tightening is 81.3%, and the success rate of cable connection is 79.8%. The multi-modal information fusion accuracy reaches 88.2%, which is improved by 15.6% compared with single visual mode. The average adaptation time of new tasks is 76 minutes, which is shortened by 17% compared with embodiment 1. After 168 hours of continuous operation, the system performance remains above 80.5%, and the performance decay is less than 2.7%.

[0060] The effectiveness of the multi-modal unified encoding architecture in complex tasks is verified through this embodiment. Multi-modal information fusion significantly improves the environmental perception ability and task execution precision of the system, and proves the technical advantages of the unified encoding architecture.

[0061] Embodiment 3

[0062] The purpose of this embodiment is to verify the memory management effect and system stability of the adaptive forgetting mechanism in long-term continuous learning.

[0063] This embodiment uses a mobile operation robot platform (TurtleBot3 + UR3 mechanical arm), equipped with a laser radar (SICK TiM571), an RGB-D camera (Kinect v2), and an IMU sensor (MPU-9250). The test scene is a dynamic warehouse environment, including comprehensive tasks such as cargo handling, path planning, and obstacle avoidance navigation.

[0064] The detailed configuration of the adaptive forgetting mechanism: the basic forgetting rate of the working memory unit is set to 0.08, and the dynamic adjustment range is 0.05-0.12; the basic forgetting rate of the short-term memory unit is set to 0.015, and the adjustment range is 0.008-0.025; the basic forgetting rate of the long-term memory unit is set to 0.003, and the adjustment range is 0.001-0.008.

[0065] The task complexity factor task_factor is calculated according to the environmental dynamics and task difficulty:

[0066] ,

[0067] Where dynamic_level is the environmental change frequency (0-1), difficulty_level is the task difficulty rating (0-1), and success_rate is the recent success rate (0-1).

[0068] Memory utilization monitoring mechanism: The access frequency and hit rate of each memory level are counted every 1000 training steps. When the working memory access frequency is >80%, the capacity is appropriately increased to 96; when the short-term memory hit rate is <60%, the retrieval threshold is adjusted from 0.7 to 0.6; when the long-term memory utilization rate is <30%, the compression rate is increased to 35:1.

[0069] The continuous learning test lasted for 7 days (168 hours), with 8 hours of learning and 16 hours of sleep each day. The environment configuration was randomly changed every day during the test, including the type of goods, placement position, obstacle distribution, etc.

[0070] Key data recorded during the experiment: On the first day, the average success rate was 78.5%, with a total of 32,000 experiences in memory; on the third day, the average success rate was 81.2%, with a total of 45,000 experiences in memory; on the fifth day, the average success rate was 83.7%, with a total of 52,000 experiences in memory; on the seventh day, the average success rate was 84.3%, with a total of 48,000 experiences in memory (the forgetting mechanism began to have a significant effect).

[0071] Forgetting effect analysis: The retention rate of important experiences (Score>0.8) reached 94.5%, the retention rate of medium important experiences (Score 0.5-0.8) was 67.3%, and the retention rate of low important experiences (Score<0.5) was 23.1%. The overall memory utilization efficiency of the system improved from 45% initially to 78% finally.

[0072] Long-term stability test results: After 168 hours of continuous operation, the system performance remained at 82.8%, with only a 1.5% decrease compared to the peak performance. The average time spent on memory retrieval remained within 15ms, meeting the real-time requirements. The anomaly detection module detected 18 system anomalies during the long-term operation, all of which were successfully handled without any safety incidents.

[0073] This embodiment verifies the effectiveness of the adaptive forgetting mechanism, and the system can maintain stable performance and reasonable memory size during long-term continuous learning, proving the technical advantages of the gradual forgetting strategy in memory management.

[0074] Embodiment 4

[0075] The purpose of this embodiment is to verify the influence of the compression rate boundary value on the system performance, and to test the technical rationality of the compression rate in the range of 20:1 to 40:1.

[0076] This embodiment uses the same hardware configuration as Embodiment 1, but tests the compression rate of the long-term memory unit in a gradient manner. The compression rate is set to 20:1, 25:1, 30:1, 35:1, and 40:1 respectively for comparative experiments, and each configuration is repeated 3 times to ensure the reliability of the results.

[0077] Compression rate 20:1 configuration: the encoder dimension of the variational autoencoder is set to 1152, the decoder dimension is 576, and the reconstruction error is controlled to be less than 0.005. Test results show that the average success rate of the system reaches 87.1±1.2%, the memory retrieval time is 18±2ms on average, and the storage space occupation is large.

[0078] Compression rate 25:1 configuration: the encoder dimension is set to 864, the decoder dimension is 576, and the reconstruction error is controlled to be less than 0.006. Test results show that the average success rate of the system reaches 86.4±1.0%, the memory retrieval time is 16±1ms on average, and the storage efficiency is good.

[0079] Compression rate 30:1 configuration: the encoder dimension is set to 576, the decoder dimension is 576, and the reconstruction error is controlled to be less than 0.008. Test results show that the average success rate of the system reaches 85.3±0.9%, the memory retrieval time is 15±1ms on average, and the performance and storage are balanced optimally.

[0080] Compression rate 35:1 configuration: the encoder dimension is set to 432, the decoder dimension is 576, and the reconstruction error is controlled to be less than 0.012. Test results show that the average success rate of the system reaches 83.7±1.1%, the memory retrieval time is 14±1ms on average, and the storage efficiency is high but the performance decreases.

[0081] Compression rate 40:1 configuration: the encoder dimension is set to 288, the decoder dimension is 576, and the reconstruction error is controlled to be less than 0.018. Test results show that the average success rate of the system reaches 81.2±1.3%, the memory retrieval time is 13±2ms on average, and the storage efficiency is the highest but the performance decreases significantly.

[0082] Through gradient test verification, the compression rate in the range of 20:1 to 40:1 can maintain the normal operation of the system, and the performance is optimal in the range of 20:1 to 30:1, the storage efficiency is higher in the range of 30:1 to 40:1, which proves the technical rationality of the compression rate range set in the claims.

[0083] Example 5

[0084] The purpose of this embodiment is to verify the technical basis and optimization effect of the weight configuration range in the multi-dimensional value quantitative evaluation mechanism.

[0085] This embodiment uses the same basic configuration as Example 1, but performs gradient optimization test on the reward evaluation weight w1, novelty evaluation weight w2, and uncertainty evaluation weight w3, and each configuration is repeated for 5 times.

[0086] Weight configuration combination 1: w1=0.3, w2=0.4, w3=0.3. Test results show that the system performs outstanding in novelty exploration, but the response to the reward signal is relatively weak, with an average success rate of 81.8±1.5% and a new task adaptation time of 118±12 minutes.

[0087] Weight configuration combination 2: w1=0.4, w2=0.3, w3=0.3. Test results show that the system performs outstanding in novelty exploration, but the response to the reward signal is relatively weak, with an average success rate of 82.1±1.3% and a new task adaptation time of 105±10 minutes.

[0088] Weight configuration combination 3: w1=0.5, w2=0.3, w3=0.2. Test results show that the system responds well to the reward signal and has high learning efficiency, with an average success rate of 85.3±1.1% and a new task adaptation time of 92±8 minutes.

[0089] Weight configuration combination 4: w1=0.6, w2=0.25, w3=0.15. Test results show that the system pays excessive attention to the reward signal and has decreased exploration ability, with an average success rate of 84.7±1.2% and a new task adaptation time of 98±9 minutes.

[0090] Weight configuration combination 5: w1=0.45, w2=0.35, w3=0.2. Test results show that the system performs excellently in complex tasks, especially suitable for scenarios requiring delicate operations, with an average success rate of 86.1±0.9% and a new task adaptation time of 88±7 minutes.

[0091] Through weight optimization experiment verification, when w1 is in the range of 0.3-0.6, w2 is in the range of 0.25-0.4, and w3 is in the range of 0.15-0.3, the system performance is optimal, proving the technical rationality and optimization basis of the weight range set in the claims.

[0092] Comparative Example 1

[0093] The purpose of this comparative example is to verify the difference in system performance when lacking a progressive memory scheduling system, and to replace the hierarchical memory network with a traditional single-layer experience replay buffer.

[0094] The comparative scheme adopts a fixed-size experience replay buffer with a capacity of 50,000 experiences, and uses the FIFO (First-In-First-Out) strategy for experience updating. The experience selection uses uniform random sampling with a batch size of 64. Other system configurations remain the same as in Example 1, including the perception module, multi-modal encoding architecture, policy generation module, etc. The total number of training samples is set to 78,000 to ensure fair comparison.

[0095] The test task is also a robotic grasping task, including the same 8 objects and training environment. 5 repeated experiments are conducted to ensure the reliability of the results.

[0096] The experimental results show that the average success rate of the system using single-layer experience replay is 71.2±2.1% in 8 grasping tasks, among which the success rate of grasping cube is 75.8±1.8%, the success rate of grasping cylinder is 69.4±2.3%, and the success rate of grasping sphere is 67.3±2.5%. The training convergence requires complete 78000 samples, and the training time is 65±3 hours. The average adaptation time of new tasks is 156±18 minutes.

[0097] Compared with Example 1: the success rate decreased by 14.1 percentage points (from 85.3% to 71.2%), the training time increased by 44% (from 45 hours to 65 hours), and the adaptation time of new tasks increased by 70% (from 92 minutes to 156 minutes). The memory utilization efficiency is only 31%, and the important experience protection rate decreases to 45%.

[0098] Through comparison, it can be seen that the lack of progressive memory scheduling system causes the system to be unable to effectively manage experiences of different importance, important experiences are easily covered, and learning efficiency is significantly reduced, proving the necessity and superiority of hierarchical memory architecture.

[0099] Comparative Example 2

[0100] The purpose of this comparative example is to verify the impact of the system performance when lacking a multi-dimensional value quantification evaluation mechanism, and a simple random experience selection strategy is used instead of intelligent importance assessment.

[0101] The comparison scheme maintains the progressive memory scheduling architecture, but the experience allocation strategy is changed to random allocation: 30% of the experience is randomly allocated to long-term memory, 50% is allocated to short-term memory, and 20% is allocated to working memory. The forgetting mechanism uses a fixed forgetting rate and does not consider the importance of experience. 5 repeated experiments are conducted.

[0102] The test conditions are the same as in Example 1, and the number of training samples is 78000. The experimental results show that the average success rate of the system is 76.8±1.7%, the training time is 52±2 hours, and the average adaptation time of new tasks is 128±15 minutes.

[0103] Compared with Example 1: the success rate decreased by 8.5 percentage points, the training time increased by 16%, and the adaptation time of new tasks increased by 39%. The memory utilization efficiency is only 52%, and the important experience protection rate decreases to 61%.

[0104] Through comparison, it can be seen that the lack of multi-dimensional value quantitative evaluation mechanism leads to unreasonable allocation of memory resources, important experiences cannot be effectively protected, learning efficiency and adaptability are significantly reduced, and the importance of intelligent importance evaluation is proved.

[0105] Comparative Example 3

[0106] The purpose of the present comparative example is to verify the performance limitations of the system in complex tasks when lacking a multi-modal unified encoding architecture, and only using visual information for training and decision-making.

[0107] The comparative scheme removes the tactile sensor and proprioceptive information, and only retains the RGB-D visual information. The memory encoder is simplified to a single-modal encoder, and the feature dimension is adjusted to 384 dimensions (256 dimensions of RGB features + 128 dimensions of depth features). Three repeated experiments are performed.

[0108] The test task adopts the precision assembly task in Example 2, and the number of training samples is 95000. The experimental results show that the average success rate of the system in the assembly task is 68.4±2.3%, and the success rate of tasks requiring fine force control (such as screw tightening) is only 54.2±3.1%.

[0109] Comparative analysis with Example 2: The overall success rate decreased by 14.3 percentage points (from 82.7% to 68.4%), and the success rate of fine operation tasks decreased by 27.1 percentage points (from 81.3% to 54.2%).

[0110] Through comparison, it can be seen that the lack of multi-modal unified encoding architecture leads to a significant decline in system performance in complex tasks requiring fine operation, and cannot effectively utilize tactile and proprioceptive information, proving the key role of multi-modal unified encoder in complex task processing.

[0111] Based on the verification results of the above examples and comparative examples, the present application realizes effective management of experiences of different time scales through a progressive memory scheduling system, realizes intelligent allocation of memory resources through a multi-dimensional value quantitative evaluation mechanism, realizes effective fusion of multi-source information through a multi-modal unified encoding architecture, realizes long-term stable memory management through an adaptive forgetting mechanism, and realizes system security protection through an anomaly detection module. Compared with the prior art, the present application has significantly improved in training efficiency, generalization ability, adaptation speed, system stability and safety, etc., and provides an effective technical solution for robot intelligent learning.

Claims

1. A scenario memory-based robot training system, characterized by, The method comprises the following steps: a perception module comprising an RGB-D camera and a 6-axis force sensor for obtaining the environmental state information of the robot; a multi-modal unified memory encoder adopting a hierarchical dimension reduction Transformer architecture, which extracts 2048-dimensional features from RGB images through ResNet-50 and reduces them to 256-dimensional visual features, extracts 128-dimensional depth features from depth images through a 3D convolutional network, extracts 128-dimensional tactile features from 6-axis force sensor data through a fully connected layer, extracts 64-dimensional proprioceptive features from joint angle information through sine-cosine encoding, and fuses them into a 576-dimensional unified feature vector through a cross-modal attention mechanism; a progressive memory scheduling system comprising a working memory unit, a short-term memory unit, and a long-term memory unit, wherein the working memory unit stores the state-action pairs of the current task through a multi-head self-attention mechanism, the short-term memory unit stores important experience fragments through an attention mechanism, and the long-term memory unit stores historical experiences through a variational autoencoder; a multi-dimensional value quantitative evaluation mechanism comprising a reward evaluation unit based on time difference error, a novelty evaluation unit based on Euclidean distance calculation, and an uncertainty evaluation unit based on the entropy value of the action probability distribution output by the policy network, for quantitatively scoring experiences; an intelligent experience hierarchical scheduler that allocates experiences to corresponding memory levels according to the comprehensive score of the multi-dimensional value quantitative evaluation mechanism; a multi-modal unified encoding retriever configured to receive the 576-dimensional unified feature vector and perform similarity matching in the progressive memory scheduling system based on the 576-dimensional unified feature vector to retrieve relevant historical experiences; a policy generation module adopting an Actor-Critic architecture configured to input the historical experiences retrieved by the multi-modal unified encoding retriever as the input features of the value function and generate a robot action policy through a policy gradient algorithm combined with the historical experiences; an execution module for executing the action policy generated by the policy generation module; an anomaly detection module that automatically switches to a safe mode and triggers an emergency stop mechanism when a sensor fails or a memory network is abnormal.

2. The episodic memory based robot training system of claim 1, wherein, The compression ratio of the long-term memory unit is controlled within the range of 20:1 to 40:1, and the reconstruction error threshold is dynamically adjusted according to the compression ratio. At a compression ratio of 20:1, the reconstruction error... Reconstruction error at a compression ratio of 40:1 .

3. The episodic memory based robot training system of claim 1, wherein, The storage capacity of the working memory unit is 64 state-action pairs, the importance of experiences is sorted through time step attention weights, the storage capacity of the short-term memory unit is 5000 experience fragments, and the storage capacity of the long-term memory unit is 50000 compressed experiences.

4. The episodic memory based robot training system of claim 1, wherein, In the intelligent experience hierarchical scheduler, experiences with a score greater than 0.8 are directly stored in the long-term memory unit and an index pointer is established in the working memory unit, experiences with a score of 0.5-0.8 are stored in the short-term memory unit, and experiences with a score less than 0.5 are stored in the temporary buffer area of the working memory unit.

5. The episodic memory based robot training system of claim 1, wherein, The comprehensive evaluation unit of the multi-dimensional value quantitative evaluation mechanism adopts a weighted fusion algorithm with configurable weights, and the evaluation weight is 0.3-0.6, the novelty evaluation weight is 0.2-0.4, the uncertainty evaluation weight is 0.1-0.3, and satisfies The comprehensive importance score is calculated by the formula .

6. The episodic memory based robot training system of claim 1, wherein, The intelligent experience hierarchical scheduler further comprises an adaptive forgetting processing unit, adopts a dynamic forgetting rate adjustment mechanism based on task performance and memory utilization rate, the basic forgetting rate of the working memory unit is 0.05-0.12, the basic forgetting rate of the short-term memory unit is 0.008-0.025, and the basic forgetting rate of the long-term memory unit is 0.001-0.

008.

7. A scenario memory-based robot training method, characterized by, The method comprises the following steps: Step one: obtain the environmental state information of the robot through an RGB-D camera and a 6-axis force sensor, and encode the multi-modal information into a 576-dimensional feature vector by using a hierarchical dimension reduction Transformer encoder; Step two: input the feature vector into a progressive memory scheduling system for similarity matching retrieval, and set the basic retrieval threshold to 0.7 by using cosine similarity calculation; Step three: quantitatively score the current experience by using a multi-dimensional value quantitative evaluation mechanism, calculate the reward evaluation score, novelty score and uncertainty score, and obtain the comprehensive importance score by weighted fusion of the configurable weights; Step four: store the experience into the corresponding memory level according to the comprehensive importance score; Step five: combine the retrieved historical experience and the current state, and generate an action policy by using a policy gradient algorithm; Step six: execute the action and obtain the environmental feedback, and update the memory network once every N training steps, wherein N is dynamically adjusted in the range of 10-50 according to the task complexity; and step seven: monitor the system state by using an abnormality detection module, and automatically switch to a safety mode when a sensor failure or a memory network abnormality is detected. 8.The episodic memory based robot training method of claim 7, wherein, The similarity matching retrieval in step two adopts a hierarchical retrieval strategy, first performs accurate matching in the working memory with a similarity threshold of 0.9, then performs fuzzy matching in the short-term memory with a similarity threshold of 0.7, and finally performs semantic matching in the long-term memory with a similarity threshold of 0.

6. 9.The episodic memory based robot training method of claim 7, wherein, The multi-dimensional value quantitative evaluation in step three comprises: calculating the reward importance score based on the time difference error TD_error ， Wherein TD_error is the time difference error, and max_TD is the maximum possible value of the time difference error; calculating the novelty score based on the Euclidean distance between the experience feature vector and the nearest neighbor experience in the memory bank ， Wherein min_distance is the Euclidean distance between the current experience and the nearest neighbor experience in the memory bank, and max_distance is the maximum Euclidean distance in the experience feature space; and calculating the uncertainty score based on the entropy value of the action probability distribution output by the policy network ， Wherein p(a) is the probability of the action a output by the policy network, and max_entropy is the maximum entropy value of the action probability distribution. 10.The situational memory based robot training method of claim 7, wherein, The adaptive forgetting update in step six adopts a strategy combining time decay and importance protection, and the forgetting probability calculation formula is ； where base_rate is the base forgetting rate; importance_score is the experience importance score and ranges from 0 to 1 ; time_decay is an exponential decay function where t is the time from when the experience was generated, τ is a time decay constant; task_factor takes a value in the range 0.5-2.0 depending on the task complexity.

Images