A robot experience learning system based on a multi-modal memory network

Abstract

The application discloses a kind of robot experience learning systems based on multimodal memory network, comprising: multimodal preprocessing module, collect multi-source original data and execute preprocessing, generate multimodal state sequence;Phase boundary mutation detection module, construct joint mutation map and extract multimodal mutation label, generate phase boundary state fragment;Memory update module, input improved sLSTM-T network, output multimodal memory state vector;Locking experience solidification module, construct self-homeomorphism mapping and execute error verification, generate locking experience structure;Generation sub-compression module, construct experience generation relationship diagram, generate minimum experience generation sub-set;Action reconstruction control module, execute structure reconstruction, generate action strategy and control robot execution.The application realizes the reliable storage, redundant compression and dynamic reconstruction generation of robot historical experience by constructing the multimodal locking experience learning mechanism based on improved sLSTM-T network.

Description

Technical Field

[0001] This invention relates to the field of robot intelligent control technology, and in particular to a robot experience learning system based on a multimodal memory network. Background Technology

[0002] As the application scope of service robots, inspection robots, warehousing robots, and collaborative robots continues to expand, robots need to perform target recognition, path approach, grasping operations, obstacle avoidance adjustments, and task feedback processing in complex environments based on visual images, depth point clouds, force feedback, joint states, chassis pose, and motion control data. Existing robot experience learning methods typically improve task execution efficiency by recording historical task data, extracting multimodal features, training temporal models, or building experience bases, enabling robots to recall similar experiences in subsequent tasks.

[0003] In existing technologies, multimodal experiences are typically stored according to fixed time windows, complete task trajectories, or unified feature vectors. Visual, depth, force, and pose data are mostly processed through splicing and fusion or unified encoding. When visual positioning deviations, missing depth point clouds, delayed force feedback, or abnormal motion feedback occur during robot task execution, existing methods struggle to determine whether a historical experience is truly valid, and also find it difficult to identify which states, actions, and feedback within that experience have stable and deducible relationships. This can easily lead to experiences with modal conflicts or incomplete feedback being written into long-term memory.

[0004] Therefore, existing robot experience learning methods still suffer from problems such as insufficient judgment of experience reliability, continuous accumulation of redundant experience, reliance on similarity matching for the reuse of historical experience, and interference of erroneous experience with decision-making in new tasks. When a new task scenario is only partially similar to a historical scenario, the robot is prone to directly applying historical actions, leading to grasping failures, path blocking, abnormal contact forces, or inaccurate action adjustments, making it difficult to achieve interpretable storage, compressed retention, and reliable transfer of multimodal experience.

[0005] Therefore, how to provide a robot experience learning system based on multimodal memory networks is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a robot experience learning system based on a multimodal memory network. This invention fully utilizes the improved scalar long short-term memory network sLSTM-T, multimodal locked experience structure construction, experience generation relationship graph, and experience structure reconstruction technology to perform joint memory modeling, locked verification, redundant experience compression, and action strategy generation for the robot's vision, depth, force perception, pose, and action states during task execution. This enables interpretable storage and reliable transfer of robot experience, and has the advantages of high multimodal experience reliability, low interference from erroneous experience, high accuracy of experience reuse, and strong adaptability to complex scenarios.

[0007] A robot experience learning system based on a multimodal memory network according to an embodiment of the present invention includes: The multimodal preprocessing module collects multimodal raw data during the robot's task execution process, preprocesses the multimodal raw data, and generates a multimodal state sequence. The phase boundary mutation detection module constructs a sparse joint mutation map for the multimodal state sequence, extracts the time stamps of visual mutation events, depth mutation events and force-sensory mutation events respectively, calculates the joint mutation degree of the three modes and uses a dual-threshold critical window to detect the joint mutation degree transition point, and generates the corresponding phase boundary state segment. The memory update module introduces the phase boundary state fragment input into the improved sLSTM-T network with modal trust gate, adaptive forgetting coefficient and cross-fragment residual channel, and performs scalar gain, attenuation and masking updates on visual state, depth state, force state, pose state and action state respectively, and outputs memory state vector. The locking experience solidification module constructs multimodal automorphic mapping pairs for the memory state vector, calculates the positive locking error and the reverse redemption error using sparse cohomology kernels, introduces a modal pruning gate to remove redundant dimensions, and solidifies the corresponding phase boundary state fragment into a locking experience structure when both types of errors fall into the dynamic threshold region and the cohomology kernel rank remains unchanged after pruning. The sub-compression module reads the state succession relationship, action succession relationship and feedback generation relationship between each locking experience structure, constructs the experience generation relationship graph, performs redundancy stripping on each locking experience structure, and generates the minimum experience generation subset. The motion reconfiguration control module collects multimodal state data corresponding to the current task when the robot performs a new task, matches the corresponding locking experience structure from the minimum experience generation subset and performs structure reconfiguration to generate a motion strategy, and controls the robot to perform the current task according to the motion strategy.

[0008] Optionally, the multimodal raw data includes visual image data, depth point cloud data, force feedback data, joint state data, chassis pose data, gripper state data, motion control data, and task result data.

[0009] Optionally, the preprocessing of the multimodal raw data to generate a multimodal state sequence includes: The visual image data, depth point cloud data, force feedback data, joint state data, chassis pose data, gripper state data, and motion control data are time-aligned according to a unified timestamp. Perform target region cropping, brightness normalization, and edge enhancement processing on visual image data to generate a visual state sequence; Perform outlier removal, spatial registration, and distance mapping on the deep point cloud data to generate a depth state sequence; Noise filtering, contact peak detection, and torque fluctuation segmentation are performed on the force feedback data to generate a force state sequence. Trajectory smoothing and state synchronization processing are performed on joint state data, chassis pose data, and gripper state data to generate a pose and motion state sequence. The visual state sequence, depth state sequence, force state sequence, and pose and motion state sequence are spliced ​​and encoded in chronological order to generate a multimodal state sequence.

[0010] Optionally, generating the corresponding phase boundary state fragment includes: Read the visual state, depth state, and force state in the multimodal state sequence according to the sampling time order, compare the visual change amplitude, depth change amplitude, and force change amplitude between the current sampling time and the previous sampling time, and generate the visual change record, depth change record, and force change record for the corresponding sampling time. Visual change records, depth change records, and force change records are compared with their respective preset mutation judgment conditions. When the amplitude of visual change reaches the visual mutation judgment condition, a visual mutation mark is generated; when the amplitude of depth change reaches the depth mutation judgment condition, a depth mutation mark is generated; and when the amplitude of force change reaches the force mutation judgment condition, a force mutation mark is generated. Using each sampling time as a graph node, time continuity edges are established between adjacent graph nodes according to the sampling time sequence. When two mutation markers of visual mutation marker, depth mutation marker, or force mutation marker appear at the same sampling time, cross-modal mutation co-occurrence relationship is established on the corresponding graph node to generate a sparse joint mutation graph. Based on the visual change amplitude, depth change amplitude, force change amplitude, and number of cross-modal mutation co-occurrence relationships corresponding to each graph node, the three-modal joint mutation degree of the corresponding graph node is generated, and the three-modal joint mutation degree is written into the sparse joint mutation graph in the order of sampling time. A dual-threshold critical window is constructed using high triggering conditions and low continuity conditions to continuously detect the trimodal joint abruptness of each graph node. Graph nodes that meet the high triggering conditions and have two abruptness markers are identified as joint abruptness transition points. Graph nodes that meet the low continuity conditions and are temporally continuous with the joint abruptness transition points are incorporated into the corresponding critical window. Based on the target recognition state, approach distance state, contact force state, action execution state, and task feedback state at the joint abruptness transition point, the target confirmation phase boundary, approach initiation phase boundary, contact establishment phase boundary, action execution phase boundary, and feedback result phase boundary are marked respectively. The multimodal state data between adjacent task phase boundaries are identified as phase boundary state segments.

[0011] Optionally, the output memory state vector includes: Based on the field order of visual state, depth state, force state, pose state, and action state, five modal input branches are constructed for the improved scalar long short-term memory network sLSTM-T. Corresponding state reading units, state change recording units, and scalar memory units are set in each modal input branch. A modal trust gate is set between each modal input branch and the scalar memory unit. The modal trust gate reads the corresponding modal state, the hidden state of the previous phase boundary, and the succession state of other modes in the same phase boundary from the current phase boundary state segment, and generates visual trust markers, depth trust markers, force trust markers, pose trust markers and motion trust markers. An adaptive forgetting unit is set after the modal trust gate. The adaptive forgetting unit determines the retention state, decay state and shielding state of the corresponding modal memory based on the trust markers of each modality, the state change amplitude between adjacent phase boundary state segments and the mutation co-occurrence relationship in the sparse joint mutation graph. A cross-segment residual channel is set between adjacent phase boundary state segments. The cross-segment residual channel reads the difference between the previous phase boundary memory state and the current phase boundary input state that is not interpreted by the corresponding modality, and writes the difference into the visual residual, depth residual, force residual, pose residual and motion residual according to the modality field. Based on the output of the modal trust gate, the output of the adaptive forgetting unit, and the output of the cross-fragment residual channel, scalar gain update, scalar decay update, and scalar masking update are performed on the visual state, depth state, force state, pose state, and action state, respectively, to generate the scalar memory state corresponding to each modality. The scalar memory states corresponding to each modality are combined according to the field order of visual state, depth state, force state, pose state and action state to generate the current phase boundary memory state. The current hidden state is updated based on the current phase boundary memory state, and the current phase boundary memory state and the current hidden state are jointly determined as the memory state vector of the corresponding task phase boundary.

[0012] Optionally, the introduction of modal pruning gates to remove redundant dimensions, when both types of errors fall within the dynamic threshold region and the cohomology kernel rank remains unchanged after pruning, solidifies the corresponding phase boundary state fragment into a locked empirical structure, including: Read the memory state vector corresponding to the kth task boundary, and split the memory state vector into state memory block, action memory block and feedback memory block according to the state field, action field and feedback field. Construct a multimodal locking coordinate set according to the field order of visual state, depth state, force state, pose state and action state. Based on the multimodal locking coordinate set, a forward automorphic mapping and a reverse automorphic mapping are constructed. The forward automorphic mapping maps the state memory block and action memory block to the feedback memory block, and the reverse automorphic mapping maps the feedback memory block and action memory block to the key state sub-block in the state memory block. The forward automorphic mapping and the reverse automorphic mapping are combined into a multimodal automorphic mapping pair. Based on the temporal continuity, modal correlation, and action feedback correlation among coordinate points in the multimodal locking coordinate set, a sparse coherence kernel is established. The differences between the output of the forward automorphic mapping and the feedback memory block, and the differences between the output of the reverse automorphic mapping and the key state sub-block are read respectively to generate the forward locking error and the reverse redemption error. Visual state, depth state, force state, pose state and action state are respectively input into the modal pruning gate. Based on the influence of each modal dimension on the forward locking error, reverse redemption error and sparse homology kernel rank, redundant dimensions are deleted, and the state dimension, action dimension and feedback dimension that make the sparse homology kernel rank unchanged are retained to generate the pruned locking coordinate set. Based on the pruned locking coordinate set, the forward locking error, reverse redemption error, and sparse homology kernel rank are recalculated. When the forward locking error falls into the forward dynamic threshold region, the reverse redemption error falls into the reverse dynamic threshold region, and the sparse homology kernel rank before and after pruning is consistent, the state memory block, action memory block, feedback memory block, multimodal automorphic mapping pair, and pruned locking coordinate set of the corresponding phase boundary state segment are combined into a locking empirical structure.

[0013] Optionally, generating the minimum empirical subset includes: Read the state memory block, action memory block, feedback memory block, multimodal automorphic mapping pairs and pruned locking coordinate set from each locking experience structure, and generate an experience structure index for each locking experience structure according to the task number, task boundary type, action primitive type and task result type. State continuity edges are established based on the continuity relationship of state memory blocks between adjacent locking experience structures. Action continuity edges are established based on the continuity relationship of action memory blocks between adjacent locking experience structures. Feedback generation edges are established based on the fact that the feedback memory block of the previous locking experience structure can serve as the state input condition of the next locking experience structure. Locking experience structures are used as graph nodes, and state continuity edges, action continuity edges, and feedback generation edges are used as graph edges to generate an experience generation relationship graph. In the experience generation relationship graph, read the incoming edge source, outgoing edge destination and feedback generation path of each locking experience structure. In accordance with the order of consistent state coverage, consistent action acceptance and consistent feedback results, register the generation path formed by the combination of multiple locking experience structures and generate the combination generation record of the corresponding locking experience structure. Select each locking experience structure as the experience structure to be stripped, temporarily delete the experience structure to be stripped and the associated graph edges, and check whether the remaining locking experience structures can cover the state memory block, action memory block and feedback memory block corresponding to the experience structure to be stripped based on the combined generation record. When the remaining locking experience structures can completely cover the state memory block, action memory block and feedback memory block, mark the experience structure to be stripped as a redundant experience structure. Delete all redundant experience structures and retain the lock experience structures that cannot be generated by combining the remaining lock experience structures. Combine the retained lock experience structures and their corresponding state continuation edges, action continuation edges and feedback generation edges into a minimum experience generation subset.

[0014] Optionally, the generation of the action strategy, and the control of the robot to perform the current task based on the action strategy, includes: Collect the visual state, depth state, force state, pose state, gripper state, and task target state in the current task, generate the current task state fragment according to the order of the multimodal state sequence fields, and extract the current state constraints, current action requirements, and target feedback constraints based on the current task state fragment; The current state constraints, current action requirements, and target feedback constraints are converted into the current task query structure. The state memory block, action memory block, feedback memory block, and pruned locking coordinate set of each locking experience structure in the minimum experience generation subset are read. Candidate locking experience structures are obtained by filtering them in the order of consistency of task boundary fields, coverage of state fields, acceptability of target feedback fields, and satisfaction of modal credibility fields. Based on the state continuation edges, action continuation edges, and feedback generation edges between candidate locking experience structures, the candidate locking experience structures are spliced ​​and sorted. Candidate locking experience structures that have modal conflicts with the current task state segment, are unreachable actions, or have inconsistent feedback are deleted, and the current task reconstruction structure is generated. Read the action memory block, multimodal automorphic mapping pair and feedback generation edge in the current task reconstruction structure, map the current state constraints to the action field to be generated, and generate the action strategy. The action strategy includes approach direction, end pose, action speed, gripping force, gripper opening and closing amount, obstacle avoidance distance and abnormal retreat action. The motion strategy is sent to the robot motion controller, robotic arm controller, and gripper controller to control the robot to execute the current task, and the actual feedback state, motion execution state, and task result state after the current task is executed are written into a new multimodal state sequence.

[0015] The beneficial effects of this invention are: This invention performs unified preprocessing on multimodal raw data such as vision, depth, force, pose, and motion during robot task execution, and further constructs a sparse joint mutation graph to identify task phase boundaries. This allows robot experience to no longer be coarsely stored according to fixed time windows or complete task trajectories, but to form phase boundary state fragments around key task stages such as target confirmation, contact establishment, action execution, and feedback changes. This reduces the interference of invalid historical data on the experience learning process and improves the task relevance of multimodal experience fragments and the accuracy of subsequent memory modeling.

[0016] This invention introduces an improved sLSTM-T network with modal trust gates, adaptive forgetting coefficients, and cross-segment residual channels to perform scalar-level gain, attenuation, and masking updates on visual, depth, force, pose, and action states. It also combines multimodal automorphic mapping pairs, sparse coherence kernels, and modal pruning gates to perform bidirectional locking verification on the memory state vector. This prevents experiences with modal conflicts, delayed feedback, or redundant dimensions from directly entering stable memory, thereby improving the credibility, interpretability, and traceability of historical experiences and reducing the impact of erroneous experiences on the robot's subsequent task decisions.

[0017] This invention constructs an experience generation relationship graph by reading the state succession relationship, action succession relationship, and feedback generation relationship between locked experience structures, and performs redundancy stripping on the locked experience structures to form a minimal experience generation subset, enabling the robot to retain representative core experience structures. When executing a new task, the system matches the corresponding locked experience structure from the minimal experience generation subset and performs structure reconstruction to generate an action strategy suitable for the current task, avoiding direct application of historical trajectories and improving the robot's accuracy of experience reuse, stability of action adjustment, and autonomous adaptation capability in complex scenarios. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a robot experience learning system based on a multimodal memory network proposed in this invention; Figure 2 This is a flowchart illustrating the locked experience structure solidification process of a robot experience learning system based on a multimodal memory network proposed in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] refer to Figure 1 and Figure 2 A robot experience learning system based on a multimodal memory network includes: The multimodal preprocessing module collects multimodal raw data during the robot's task execution process, preprocesses the multimodal raw data, and generates a multimodal state sequence. The phase boundary mutation detection module constructs a sparse joint mutation map for the multimodal state sequence, extracts the time stamps of visual mutation events, depth mutation events and force-sensory mutation events respectively, calculates the joint mutation degree of the three modes and uses a dual-threshold critical window to detect the joint mutation degree transition point, and generates the corresponding phase boundary state segment. The memory update module introduces the phase boundary state fragment input into the improved sLSTM-T network with modal trust gate, adaptive forgetting coefficient and cross-fragment residual channel, and performs scalar gain, attenuation and masking updates on visual state, depth state, force state, pose state and action state respectively, and outputs memory state vector. The locking experience solidification module constructs multimodal automorphic mapping pairs for the memory state vector, calculates the positive locking error and the reverse redemption error using sparse cohomology kernels, introduces a modal pruning gate to remove redundant dimensions, and solidifies the corresponding phase boundary state fragment into a locking experience structure when both types of errors fall into the dynamic threshold region and the cohomology kernel rank remains unchanged after pruning. The sub-compression module reads the state succession relationship, action succession relationship and feedback generation relationship between each locking experience structure, constructs the experience generation relationship graph, performs redundancy stripping on each locking experience structure, and generates the minimum experience generation subset. The motion reconfiguration control module collects multimodal state data corresponding to the current task when the robot performs a new task, matches the corresponding locking experience structure from the minimum experience generation subset and performs structure reconfiguration to generate a motion strategy, and controls the robot to perform the current task according to the motion strategy.

[0021] In this embodiment, the multimodal raw data includes visual image data, depth point cloud data, force feedback data, joint state data, chassis pose data, gripper state data, motion control data, and task result data.

[0022] In this embodiment, the preprocessing of the multimodal raw data to generate a multimodal state sequence includes: The visual image data, depth point cloud data, force feedback data, joint state data, chassis pose data, gripper state data, and motion control data are time-aligned according to a unified timestamp. Perform target region cropping, brightness normalization, and edge enhancement processing on visual image data to generate a visual state sequence; Perform outlier removal, spatial registration, and distance mapping on the deep point cloud data to generate a depth state sequence; Noise filtering, contact peak detection, and torque fluctuation segmentation are performed on the force feedback data to generate a force state sequence. Trajectory smoothing and state synchronization processing are performed on joint state data, chassis pose data, and gripper state data to generate a pose and motion state sequence. The visual state sequence, depth state sequence, force state sequence, and pose and motion state sequence are spliced ​​and encoded in chronological order to generate a multimodal state sequence.

[0023] In this embodiment, generating the corresponding phase boundary state fragment includes: Read the visual state, depth state, and force state in the multimodal state sequence according to the sampling time order, compare the visual change amplitude, depth change amplitude, and force change amplitude between the current sampling time and the previous sampling time, and generate the visual change record, depth change record, and force change record for the corresponding sampling time. Visual change records, depth change records, and force change records are compared with corresponding preset mutation judgment conditions. A visual mutation marker is generated when the amplitude of visual change reaches the visual mutation judgment condition; a depth mutation marker is generated when the amplitude of depth change reaches the depth mutation judgment condition; and a force mutation marker is generated when the amplitude of force change reaches the force mutation judgment condition. The preset mutation judgment conditions are as follows: Visual mutation determination criteria: A visual abrupt change is determined when the pixel displacement between the target center in the current frame and the target center in the previous frame is greater than 15px, or when the intersection-union ratio of the target segmentation contours in the current frame and the previous frame is less than 0.65.

[0024] Criteria for determining deep mutation: When the difference between the average depth value of the target leading edge in the current frame and that in the previous frame exceeds 0.12m, or when the proportion of effective points in the depth point cloud in the target area in the current frame decreases by more than 25%, a sudden change in depth is determined.

[0025] Criteria for determining sudden changes in force perception: When the resultant force increment of the end six-dimensional force exceeds 2.0N in two consecutive sampling periods, or the shear force component increases by more than 40% relative to the previous period, it is determined that a sudden change in force perception has occurred. Using each sampling time as a graph node, temporal continuation edges are established between adjacent graph nodes according to the sampling time sequence. When two mutation markers (visual mutation marker, depth mutation marker, or force mutation marker) appear simultaneously at the same sampling time, a cross-modal mutation co-occurrence relationship is established on the corresponding graph node to generate a sparse joint mutation graph. The specific steps for generating the sparse joint mutation graph are as follows: The visual, depth, and force variation amplitudes at all sampling times are read sequentially. These three types of amplitudes are then arranged into a three-dimensional mutation vector in the order of visual, depth, and force. The three-dimensional mutation vectors are sequentially written into the columns of a matrix to generate a mutation vector matrix. The mutation vector matrix is ​​scanned column by column. If the visual mutation marker and depth mutation marker are both 1, or the visual mutation marker and force mutation marker are both 1, or the depth mutation marker and force mutation marker are both 1, then the corresponding sampling time node is recorded as a cross-modal co-occurrence node. In the graph, add cross-modal co-occurrence edges between nodes and the previous co-occurring node; establish unidirectional temporal continuation edges for adjacent nodes along the entire time axis according to the sampling sequence number; calculate the length of cross-modal co-occurrence edges, and delete the cross-edge when the time interval between any two cross-modal co-occurrence nodes is greater than the set upper limit, keeping only the cross-modal edges whose co-occurrence time interval does not exceed the upper limit; finally, delete isolated nodes that contain only a single modal mutation and have no temporal continuation edges or cross-modal edges, and retain the combination relationship between temporal continuation edges and cross-modal co-occurrence edges to obtain a sparse joint mutation graph containing only effective mutation connections; Based on the visual change amplitude, depth change amplitude, force change amplitude, and the number of cross-modal abrupt co-occurrence relationships corresponding to each graph node, the trimodal joint abruptness of the corresponding graph node is generated. The trimodal joint abruptness is then written into the sparse joint abruptness graph in the order of sampling time. Specifically, the generation of the trimodal joint abruptness of the corresponding graph node is as follows: At the current sampling time node, the visual change amplitude, depth change amplitude, and force change amplitude are read sequentially. The visual change amplitude is multiplied by a preset weight of 0.40, the depth change amplitude by a preset weight of 0.35, and the force change amplitude by a preset weight of 0.25. The three results are added together to obtain the first step composite value. The number of cross-modal mutation co-occurrence relationships of the node is counted. If any one of the visual-depth co-occurrence, visual-force co-occurrence, or depth-force co-occurrence occurs simultaneously, the co-occurrence count is recorded as 1. If two types occur, it is recorded as 2. If all three types occur, it is recorded as 3. The co-occurrence count is then multiplied by the co-occurrence weight of 0.10 to obtain the second step compensation value. Finally, the first step composite value and the second step compensation value are accumulated to obtain the three-modal joint mutation degree of the node. The values ​​are written into the node attribute table of the sparse joint mutation graph in the order of sampling time. A dual-threshold critical window is constructed using high-trigger and low-continuity conditions to continuously detect the trimodal joint abruptness of each graph node. Graph nodes that meet the high-trigger condition and simultaneously have two abruptness markers are identified as joint abruptness transition points. Graph nodes that meet the low-continuity condition and are temporally continuous with the joint abruptness transition points are incorporated into the corresponding critical window. Based on the target recognition state, approach distance state, contact force state, action execution state, and task feedback state at the joint abruptness transition point, the target confirmation phase boundary, approach initiation phase boundary, contact establishment phase boundary, action execution phase boundary, and feedback result phase boundary are marked respectively. The multimodal state data between adjacent task phase boundaries are identified as phase boundary state segments. The dual-threshold critical window constructed using high-trigger and low-continuity conditions is as follows: First, a sliding window with a length of 120 sampling points is established for all three-modal joint mutation degrees in the order of sampling time. The average value and standard deviation of the joint mutation degree are calculated in each window. The average value plus twice the standard deviation is used to obtain the high trigger threshold of this window, and the average value plus one standard deviation is used to obtain the low continuation threshold of this window. Scanning point by point along the time axis from front to back, if the joint mutation degree of the current node is not lower than the high trigger threshold of the window and the node has at least one pair of co-occurrence markers of visual mutation marker and depth mutation marker, or visual mutation marker and force mutation marker, or depth mutation marker and force mutation marker, then the node is set as the joint mutation degree transition point and a new critical window is opened. Continue scanning forward. When the joint mutation degree of the node is not lower than the low continuation threshold of the window and the time interval between the node and the last node of the opened critical window does not exceed 60 sampling points, the node is included in the current critical window. If a node with a joint mutation degree lower than the low continuation threshold or a time interval exceeding 60 sampling points occurs, the current critical window is closed and waits for the next node that meets the high trigger threshold condition to reopen. When each critical window closes, the target identification state mark is used to confirm the phase boundary based on the target identification state mark of the first transition point in the window, the approach start phase boundary is used to mark the first depth continuous convergence state mark in the window, the contact phase boundary is used to mark the first force rapid increase state mark in the window, the action execution phase boundary is used to mark the action execution phase boundary based on the gripper closure and end attitude stabilization state mark in the window, and the feedback result phase boundary is used to mark the task success or failure feedback state mark before the end of the window. The multimodal state data continuously stored between adjacent marked phase boundaries are cut and saved as independent phase boundary state fragments.

[0026] In this embodiment, the output memory state vector includes: Based on the field order of visual state, depth state, force state, pose state, and action state, five modal input branches are constructed for the improved scalar long short-term memory network sLSTM-T. Corresponding state reading units, state change recording units, and scalar memory units are set in each modal input branch. The improved scalar long short-term memory network sLSTM-T is constructed as follows: Based on the input gate, forget gate, output gate, and scalar memory unit of the traditional sLSTM network, this paper decomposes the single-path input structure of the traditional sLSTM into visual state input branches, depth state input branches, force state input branches, pose state input branches, and action state input branches. A state readout unit and a state change recording unit are set up for each branch. The unified forget gate in the traditional sLSTM is transformed into an adaptive forgetting unit that operates separately for each modality, determining the retention, decay, or masking operation based on the change amplitude, abrupt change marker, and the memory state of the previous phase boundary of the corresponding modality. In each modal input branch and the scalar memory unit... A modal trust gate is added between the scalar memory units. The modal trust gate receives the current modal state, the other modal states within the same phase boundary, and the hidden state of the previous phase boundary, and sends the output to the scalar memory unit of the corresponding modality. A cross-segment residual channel is added between adjacent phase boundary state segments. The cross-segment residual channel connects the memory state of the previous phase boundary, the input state of the current phase boundary, and the current scalar memory unit, and is used to write the difference part that is not interpreted by the current modality. Finally, the scalar memory states output from the five modal branches are concatenated in the order of visual, depth, force, pose, and action fields, and connected to the output gate to generate the memory state vector of the current task phase boundary. A modal trust gate is set between each modal input branch and the scalar memory unit. The modal trust gate reads the corresponding modal state, the hidden state of the previous phase boundary, and the successive states of other modes within the same phase boundary from the current phase boundary state segment, generating visual trust tags, depth trust tags, force trust tags, pose trust tags, and action trust tags. The construction of the modal trust gate is as follows: One-dimensional weighted registers are inserted into each of the five input branches: vision, depth, force, pose, and motion. Each branch reads the reference value of the current value of the current modality and the corresponding dimension of the hidden state of the previous phase boundary, and calculates the absolute magnitude of the difference between the two. Simultaneously, cross-monitoring buffers are established between each pair of modalities. Each buffer caches the synchronization trend of the two modalities at the current phase boundary in the most recent 6 sampling points, and outputs a conflict flag when the synchronization rate is lower than 0.65. The difference magnitude of the current modality is mapped to the 0-1 interval and multiplied by the difference weight of 0.55. The conflict flag of the cross-buffer is multiplied by the conflict weight of 0.35, and the residual error of the previous phase boundary is multiplied by the residual weight of 0.10. The three weighted results are summed to obtain the trust score of the current modality. Finally, the trust score is compared with a preset threshold of 0.48. If the trust score is not lower than the threshold, a trust flag of 1 is output; otherwise, a trust flag of 0 is output. The trust flags of vision, depth, force, pose, and motion are written to the corresponding scalar memory units respectively. An adaptive forgetting unit is set after the modal trust gate. Based on the modal trust markers, the state change amplitudes between adjacent phase boundary state segments, and the co-occurrence relationships of mutations in the sparse joint mutation graph, the adaptive forgetting unit determines the retained state, decayed state, and masked state of the corresponding modal memory. The construction of the adaptive forgetting unit is as follows: A first-level forgetting decision buffer is established in series after the modal trust gate of each modal input branch. First, the trust flag of the current modality, the state change amplitude from the previous phase boundary to the current phase boundary, and the number of cross-modal mutation co-occurrences connected to the current node in the sparse joint mutation graph are read. These three values ​​are mapped to three levels of scoring tables: retention, decay, and masking. If the trust flag is 1 and the state change amplitude falls within the range of the historical fluctuation mean of the current modality ± 1.0 standard deviation, and the number of cross-modal mutation co-occurrences does not exceed 1, the scoring table output is retained; if .... If the state change amplitude falls within the mean ± 2.5 standard deviations, or the number of cross-modal mutation co-occurrences is equal to 2, the scoring table output is decayed, the memory weight of this modality is multiplied by 0.55 and written back; if the trust flag is 0 or the state change amplitude exceeds the mean ± 2.5 standard deviations, or the number of cross-modal mutation co-occurrences reaches 3, the scoring table output is masked, the current input of this modality is completely set to zero and the update of the previous memory is frozen; the decision result is confirmed by the buffer and transmitted to the corresponding scalar memory unit to control the retention, decay or masking behavior of modal memory in this phase boundary; A cross-segment residual channel is set between adjacent phase boundary state segments. The cross-segment residual channel reads the difference between the previous phase boundary memory state and the current phase boundary input state that is not interpreted by the corresponding modality, and writes the difference into the visual residual, depth residual, force residual, pose residual and motion residual according to the modality fields. The construction of the cross-segment residual channel is as follows: A parallel residual bus is inserted between the five modal branches. After completing the adaptive forgetting decision, each modal branch immediately subtracts the current phase boundary input state from the previous phase boundary memory state element by element, and then takes the absolute value to obtain the difference vector. The difference vector is filtered according to the modal preset residual thresholds to remove minor fluctuations that are not enough to affect the task. The specific thresholds are: visual pixel deviation less than 12px, depth distance deviation less than 0.08m, force resultant force deviation less than 1.1N, end pose deviation less than 3.5mm, and motion speed deviation less than 14mm·s. -1 All components are set to zero; the remaining difference components are smoothed over time using a fourth-order circular buffer, and the smoothed residual values ​​for each mode are obtained by weighting the latest value (0.4), the first previous value (0.3), the second previous value (0.2), and the third previous value (0.1). The smoothed residual values ​​are written into the visual residual, depth residual, force residual, pose residual, and motion residual registers, respectively. The register outputs are connected to the current phase boundary scalar memory unit, so that the modal differences that are not currently interpreted can continue to be compensated or corrected in the next phase boundary. In the scalar memory unit, based on the output of the modal trust gate, the output of the adaptive forgetting unit, and the output of the cross-segment residual channel, scalar-level gain updates, scalar-level decay updates, and scalar-level masking updates are performed on the visual state, depth state, force state, pose state, and action state, respectively, to generate the scalar memory state corresponding to each modality. Specifically, the generation of the scalar memory state corresponding to each modality is as follows: Within the five scalar memory units of vision, depth, force, pose, and motion, the trust flag, forgetting decision flag, and cross-segment smoothing residual value of the current modality are read. If the trust flag is 1 and the forgetting decision is to retain, then 0.70 times the previous phase boundary modal memory value and 0.30 times the current input value are added together, and then 0.30 times the residual value is added, and the corresponding modal memory updates towards the latest input. If the trust flag is 1 and the forgetting decision is to decay, then 0.40 times the previous phase boundary modal memory value and 0.10 times the current input value are added together, and then 0.20 times the residual value is added, and the corresponding modal memory converges slowly in a conservative manner. If the trust flag is 0 or the forgetting decision is to mask, then only 0.10 times the previous phase boundary modal memory value is retained and the current input and residual input are masked, and the corresponding modal memory remains frozen. The update operation is performed in parallel in the five modal branches. After completion, the new scalar memory values ​​of vision, depth, force, pose, and motion are written sequentially into the current phase boundary memory vector. The scalar memory states corresponding to each modality are combined according to the field order of visual state, depth state, force state, pose state and action state to generate the current phase boundary memory state. The current hidden state is updated based on the current phase boundary memory state, and the current phase boundary memory state and the current hidden state are jointly determined as the memory state vector of the corresponding task phase boundary.

[0027] In this embodiment, the introduction of modal pruning gates to remove redundant dimensions, when both types of errors fall within the dynamic threshold region and the cohomology kernel rank remains unchanged after pruning, solidifies the corresponding phase boundary state fragment into a locked empirical structure, including: Read the memory state vector corresponding to the k-th task boundary, and split the memory state vector into state memory blocks, action memory blocks, and feedback memory blocks according to the state field, action field, and feedback field. Construct a multimodal locking coordinate set according to the field order of visual state, depth state, force state, pose state, and action state. Specifically, the splitting of the memory state vector into state memory blocks, action memory blocks, and feedback memory blocks according to the state field, action field, and feedback field is as follows: A fixed field index is assigned to the memory state vector: bits 0-127 are for visual state, bits 128-191 for depth state, bits 192-223 for force state, bits 224-271 for pose state, bits 272-335 for motion state, and bits 336-351 for feedback result. During splitting, bits 0-271 are first copied consecutively to form a state memory block, completely preserving the information of the four modalities: visual, depth, force, and pose. Then, bits 272-335 are copied to form a motion memory block, corresponding to control variables such as approach direction, end effector posture, execution speed, and gripping force at the current phase boundary. Finally, bits 336-351 are copied to form a feedback memory block, saving result information such as successful grasping label, lateral displacement, peak gripping force, and task time. After splitting, the five modalities are arranged sequentially within the state memory block. Based on a multimodal locking coordinate set, forward and reverse self-homeomorphic mappings are constructed. The forward self-homeomorphic mapping maps state memory blocks and action memory blocks to feedback memory blocks, while the reverse self-homeomorphic mapping maps feedback memory blocks and action memory blocks to key state sub-blocks within the state memory block. These forward and reverse self-homeomorphic mappings are combined into a multimodal self-homeomorphic mapping pair. Specifically, the construction of forward and reverse self-homeomorphic mappings based on the multimodal locking coordinate set is as follows: The state and action memory blocks of all locking experience structures in the training samples are concatenated column-wise to form a source matrix of size N×336, where N is the number of locking experience entries, the first 272 columns are state fields, and the last 64 columns are action fields. The corresponding feedback memory blocks are then organized into a target matrix of size N×16. Least square regression with a ridge coefficient of 0.01 is used to solve for the mapping weight matrix on the source and target matrices, resulting in a forward mapping of size 16×336. Singular value decomposition is performed on the forward mapping, and the first r rows with a cumulative singular value contribution rate of 95% are truncated to generate a compressed mapping, i.e., a forward automorphic mapping matrix. A reverse automorphic mapping matrix is ​​then constructed. During homeomorphism mapping, the key state dimensions with the top 80 mutual information values ​​in the state memory block are first selected by sorting the state-feedback mutual information to generate key state sub-blocks. Then, the feedback memory block and the action memory block are concatenated column by column to form a target matrix and a source matrix of size N×80. The reverse mapping W_rev of size 80×80 is obtained by using the same ridge regression process as the forward mapping. Singular value compression is also performed to obtain the reverse homeomorphism mapping matrix. The forward homeomorphism mapping matrix and the reverse homeomorphism mapping matrix are encapsulated into a multimodal homeomorphism mapping pair. The mapping pair and the corresponding locking coordinate set are written into the locking experience structure. A sparse cohomology kernel is established based on the temporal continuity, modal correlation, and action feedback correlation among coordinate points in the multimodal locking coordinate set. The differences between the output of the forward automorphic mapping and the feedback memory block, and the differences between the output of the reverse automorphic mapping and the key state sub-block are read respectively to generate the forward locking error and the reverse redemption error. Specifically, the sparse cohomology kernel is established as follows: The process involves reading the time sequence number, modal type, action field number, and feedback field number of all coordinate points in the multimodal locking coordinate set, and recording each coordinate point as a kernel node. Then, the sampling intervals between adjacent kernel nodes are compared in chronological order. When the sampling interval does not exceed 80ms, a temporal continuation edge is established between the corresponding kernel nodes. Next, the field correlation coefficients between different modal kernel nodes are compared. When the correlation coefficient between the visual field and the depth field, the depth field and the force field, or the pose field and the action field is not less than 0.72, a modal association edge is established between the corresponding kernel nodes. Then, the mapping contribution value between the action field and the feedback field is read. When the contribution of the action field to the feedback field is not less than 15%, an action-feedback association edge is established between the corresponding kernel nodes. After all edges are established, only node combinations that simultaneously possess temporal continuation edges and modal association edges, or simultaneously possess temporal continuation edges and action-feedback association edges, are retained. The remaining isolated nodes and single-edge nodes are deleted, ultimately forming a sparse coherence kernel composed of temporal continuation relationships, modal association relationships, and action-feedback association relationships. The positive locking error and the reverse redemption error are generated as follows: The state memory block and action memory block are input into the forward automorphism mapping to generate the prediction feedback vector. Then, the prediction feedback vector is compared with the true feedback memory block field by field. For each field, the absolute difference between the predicted value and the true value is calculated. All differences are accumulated and divided by the number of feedback fields to obtain the forward locking error. Then, the feedback memory block and action memory block are input into the reverse automorphism mapping to generate the prediction key state vector. Then, the prediction key state vector is compared with the key state sub-block field by field. For each field, the absolute difference between the predicted value and the true value is calculated. All differences are accumulated and divided by the number of key state fields to obtain the reverse redemption error. Finally, the forward locking error and the reverse redemption error are written into the kernel attribute table of the corresponding sparse coherent kernel. Visual state, depth state, force state, pose state, and action state are input into the modal pruning gate. Based on the influence of each modal dimension on forward locking error, backward redemption error, and sparse homology kernel rank, redundant dimensions are removed, retaining only the state dimension, action dimension, and feedback dimension that maintain the sparse homology kernel rank. This generates a pruned locking coordinate set, specifically as follows: Following the field order of visual state, depth state, force state, pose state, and motion state, individual dimensions in the multimodal locking coordinate set are selected one by one as dimensions to be pruned. First, the dimensions to be pruned and their connected temporal continuation edges, modal association edges, and motion feedback association edges are temporarily deleted from the multimodal locking coordinate set to generate a temporary locking coordinate set. Based on this temporary locking coordinate set, the forward locking error, reverse redemption error, and sparse homology kernel rank are recalculated. The recalculated results are compared with the forward locking error, reverse redemption error, and sparse homology kernel rank before deletion. After deleting the dimensions to be pruned, the forward locking error... If the increase in value does not exceed 0.03, the increase in reverse redemption error does not exceed 0.03, and the sparse homology kernel rank remains the same as before deletion, the dimension is determined as a redundant dimension and permanently deleted; if, after deleting the dimension to be pruned, the increase in forward locking error exceeds 0.03, the increase in reverse redemption error exceeds 0.03, or the sparse homology kernel rank changes, the dimension to be pruned and its connecting edges are restored; after completing the traversal of all dimensions in the above manner, the retained state dimension, action dimension, feedback dimension, and corresponding time continuation edges, modal association edges, and action feedback association edges are combined to generate the pruned locking coordinate set; Based on the pruned locking coordinate set, the forward locking error, reverse redemption error, and sparse homology kernel rank are recalculated. When the forward locking error falls into the forward dynamic threshold region, the reverse redemption error falls into the reverse dynamic threshold region, and the sparse homology kernel rank before and after pruning is consistent, the state memory block, action memory block, feedback memory block, multimodal automorphic mapping pair, and the pruned locking coordinate set of the corresponding phase boundary state segment are combined into a locking empirical structure. Specifically, the forward locking error, reverse redemption error, and sparse homology kernel rank are recalculated based on the pruned locking coordinate set as follows: Read the state dimension, action dimension, and feedback dimension retained in the locked coordinate set after pruning. Rearrange the values ​​corresponding to the retained state dimension in the original field order to form a pruning state memory block, rearrange the values ​​corresponding to the retained action dimension in the original field order to form a pruning action memory block, and rearrange the values ​​corresponding to the retained feedback dimension in the original field order to form a pruning feedback memory block. Input the pruning state memory block and the pruning action memory block into a forward automorphic mapping to obtain the pruning prediction feedback block. Subtract the pruning prediction feedback block from the pruning feedback memory block field by field and take the absolute value. Add all the absolute differences and divide by the number of pruning feedback fields to obtain the recalculated positive feedback. To calculate the locking error; input the pruning feedback memory block and the pruning action memory block into the reverse automorphism mapping to obtain the pruning prediction key state block. Subtract the key state fields in the pruning prediction key state block from the key state fields in the pruning state memory block field by field and take the absolute value. Add all the absolute differences and divide by the number of key state fields to obtain the recalculated reverse redemption error. Read the time continuation edges, modal association edges and action feedback association edges retained in the pruning locking coordinate set. Generate a pruning adjacency matrix according to the node number. Write 1 for the positions with connections and 0 for the positions without connections in the matrix. Perform rank calculation on the pruning adjacency matrix to obtain the recalculated sparse homology kernel rank. The construction of the locking empirical structure is as follows: First, assign a unique index number to the current phase boundary state segment. Then, write the pruned state vector, action vector, and feedback vector into three fixed-length storage areas in sequence. The state vector area occupies 0 to 271 bits according to the four types of information: visual, depth, force, and pose. The action vector area occupies 272 to 335 bits, and the feedback vector area occupies 336 to 351 bits. Register the starting address of the state vector, the starting address of the action vector, the starting address of the feedback vector, the address of the mapping pair storage page, and the address of the locking coordinate set adjacency matrix corresponding to the index number in the locking experience structure index table. First, write the compressed forward mapping matrix, then write the compressed reverse mapping matrix. The total page length is fixed at 4096 bytes. Read the time continuation edges, modal association edges, and action feedback association edges retained after pruning. Multiply the starting point number of each edge by 65,536 and add the ending point number to generate a 32-bit integer key. Write the keys sequentially into the sparse edge key table and establish an association with the adjacency matrix address. Record the three indicators corresponding to the index number in the locking experience directory: forward locking error, reverse redemption error, and sparse homology kernel rank. Set the status flag to one to indicate that the experience has been solidified.

[0028] In this embodiment, generating the minimum empirical subset includes: Read the state memory block, action memory block, feedback memory block, multimodal automorphic mapping pairs and pruned locking coordinate set from each locking experience structure, and generate an experience structure index for each locking experience structure according to the task number, task boundary type, action primitive type and task result type. State continuity edges are established based on the continuity relationship of state memory blocks between adjacent locking experience structures; action continuity edges are established based on the continuity relationship of action memory blocks between adjacent locking experience structures; and feedback generation edges are established based on the fact that the feedback memory block of the previous locking experience structure can serve as the state input condition for the subsequent locking experience structure. Locking experience structures are treated as graph nodes, and state continuity edges, action continuity edges, and feedback generation edges are treated as graph edges, thus generating an experience generation relationship graph. Specifically, the generation of the experience generation relationship graph is as follows: Arrange all locking experience structures in ascending order by task number, and then in ascending order by phase boundary within the same task number, to obtain a time-progressing index sequence. Read adjacent locking experience structures from the sequence. First, compare the Euclidean distance between the first 272 bits of the state vector of the later locking experience structure and the last 272 bits of the state vector of the earlier locking experience structure. If the distance does not exceed 0.25, add a state continuation edge between the two locking experience structures. Next, compare the mean absolute difference between the 64 bits of the action vector of the later locking experience structure and the 64 bits of the action vector of the earlier locking experience structure. If the mean does not exceed 0.12 and the direction cosine is not less than 0.91, add an action continuation edge and check the previous locking... The correlation rate between the 16-bit feedback vector of the experience structure and the corresponding fields of the task success mark, lateral displacement, and peak clamping force in the state vector of the next locking experience structure is calculated. When all three items meet the correlation threshold of 90%, a feedback generation edge is added. For each edge creation operation, the starting index is multiplied by the maximum value of the node number + 1 and then the ending index is added to generate a unique edge key. The edge key is written into the state edge key table, action edge key table, and feedback edge key table according to the edge type. After completing the traversal of all sequences, the locking experience structure index is used as the node number, and the edge keys recorded in the three types of edge key tables are used as the connection relationship. The corresponding position in the adjacency matrix is ​​set to 1, and the rest are set to zero. Finally, an experience generation relationship graph containing state continuation edges, action continuation edges, and feedback generation edges is obtained. In the experience generation relationship graph, read the incoming edge source, outgoing edge destination and feedback generation path of each locking experience structure. In accordance with the order of consistent state coverage, consistent action acceptance and consistent feedback results, register the generation path formed by the combination of multiple locking experience structures and generate the combination generation record of the corresponding locking experience structure. Select each locking experience structure as the experience structure to be stripped, temporarily delete the experience structure to be stripped and the associated graph edges, and check whether the remaining locking experience structures can cover the state memory block, action memory block and feedback memory block corresponding to the experience structure to be stripped based on the combined generation record. When the remaining locking experience structures can completely cover the state memory block, action memory block and feedback memory block, mark the experience structure to be stripped as a redundant experience structure. Delete all redundant experience structures and retain the lock experience structures that cannot be generated by combining the remaining lock experience structures. Combine the retained lock experience structures and their corresponding state continuation edges, action continuation edges and feedback generation edges into a minimum experience generation subset.

[0029] In this embodiment, the generation of the action strategy, which controls the robot to perform the current task based on the action strategy, includes: Collect the visual state, depth state, force state, pose state, gripper state, and task target state in the current task, generate the current task state fragment according to the order of the multimodal state sequence fields, and extract the current state constraints, current action requirements, and target feedback constraints based on the current task state fragment; The current state constraints, current action requirements, and target feedback constraints are converted into the current task query structure. The state memory block, action memory block, feedback memory block, and pruned locking coordinate set of each locking experience structure in the minimum experience generation subset are read. Candidate locking experience structures are selected according to the following order: consistency of task boundary fields, coverage of state fields, acceptability of target feedback fields, and satisfaction of modal reliability fields. Specifically, the selection of candidate locking experience structures involves: Read the task boundary type number, state field vector, action requirement vector, and target feedback field vector from the current task query structure. Compare the task boundary type number with the boundary type number of each closed experience structure in the minimum experience generation subset. Only the closed experience structures with completely identical numbers are retained and added to the first round list. For each closed experience structure in the first round list, compare the difference between the state vector (0 to 271) and the corresponding bit of the current state field vector. When all differences do not exceed the preset state coverage threshold of 0.22 and the coverage ratio is not less than 90%, the structure is retained and added to the second round list. Read the feedback vector from 336 to 351 of each locking experience structure in the second round list, and compare the three fields of task success mark, lateral displacement target, and clamping force peak value with the target feedback field one by one. When the task success mark can be aligned, the lateral displacement difference is not greater than 2.5 mm, and the clamping force peak value difference is not greater than 0.8 N, it is retained and entered into the third round list. Read the modal confidence field flags recorded in the pruned locking coordinate set for each locking experience structure in the third round list. When the visual confidence flag, depth confidence flag, and force confidence flag are all one, and at least one of the pose confidence flag or action confidence flag is one, add the locking experience structure to the candidate set. After completing the traversal, sort the candidate set from high to low according to the state coverage ratio. If the coverage ratios are the same, sort them from small to large according to the difference in the peak clamping force. Output the sorted candidate locking experience structures. Based on the state continuity edges, action continuity edges, and feedback generation edges between candidate locking experience structures, the candidate locking experience structures are concatenated and sorted. Candidate locking experience structures that have modal conflicts with the current task state segment, are unreachable by actions, or have inconsistent feedback are deleted. The current task reconstruction structure is then generated, specifically as follows: Read the node numbers of any two empirical structures from the candidate locking empirical structure list. If there is a state continuity edge between the two in the empirical generation relationship graph and the continuity interval does not exceed four phase boundaries, then establish a sequential splicing relationship between the two nodes. Then check the motion vectors of adjacent nodes in the splicing chain. If the angle difference between the approach direction of the motion vector of the end node and the approach direction of the motion vector of the next node is less than 15 degrees and the difference in the clamp closing speed does not exceed 12 mm / s, then retain the splicing chain. Read the feedback vectors and state vectors of adjacent nodes in the splicing chain. If the difference between the peak clamping force in the feedback of the previous node and the allowable force threshold in the state of the next node is not greater than 0.9 Newtons and the attitude deviation is not greater than 4 mm, then the feedback generation relationship is considered valid. After completing the detection of all splicing chains, sort the splicing chains from longest to shortest chain length and retain only the first one. Three concatenated chains with the longest length that satisfy action continuity and feedback consistency are selected as candidate chains. A node-by-node conflict check is performed on the modal trust flag of each node in each candidate chain. When either the visual trust flag or the depth trust flag is zero and the conflict persists for more than two nodes, the corresponding node and its subsequent nodes are removed from the chain. The action reachability of the chain is recalculated. If the Euclidean distance between the end pose of any node and the relative pose of the target is greater than seven millimeters or the attitude rotation angle is greater than eight degrees, the node and its subsequent nodes are removed. The feedback consistency of the three chains after the removal operation is re-evaluated. If the number of remaining nodes in any chain is less than two or the task success flag of the end node is inconsistent with the target feedback requirements, the entire chain is deleted. Finally, the nodes in the remaining chains are concatenated in chronological order to form the current task reconstruction structure. Read the action memory block, multimodal automorphic mapping pairs, and feedback generation edges from the current task reconstruction structure. Map the current state constraints to the action fields to be generated, and generate the action strategy. The action strategy includes approach direction, end effector attitude, action speed, gripping force, gripper opening and closing amount, obstacle avoidance distance, and abnormal retreat action. Specifically, the generated action strategy is as follows: The algorithm reads the action memory blocks arranged chronologically in the current task reconstruction structure, and extracts the approach direction, end effector attitude, action speed, gripping force, gripper opening / closing amount, obstacle avoidance distance, and abnormal retreat action fields from each action memory block to generate a candidate action field table. Then, it reads the target center position, target depth leading edge, current end effector attitude, current gripper opening / closing amount, and nearest obstacle distance from the current state constraints, inputting these state fields into the inverse mapping channel of the multimodal automorphic mapping pair to obtain the action fields to be generated corresponding to the current state constraints. Finally, it compares each action field to be generated with the candidate action field table, retaining those with an approach direction difference of no more than 12 degrees, an end effector attitude deviation of no more than 5 mm, an action speed difference of no more than 15 mm / s, a gripping force difference of no more than 1 Newton, and a gripping... Candidate action fields with a claw opening / closing difference of no more than six millimeters are selected. Based on the feedback generation, the historical feedback results corresponding to each candidate action field are read, and candidate action fields with lateral displacement exceeding five millimeters, peak clamping force exceeding the safety threshold, or marked as failed in the corresponding historical feedback are deleted. Then, the remaining candidate action fields are sorted from high to low according to state coverage ratio, feedback consistency, and action change amount. The candidate action field with the highest ranking is selected as the main action field, and the obstacle avoidance distance is corrected according to the nearest distance to the current obstacle. When the nearest distance to the obstacle is less than 200 millimeters, the obstacle avoidance distance is increased by 30 millimeters. The approach direction, end attitude, action speed, clamping force, claw opening / closing amount, corrected obstacle avoidance distance, and abnormal retreat action in the main action field are combined to generate the action strategy for the current task. The motion strategy is sent to the robot motion controller, robotic arm controller, and gripper controller to control the robot to execute the current task, and the actual feedback state, motion execution state, and task result state after the current task is executed are written into a new multimodal state sequence.

[0030] Example 1: In an indoor item organization scenario, the robot needs to identify and grasp transparent cups, metal cans, cardboard boxes, and soft packaging bags from the control panel and place the target items into the corresponding storage boxes. The scenario includes reflective surfaces, partial occlusion, and temporary obstacles. The robot is equipped with one RGB camera, one depth camera, one six-dimensional force sensor, six robotic arm joint encoders, one chassis odometer, and one electric gripper. This example selects 1200 historical organization tasks as training samples and 300 new tasks not used in the training as test samples. In the training samples, there are 320 transparent cup tasks, 280 metal can tasks, 300 cardboard box tasks, and 300 soft packaging bag tasks. Each task lasts an average of 18.4 seconds. The RGB image resolution is 640×480 pixels, the depth point cloud contains approximately 28,600 valid points per frame, the force sampling frequency is 200Hz, the joint state sampling frequency is 50Hz, and the chassis pose sampling frequency is 50Hz.

[0031] After the data enters the processing flow, the system first performs unified preprocessing on the multimodal raw data. RGB images are read continuously according to task number, resulting in 216,000 frames. Approximately 34% of the transparent cup images have reflective bright areas, with an average brightness fluctuation of 0.27. The system normalizes the image brightness to the 0-1 range and crops the target candidate regions. The average size of the cropped target region is 184×176 pixels. Invalid points less than 0.18 meters and greater than 3.2 meters are removed from the depth point cloud. Before processing, the average number of outliers per frame was 2140, which was reduced to 326 after processing. Force perception data is filtered using a 5-point median filter. Before filtering, the peak value of the idle fluctuation was 0.42N, which was reduced to 0.11N after filtering. All data are aligned 20ms to form 1200 multimodal state sequences, with each sequence containing an average of 920 state points.

[0032] Taking a single training sample of grasping a transparent cup as an example, the robot's visual target bounding box jittered continuously in the initial stage, with the target center offset reaching a maximum of 23 pixels in the first 10 frames. After preprocessing, the target center offset was reduced to 8 pixels. The system constructed a sparse joint mutation graph for this multimodal state sequence, using each 20ms sampling point as a graph node, generating a total of 934 graph nodes. At node 126, the visual target bounding box stably overlapped, and the visual change amplitude decreased from 0.31 to 0.08; at node 264, the depth leading edge distance changed from 0.74 meters to 0.39 meters; at node 392, the end contact force increased from 0.3N to 3.9N; at node 421, the gripper opening and closing distance decreased from 64 mm to 41 mm; and at node 438, the lateral force increased from 0.7N to 2.6N, accompanied by a target center offset of 12 pixels. The system established target confirmation phase boundaries, approach initiation phase boundaries, contact establishment phase boundaries, action execution phase boundaries, and feedback result phase boundaries at these nodes, respectively. The traditional fixed window method divides the task into 37 segments, while the present invention obtains 5 phase boundary state segments, each of which corresponds to a clear task state change.

[0033] In the improved sLSTM-T network processing, the system establishes input branches for vision, depth, force, pose, and motion. Taking the 392nd contact as an example to establish a phase boundary, the confidence level of the target position displayed by the visual state is 0.61, the confidence level of the target leading edge displayed by the depth state is 0.84, the confidence level of the contact peak displayed by the force state is 0.91, the confidence level of the end effector arrival error displayed by the pose state is 4.2 mm, and the confidence level of the gripper closing speed displayed by the motion state is 32 mm / s. The modal trust gate determines the trust values ​​as follows: visual trust value is 0.58, depth trust value is 0.82, force trust value is 0.93, pose trust value is 0.88, and motion trust value is 0.76. Due to a 16 mm deviation between the visual and depth centers, the visual branch enters attenuation update; due to the consistency between the force contact peak and the end effector position change, the force branch enters gain update; and due to the visual offset remaining in the previous phase boundary not being interpreted by the current depth, the cross-segment residual channel writes 0.17 to the visual residual. After the sLSTM-T update, the force response in this phase boundary memory state increased from 0.52 to 0.81, the visual offset response decreased from 0.49 to 0.22, and the pose response remained at 0.74.

[0034] The system solidifies the locked-in empirical structure of the memory state vector. Taking the transparent cup sample as an example, the state memory block records the target category, target center, cup depth leading edge, and end pose; the action memory block records the approach direction as 26 degrees to the left front, the gripper closing speed as 18 mm / s, and the gripping force as 6.2 N; and the feedback memory block records the target lateral displacement as 2.8 mm and the grasping result as successful. After constructing multimodal automorphic mapping pairs, the positive locking error is 0.064, and the negative redemption error is 0.071, both falling within the task's dynamic threshold of 0.12. The modal pruning gate sequentially removes the background color, ambient brightness, and non-contact joint micro-vibration dimensions. Before pruning, the sparse homology kernel rank was 6, and it remained 6 after pruning. When attempting to remove the force peak dimension, the homology kernel rank decreased from 6 to 4, but the system retained this dimension. This segment was ultimately solidified into a locked-in empirical structure. Statistics from 1200 training tasks yielded a total of 7318 phase boundary state fragments, of which 5156 were solidified into locked empirical structures, 1435 were recorded as states to be verified, and 727 were rejected from being written due to the invalidity of bidirectional derivation.

[0035] The system further constructs an experience generation relationship graph. 5156 locking experience structures serve as graph nodes, with 6284 state continuation edges, 4917 action continuation edges, and 3726 feedback generation edges. The system performs redundancy stripping on each locking experience structure. For example, sample A is a successful experience of "side-front approach + low-speed closure + force confirmation" for a transparent cup; sample B is a successful experience of "side-front approach + low-speed closure + force confirmation" for a metal can; and sample C is a sliding experience of "direct-front approach + fast closure + visual low trust" for a transparent cup. The system finds that the stable contact portion in sample B can be generated by combining the approach structure of sample A with the feedback structure on the surface of another metal can, and therefore strips the repetitive approach structures in sample B. After stripping, the 5156 locking experience structures are compressed into 812 minimal experience generators, reducing storage usage from 1.28GB to 0.31GB, and the average time for new task matching from 386ms in the traditional experience library to 104ms.

[0036] In 300 new task tests, taking a transparent cup sorting task that was not trained as an example, the robot's initial visual target center deviation was 19 pixels, and the depth point cloud of the cup rim region was missing about 22%. The traditional method matched a historical trajectory that approached from directly in front and closed the gripper at a speed of 31 mm / s. After execution, the cup body's lateral displacement reached 13.4 mm, the gripper was adjusted once, and the task took 23.6 seconds. After reading the current state, this invention matched the locking experience structure of "transparent target, low visual trust, local depth loss, and force-sensory priority confirmation" from the minimum experience generation subset, and reconstructed the action strategy of approaching from the left front at 28 degrees, end-effector downward speed of 36 mm / s, gripper closing speed of 17 mm / s, and force trigger threshold of 3.2 N. After the robot executed the action, the cup body's lateral displacement was 2.9 mm, the maximum gripping force was 6.4 N, no secondary adjustment was required, and the task took 16.9 seconds.

[0037] In the same batch of 300 test tasks, the overall success rate of the traditional method was 82.7%, while that of this invention was 95.0%; the success rate for transparent and reflective targets was 73.1% for the traditional method and 92.3% for this invention; the success rate for flexible packaging bags was 80.8% for the traditional method and 93.6% for this invention; the average task completion time was 21.4 seconds for the traditional method and 17.2 seconds for this invention; the average number of replanning attempts was 1.16 for the traditional method and 0.39 for this invention; the number of clamping force exceeding limits was 29 for the traditional method and 6 for this invention; the number of failures due to incorrect experience calls was 27 for the traditional method and 4 for this invention; and the average matching time of the experience base was 386ms for the traditional method and 104ms for this invention.

[0038] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A robot experience learning system based on a multimodal memory network, characterized in that, include: The multimodal preprocessing module collects multimodal raw data during the robot's task execution process, preprocesses the multimodal raw data, and generates a multimodal state sequence. The phase boundary mutation detection module constructs a sparse joint mutation map for the multimodal state sequence, extracts the time stamps of visual mutation events, depth mutation events and force-sensory mutation events respectively, calculates the joint mutation degree of the three modes and uses a dual-threshold critical window to detect the joint mutation degree transition point, and generates the corresponding phase boundary state segment. The memory update module introduces the phase boundary state fragment input into the improved sLSTM-T network with modal trust gate, adaptive forgetting coefficient and cross-fragment residual channel, and performs scalar gain, attenuation and masking updates on visual state, depth state, force state, pose state and action state respectively, and outputs memory state vector. The locking experience solidification module constructs multimodal automorphic mapping pairs for the memory state vector, calculates the positive locking error and the reverse redemption error using sparse cohomology kernels, introduces a modal pruning gate to remove redundant dimensions, and solidifies the corresponding phase boundary state fragment into a locking experience structure when both types of errors fall into the dynamic threshold region and the cohomology kernel rank remains unchanged after pruning. The sub-compression module reads the state succession relationship, action succession relationship and feedback generation relationship between each locking experience structure, constructs the experience generation relationship graph, performs redundancy stripping on each locking experience structure, and generates the minimum experience generation subset. The motion reconfiguration control module collects multimodal state data corresponding to the current task when the robot performs a new task, matches the corresponding locking experience structure from the minimum experience generation subset and performs structure reconfiguration to generate a motion strategy, and controls the robot to perform the current task according to the motion strategy.

2. The robot experience learning system based on a multimodal memory network according to claim 1, characterized in that, The multimodal raw data includes visual image data, depth point cloud data, force feedback data, joint state data, chassis pose data, gripper state data, motion control data, and task result data.

3. The robot experience learning system based on a multimodal memory network according to claim 1, characterized in that, The preprocessing of the multimodal raw data to generate a multimodal state sequence includes: The visual image data, depth point cloud data, force feedback data, joint state data, chassis pose data, gripper state data, and motion control data are time-aligned according to a unified timestamp. Perform target region cropping, brightness normalization, and edge enhancement processing on visual image data to generate a visual state sequence; Perform outlier removal, spatial registration, and distance mapping on the deep point cloud data to generate a depth state sequence; Noise filtering, contact peak detection, and torque fluctuation segmentation are performed on the force feedback data to generate a force state sequence. Trajectory smoothing and state synchronization processing are performed on joint state data, chassis pose data, and gripper state data to generate a pose and motion state sequence. The visual state sequence, depth state sequence, force state sequence, and pose and motion state sequence are spliced ​​and encoded in chronological order to generate a multimodal state sequence.

4. The robot experience learning system based on a multimodal memory network according to claim 1, characterized in that, The generation of the corresponding phase boundary state fragment includes: Read the visual state, depth state, and force state in the multimodal state sequence according to the sampling time order, compare the visual change amplitude, depth change amplitude, and force change amplitude between the current sampling time and the previous sampling time, and generate the visual change record, depth change record, and force change record for the corresponding sampling time. Visual change records, depth change records, and force change records are compared with their respective preset mutation judgment conditions. When the amplitude of visual change reaches the visual mutation judgment condition, a visual mutation mark is generated; when the amplitude of depth change reaches the depth mutation judgment condition, a depth mutation mark is generated; and when the amplitude of force change reaches the force mutation judgment condition, a force mutation mark is generated. Using each sampling time as a graph node, time continuity edges are established between adjacent graph nodes according to the sampling time sequence. When two mutation markers of visual mutation marker, depth mutation marker, or force mutation marker appear at the same sampling time, cross-modal mutation co-occurrence relationship is established on the corresponding graph node to generate a sparse joint mutation graph. Based on the visual change amplitude, depth change amplitude, force change amplitude, and number of cross-modal mutation co-occurrence relationships corresponding to each graph node, the three-modal joint mutation degree of the corresponding graph node is generated, and the three-modal joint mutation degree is written into the sparse joint mutation graph in the order of sampling time. A dual-threshold critical window is constructed using high triggering conditions and low continuity conditions to continuously detect the trimodal joint abruptness of each graph node. Graph nodes that meet the high triggering conditions and have two abruptness markers are identified as joint abruptness transition points. Graph nodes that meet the low continuity conditions and are temporally continuous with the joint abruptness transition points are incorporated into the corresponding critical window. Based on the target recognition state, approach distance state, contact force state, action execution state, and task feedback state at the joint abruptness transition point, the target confirmation phase boundary, approach initiation phase boundary, contact establishment phase boundary, action execution phase boundary, and feedback result phase boundary are marked respectively. The multimodal state data between adjacent task phase boundaries are identified as phase boundary state segments.

5. A robot experience learning system based on a multimodal memory network according to claim 1, characterized in that, The output memory state vector includes: Based on the field order of visual state, depth state, force state, pose state, and action state, five modal input branches are constructed for the improved scalar long short-term memory network sLSTM-T. Corresponding state reading units, state change recording units, and scalar memory units are set in each modal input branch. A modal trust gate is set between each modal input branch and the scalar memory unit. The modal trust gate reads the corresponding modal state, the hidden state of the previous phase boundary, and the succession state of other modes in the same phase boundary from the current phase boundary state segment, and generates visual trust markers, depth trust markers, force trust markers, pose trust markers and motion trust markers. An adaptive forgetting unit is set after the modal trust gate. The adaptive forgetting unit determines the retention state, decay state and shielding state of the corresponding modal memory based on the trust markers of each modality, the state change amplitude between adjacent phase boundary state segments and the mutation co-occurrence relationship in the sparse joint mutation graph. A cross-segment residual channel is set between adjacent phase boundary state segments. The cross-segment residual channel reads the difference between the previous phase boundary memory state and the current phase boundary input state that is not interpreted by the corresponding modality, and writes the difference into the visual residual, depth residual, force residual, pose residual and motion residual according to the modality field. Based on the output of the modal trust gate, the output of the adaptive forgetting unit, and the output of the cross-fragment residual channel, scalar gain update, scalar decay update, and scalar masking update are performed on the visual state, depth state, force state, pose state, and action state, respectively, to generate the scalar memory state corresponding to each modality. The scalar memory states corresponding to each modality are combined according to the field order of visual state, depth state, force state, pose state and action state to generate the current phase boundary memory state. The current hidden state is updated based on the current phase boundary memory state, and the current phase boundary memory state and the current hidden state are jointly determined as the memory state vector of the corresponding task phase boundary.

6. A robot experience learning system based on a multimodal memory network according to claim 1, characterized in that, The introduction of modal pruning gates to remove redundant dimensions, when both types of errors fall within the dynamic threshold region and the cohomology kernel rank remains unchanged after pruning, solidifies the corresponding phase boundary state fragment into a locked empirical structure, including: Read the memory state vector corresponding to the kth task boundary, and split the memory state vector into state memory block, action memory block and feedback memory block according to the state field, action field and feedback field. Construct a multimodal locking coordinate set according to the field order of visual state, depth state, force state, pose state and action state. Based on the multimodal locking coordinate set, a forward automorphic mapping and a reverse automorphic mapping are constructed. The forward automorphic mapping maps the state memory block and action memory block to the feedback memory block, and the reverse automorphic mapping maps the feedback memory block and action memory block to the key state sub-block in the state memory block. The forward automorphic mapping and the reverse automorphic mapping are combined into a multimodal automorphic mapping pair. Based on the temporal continuity, modal correlation, and action feedback correlation among coordinate points in the multimodal locking coordinate set, a sparse coherence kernel is established. The differences between the output of the forward automorphic mapping and the feedback memory block, and the differences between the output of the reverse automorphic mapping and the key state sub-block are read respectively to generate the forward locking error and the reverse redemption error. Visual state, depth state, force state, pose state and action state are respectively input into the modal pruning gate. Based on the influence of each modal dimension on the forward locking error, reverse redemption error and sparse homology kernel rank, redundant dimensions are deleted, and the state dimension, action dimension and feedback dimension that make the sparse homology kernel rank unchanged are retained to generate the pruned locking coordinate set. Based on the pruned locking coordinate set, the forward locking error, reverse redemption error, and sparse homology kernel rank are recalculated. When the forward locking error falls into the forward dynamic threshold region, the reverse redemption error falls into the reverse dynamic threshold region, and the sparse homology kernel rank before and after pruning is consistent, the state memory block, action memory block, feedback memory block, multimodal automorphic mapping pair, and pruned locking coordinate set of the corresponding phase boundary state segment are combined into a locking empirical structure.

7. A robot experience learning system based on a multimodal memory network according to claim 1, characterized in that, The generation of the minimum empirical subset includes: Read the state memory block, action memory block, feedback memory block, multimodal automorphic mapping pairs and pruned locking coordinate set from each locking experience structure, and generate an experience structure index for each locking experience structure according to the task number, task boundary type, action primitive type and task result type. State continuity edges are established based on the continuity relationship of state memory blocks between adjacent locking experience structures. Action continuity edges are established based on the continuity relationship of action memory blocks between adjacent locking experience structures. Feedback generation edges are established based on the fact that the feedback memory block of the previous locking experience structure can serve as the state input condition of the next locking experience structure. Locking experience structures are used as graph nodes, and state continuity edges, action continuity edges, and feedback generation edges are used as graph edges to generate an experience generation relationship graph. In the experience generation relationship graph, read the incoming edge source, outgoing edge destination and feedback generation path of each locking experience structure. In accordance with the order of consistent state coverage, consistent action acceptance and consistent feedback results, register the generation path formed by the combination of multiple locking experience structures and generate the combination generation record of the corresponding locking experience structure. Select each locking experience structure as the experience structure to be stripped, temporarily delete the experience structure to be stripped and the associated graph edges, and check whether the remaining locking experience structures can cover the state memory block, action memory block and feedback memory block corresponding to the experience structure to be stripped based on the combined generation record. When the remaining locking experience structures can completely cover the state memory block, action memory block and feedback memory block, mark the experience structure to be stripped as a redundant experience structure. Delete all redundant experience structures and retain the lock experience structures that cannot be generated by combining the remaining lock experience structures. Combine the retained lock experience structures and their corresponding state continuation edges, action continuation edges and feedback generation edges into a minimum experience generation subset.

8. A robot experience learning system based on a multimodal memory network according to claim 1, characterized in that, The generated action strategy, based on which the robot is controlled to perform the current task, includes: Collect the visual state, depth state, force state, pose state, gripper state, and task target state in the current task, generate the current task state fragment according to the order of the multimodal state sequence fields, and extract the current state constraints, current action requirements, and target feedback constraints based on the current task state fragment; The current state constraints, current action requirements, and target feedback constraints are converted into the current task query structure. The state memory block, action memory block, feedback memory block, and pruned locking coordinate set of each locking experience structure in the minimum experience generation subset are read. Candidate locking experience structures are obtained by filtering them in the order of consistency of task boundary fields, coverage of state fields, acceptability of target feedback fields, and satisfaction of modal credibility fields. Based on the state continuation edges, action continuation edges, and feedback generation edges between candidate locking experience structures, the candidate locking experience structures are spliced ​​and sorted. Candidate locking experience structures that have modal conflicts with the current task state segment, are unreachable actions, or have inconsistent feedback are deleted, and the current task reconstruction structure is generated. Read the action memory block, multimodal automorphic mapping pair and feedback generation edge in the current task reconstruction structure, map the current state constraints to the action field to be generated, and generate the action strategy. The action strategy includes approach direction, end pose, action speed, gripping force, gripper opening and closing amount, obstacle avoidance distance and abnormal retreat action. The motion strategy is sent to the robot motion controller, robotic arm controller, and gripper controller to control the robot to execute the current task, and the actual feedback state, motion execution state, and task result state after the current task is executed are written into a new multimodal state sequence.

Images