Robot control and benchmarking method, system, electronic device and storage medium

Abstract

The application provides a robot control and benchmark evaluation method and system, electronic equipment and storage medium, belonging to the field of robot learning and embodied intelligence. The method samples from historical RGB frames according to a fixed budget, respectively obtains two-dimensional semantic features and implicit three-dimensional spatial features, and aligns according to the same time sequence and local grid position; the three-dimensional evidence is selectively written into the semantic memory by using the semantic budget gate residual, and is acted on the action hidden state by the mask cross attention and scale-offset modulation, so as to accurately predict and execute the action block. The application also constructs a non-Markov evaluation benchmark containing twelve tasks including four types of tasks, i.e. occlusion recovery, single object three-dimensional position, multi-object absolute layout and multi-object relative position, and evaluates the ability of keeping and using the historical three-dimensional state by a strict task success rate evaluation strategy. The application solves the problems of insufficient memory of historical three-dimensional spatial information and lack of a special benchmark for evaluating the three-dimensional memory ability of a model.

Description

Technical Field

[0001] This invention relates to the field of robot learning and embodied intelligence, and more particularly to a robot control and benchmark evaluation method, system, electronic device and storage medium. Background Technology

[0002] When robots perform long-term manipulation in open environments, current observations often fail to fully reflect the task status due to object occlusion, scene changes, or targets being removed from the field of view. Correct actions rely on the position, posture, and spatial relationships of historical objects. Existing vision-language-action strategies primarily predict based on the current image. While some memory enhancement methods incorporate historical images or 2D visual features, they focus more on preserving appearance, object identity, or task progress, struggling to stably maintain depth, height, orientation, support relationships, and the 3D layout of multiple objects. Using explicit 3D representations such as point cloud maps and scene graphs increases sensor, mapping, and computational overhead. Furthermore, existing robot manipulation benchmarks focus on skill learning, transfer, and long-term execution, lacking specific evaluation of historical 3D spatial memory capabilities, making it difficult to determine whether the strategy truly preserves and utilizes occluded or changed 3D scene states. Therefore, how to maintain effective 3D spatial information from historical RGB observations under a fixed memory budget and systematically evaluate this capability remains a problem to be solved in the field of robot manipulation. Summary of the Invention

[0003] To address the aforementioned shortcomings in existing technologies, this invention provides a robot manipulation and benchmark evaluation method, system, electronic device, and storage medium. This invention does not require depth sensor output, point clouds, or explicit 3D maps as necessary inputs to the motion model; it uses only 2D images to obtain implicit 3D memory of modulated robot movements, thus improving the model's memory capacity. Furthermore, the benchmark evaluation method provided by this invention is designed with four categories based on the characteristics of 3D memory scenes, enabling separate evaluation of the strategy's ability to retain and utilize different types of historical 3D states. This solves the problems of insufficient memory of historical 3D spatial information and the lack of a dedicated benchmark for evaluating the model's 3D memory capacity.

[0004] To achieve the above objectives, the technical solution adopted by this invention is: a robot control and benchmark evaluation method, comprising the following steps: S1. Prepare current input and history cache: Obtain the robot's current observations and historical observations; whereby historical observations include RGB image sequences acquired in chronological order; S2, Fixed Budget Time Sampling: Sampling with a fixed memory token budget. The following steps are performed: RGB image sequences are sampled according to a fixed budget; the sampling results are arranged in temporal and local grid order; and a validity mask is used. Exclude the padding positions on the right; where each sampling frame retains two-dimensional local tokens; S3, parallel extraction of semantic memory and three-dimensional memory: based on the results of fixed budget time sampling, use two-dimensional visual encoder and memory encoder to extract semantic memory from the sampling frame ; at the same time, use the frozen space-depth encoder to extract dense implicit three-dimensional features from the same sampling frame, and get three-dimensional memory corresponding to each semantic token through local average pooling ; S4, token-by-token alignment and gated residual fusion: concatenate semantic memory and three-dimensional memory in the time dimension and local spatial position, and fuse three-dimensional memory into semantic memory selectively through semantic budget gated residual, obtaining fused memory , and the number of semantic memory tokens and feature dimensions remain unchanged before and after fusion; S5, memory-conditioned action modulation: make the action hidden state retrieve the fused memory through masked cross-attention , project the retrieval result into scale parameters and offset parameters and modulate the action hidden state, and then predict the action block from the flow matching action head; S6, closed-loop update: execute at least part of the action in the action block, add the new observation obtained after execution to the history buffer, and enter the next control cycle; S7, benchmark construction and evaluation: construct an evaluation task with a historical three-dimensional state variable, make the to-be-tested strategy executing S1 to S6 run in a closed loop in the evaluation task, and evaluate the ability of the to-be-tested strategy to maintain and utilize three-dimensional memory based on the historical three-dimensional state variable and the current observation.

[0005] Further, the S3 comprises: based on the results of fixed budget time sampling, use the frozen two-dimensional visual encoder to output local semantic features ; concatenate each local semantic feature in time sequence, and superimpose frame order and local grid position encoding ; based on the superimposed results, use the memory encoder to generate semantic memory ; at the same time, use the frozen space-depth encoder to extract implicit three-dimensional features from the same sampling frame and output a dense feature grid; divide the dense feature grid into non-overlapping regions corresponding to each of the two-dimensional local tokens , and perform average pooling processing on each non-overlapping region to obtain three-dimensional memory corresponding to each semantic token .

[0006] Further, the S4 comprises: Building semantic memory in the order of historical observation And three-dimensional memory Correspondence in the time dimension, and pool three-dimensional memory Tokens into three-dimensional memory tokens corresponding to local areas of semantic memory To build local spatial position correspondence; According to the correspondence built, write three-dimensional memory tokens to the corresponding semantic memory Token position; Based on the writing result, generate three-dimensional residual and gate value according to semantic memory And three-dimensional memory ; Act on three-dimensional residual with gate value, validity mask And residual scaling coefficient, and fuse the action result with the corresponding semantic feature token to obtain fused memory , and the number of tokens and feature dimension remain unchanged before and after fusion, forming a three-dimensional memory enhanced historical memory.

[0007] Further, the three-dimensional memory The objective function when selectively writing into semantic memory The expression is as follows:

[0008]

[0009] ,

[0010]

[0011]

[0012]

[0013]

[0014] Wherein, Indicates the objective function, Indicates the flow matching action loss, Indicates the auxiliary loss, Indicates the real action block in the training sample, Indicates the flow matching time variable Corresponding noisy action state, The value range of is 0 to 1, Indicates the standard Gaussian distribution Gaussian noise sampled at time t, represents the conditional context composed of the current RGB image observation, language instruction, robot state and fused memory ; represents the conditional velocity field predicted by the flow matching action head based on the noisy action state and the conditional context, represents the target velocity field, E represents the expectation of the training sample, the flow matching time variable τ and the Gaussian noise ε ; represents the weight coefficient of the residual proportion loss, represents the residual proportion loss, represents the gating interval loss, represents the weight coefficient of the semantic consistency loss, represents the semantic consistency loss, represents the semantic cosine similarity threshold, represents the cosine similarity of the mean direction of the semantic memory before and after fusion, represents the stop gradient applied to the mean direction of the semantic memory before fusion, represents the weight of the lower limit of the gating penalty, represents the lower limit of the gate value, represents the average gate value of the valid token, represents the weight of the upper limit of the gating penalty, represents the upper limit of the gate value, represents the number of valid memory positions, represents the index of the valid memory position, represents the validity mask of the th valid memory position, represents the relative write ratio of the th valid memory position, represents the residual relative proportion threshold, represents the mask mean direction of the fused memory, represents the stop gradient operator.

[0015] Further, the S5 comprises: taking the action hidden state as a query, taking the fused memory as a key and a value, performing mask cross-attention retrieval of the fused memory ; projecting the retrieval result into a scale parameter and an offset parameter; based on the projection result, modulating the action hidden state, and predicting the action block by the flow matching action head.

[0016] Further, the S6 comprises: building an evaluation task provided with a historical three-dimensional state variable; In the prelude stage of the task, the to-be-tested strategy performing S1-S6 obtains historical three-dimensional state variables from historical observations, wherein the historical three-dimensional state variables are used to affect the correct action of the robot; After the setting and recording of the historical three-dimensional state variables are completed, the to-be-tested strategy in the decision stage makes an action decision on the evaluation task based on the current observation, hides the historical three-dimensional state variables or makes the current observation ambiguous, so that the current observation is not sufficient to uniquely determine the correct action; The to-be-tested strategy is controlled to perform closed-loop in the task environment; The final physical state in the task environment is read when the task is terminated; The task success result is output according to the machine executable success predicate of the final physical state.

[0017] The present application provides a robot control and benchmark evaluation system, comprising: An observation acquisition module is configured to acquire current observation and historical observation of the robot; A feature extraction module is configured to extract semantic memory from the historical observation And three-dimensional memory representing the three-dimensional space state of the scene ; An association and memory enhancement module is configured to associate the semantic memory With the three-dimensional memory In the time dimension and the local spatial position, and form a fusion memory of three-dimensional memory enhancement ; An action generation module is configured to generate robot action according to the current observation and the historical memory; A history update module is configured to add new observation to the historical observation after executing at least part of the robot action; A benchmark construction and evaluation module is configured to obtain historical three-dimensional state variables from the historical observation, and construct and evaluate the benchmark of the three-dimensional memory ability of the robot based on the historical three-dimensional state variables and the current observation.

[0018] The present application provides an electronic device comprising a processor and a memory storing a computer program, wherein the computer program is executed by the processor to make the electronic device perform the robot control and benchmark evaluation method.

[0019] The present application provides a computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the robot control and benchmark evaluation method.

[0020] The present application has the following beneficial effects: Compared with the prior art, the present application has at least the following beneficial effects: Extracting implicit three-dimensional memories from historical sequences of RGB images, without additional depth sensors, explicit point cloud reconstruction, scene graphs, or external mapping pipelines.

[0021] Semantic memories and three-dimensional memories are aligned one-to-one in the temporal dimension and local spatial locations, enabling historical height, depth, orientation, support, and relative layout information to be localized to fixed memory tokens.

[0022] Three-dimensional memory write intensity and range are limited by gating and semantic budgeting, avoiding overwhelming pre-trained semantic representations with geometric features, and preserving near-original policies when three-dimensional information is not needed.

[0023] The number of tokens before and after fusion is fixed, and the action-side memory cross-attention complexity is , and the interface size does not grow with task length.

[0024] Memory conditioning scale-offset modulation provides bounded modification over the current observation and original action head, facilitating modular integration into existing vision-language-action policies.

[0025] 3D-MAM separates occlusion recovery, single-object location, absolute layout, and relative layout across four categories of twelve non-Markovian tasks, revealing spatial state ambiguities that two-dimensional appearance memories cannot discover.

[0026] Uniform seed-aligned training / evaluation splits and strict success criteria enable three-dimensional memory capabilities of different policies to be compared under the same object assignment, target location, and task randomization. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 A flowchart of the method of the present application.

[0028] Figure 2 A scene overview diagram of the four categories of twelve three-dimensional spatial memory tasks of the 3D-MAM of the present application.

[0029] Figure 3 A schematic diagram of the system structure of the present application. DETAILED DESCRIPTION

[0030] The specific embodiments of the present application are described below to enable those skilled in the art to understand the present application, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application as defined in the appended claims, and all inventions utilizing the concept of the present application are within the scope of protection.

[0031] Example 1 As Figure 1 ​As shown, the present application provides a robot control and benchmark evaluation method, the control method comprising obtaining the current observation of the robot and the historical observation obtained before the current time; extracting semantic features and spatial features representing the three-dimensional space state of the scene from the historical observation; establishing the association of semantic memory and spatial memory in the time dimension and the local spatial position, and forming a three-dimensional memory enhanced fusion memory based on the association ; generating robot actions according to the current observation and the fusion memory ; executing at least part of the robot actions, and adding the new observation obtained after execution to the historical observation. Specifically, the implementation method is as follows: S1, preparing current input and historical cache: obtaining the current observation of the robot and the historical observation; wherein the historical observation includes a sequence of RGB images obtained in time sequence; S2, fixed budget time sampling: under the fixed memory token budget , sampling the RGB image sequence under the fixed budget, arranging the sampling results in time and local grid sequence, and using the validity mask to exclude the right side filling position; wherein each sampling frame retains two-dimensional local tokens; S3, parallel extraction of semantic memory and three-dimensional memory: based on the fixed budget time sampling result, using a two-dimensional visual encoder and a memory encoder to extract semantic memory from the sampling frame; at the same time, using a frozen space-depth encoder to extract dense implicit three-dimensional features from the same sampling frame, and performing local average pooling to obtain three-dimensional memory corresponding to each semantic token , the implementation method is as follows: Based on the fixed budget time sampling result, use the frozen two-dimensional visual encoder to output local semantic features ; Splice each local semantic feature in time sequence, and superimpose frame order and local grid position coding ; Based on the superimposed result, use the memory encoder to generate semantic memory ; At the same time, use the frozen space-depth encoder to extract implicit three-dimensional features from the same sampling frame and output a dense feature grid; Divide the dense feature grid into non-overlapping regions corresponding to two-dimensional local tokens one by one, and perform average pooling processing on each non-overlapping region to obtain three-dimensional memory corresponding to each semantic token .

[0032] In this embodiment, semantic memory is extracted from historical observations by a two-dimensional visual encoder and a memory encoder; spatial features (i.e., three-dimensional features) are extracted from RGB images of historical observations by a spatial-depth encoder, and the extraction of spatial features does not require depth sensor outputs, explicit point clouds, or explicit three-dimensional maps as necessary inputs.

[0033] S4, token-by-token alignment and gated residual fusion: semantic memory and three-dimensional memory are spliced in the temporal dimension and local spatial position, and three-dimensional memory is gated by semantic budget residual. , resulting in fused memory , and the number of tokens and feature dimensions of semantic memory before and after fusion remain unchanged, and the implementation method is as follows: According to the order of historical observations, semantic memory and three-dimensional memory are established in the corresponding relationship in the temporal dimension, and three-dimensional memory is pooled into three-dimensional memory tokens corresponding to the local area of semantic memory to establish a local spatial position correspondence; According to the established correspondence, write three-dimensional memory tokens to the corresponding semantic memory token position; Based on the writing result, generate three-dimensional residual and gate value according to semantic memory and three-dimensional memory ; The gate value, validity mask and residual scaling coefficient act on the three-dimensional residual, and the result of the action is fused with the corresponding semantic feature token to obtain fused memory , and the number of tokens and feature dimensions before and after fusion remain unchanged, forming a three-dimensional memory-enhanced historical memory.

[0034] In this embodiment, the temporal correspondence is established according to the order of historical observations, and the spatial features are divided or pooled into three-dimensional memory tokens corresponding to the local area of semantic features to establish a local spatial position correspondence.

[0035] Write three-dimensional memory tokens to the corresponding semantic memory token position; generate three-dimensional residual and gate value according to semantic memory and three-dimensional memory, and act the gate value, validity mask and residual scaling coefficient on the three-dimensional residual, and fuse the obtained result with the corresponding semantic feature token.

[0036] In this embodiment, the present invention provides a training method for an implicit three-dimensional memory-enhanced action model, comprising: acquiring training samples containing the robot's current observations, historical observations, and target actions; inputting the current observations and historical observations into the action model to be trained, wherein the action model to be trained extracts semantic features (i.e., semantic memory) and spatial features, forms a three-dimensional memory-enhanced historical memory, and generates a predicted action; calculating an action loss based on the predicted action and the target action, and calculating at least one auxiliary loss based on the spatial information of the historical memory written into the state; and updating at least some parameters of the action model to be trained based on the action loss and the auxiliary loss.

[0037] The auxiliary loss includes at least one of the following: residual ratio loss when the proportion of spatial information written to the original semantic features exceeds a preset threshold; threshold interval loss when the average threshold of the effective position is lower than the lower threshold or higher than the upper threshold; and semantic consistency loss when the similarity of semantic features before and after fusion is lower than a preset threshold.

[0038] In the training method, the parameters of the two-dimensional visual encoder used to extract semantic features and the spatial-depth encoder used to extract spatial features are kept frozen during training; the gating network is initialized to make the spatial information writing coefficients in the early stage of training at a preset low value; at least one of the memory encoder, gating fusion module, action network and adaptation module is updated based on action loss and auxiliary loss.

[0039] S5. Conditioned Action Modulation: This technique involves hiding actions and then fusing them into memory through masked cross-attention retrieval. The retrieval results are projected onto scale and offset parameters and the action hiding state is modulated. Then, the action block is predicted by the stream matching action head. The implementation method is as follows: Using the hidden state of the action as a query, the fused memory will be used. As keys and values, perform masked cross-attention retrieval and fusion memory. ; The search results are projected onto scale and offset parameters; Based on the projection results, the action hidden state is modulated, and the action block is predicted by the stream matching action head.

[0040] In this embodiment, the action hidden state of the action network is used as a query, and the fusion memory is used as a key and value to perform cross attention; at least one of the scale parameter, offset parameter and residual modulation amount is generated based on the cross attention result to modulate the action hidden state and generate robot actions.

[0041] Robot actions include a sequence of actions consisting of multiple consecutive actions; after executing at least part of the actions in the action sequence, the current observation is reacquired and a new action sequence is generated; the current observation includes at least one of a third-person image, a wrist image, and the robot's body state.

[0042] S6, Closed-loop update: Execute at least some of the actions in the action block, add the new observations obtained after execution to the history cache, and enter the next control cycle; S7. Benchmark Construction and Evaluation: Construct an evaluation task with historical three-dimensional state variables, enabling the policy under test (S1 to S6) to run in a closed loop within the evaluation task. Based on the historical three-dimensional state variables and current observations, evaluate the policy under test's ability to retain and utilize three-dimensional memory. The implementation method is as follows: In the preliminary stage of the task, the test policy that executes S1 to S6 obtains historical three-dimensional state variables from historical observations, where the historical three-dimensional state variables are used to influence the correct actions of the robot. Hiding historical three-dimensional state variables or making current observations ambiguous in subsequent decision-making stages, so that current observations are insufficient to uniquely determine the correct action; Control the closed-loop execution of the strategy under test within the task environment; Read the final physical state of the task environment when the task terminates; Based on the final physical state, the machine can execute a successful predicate to output the task success result.

[0043] The present invention will be further described below.

[0044] (a) Fixed budget sampling and semantic memory In this embodiment, to prevent the memory interface size from increasing with task duration, the number of sampling frames is... satisfy The set of sampled frame indices is denoted as:

[0045] in, Indicates a fixed memory token budget. This represents the number of two-dimensional local tokens retained in each sampling frame. Represents the set of time indices for the sampled frames. Indicates the first in the sampled frame index set The time index of each frame.

[0046] For the first Sample frames The two-dimensional visual encoder outputs local semantic features. Local semantic features of each frame Assembled in chronological order And superimpose frame order and local grid position encoding , by memory encoder Generate semantic memory :

[0047] in, This represents the sequence obtained by splicing the local semantic features of each sampled frame in chronological order.

[0048] This embodiment sets a fixed memory budget. =512, reserved per frame =16 fours With 4 local units, a maximum of 32 historical frames can be uniformly sampled. The 2D vision branch encodes each sampled frame into 16 2048-dimensional local semantic tokens, with a positional encoding dimension of 768; if there are fewer than 512 valid positions, padding is added on the right and a Boolean validity mask is generated. The memory encoder maps semantic tokens to... semantic memory .

[0049] (II) Retrieval and token-by-token alignment of implicit three-dimensional memory Spatial-depth encoders output on the same sampled frame Dense feature mesh Divide the dense feature mesh into sections with... Each two-dimensional local token corresponds to a non-overlapping region. Perform average pooling for each region:

[0050] in, Indicates the first The sampling frame number Three-dimensional memory tokens corresponding to each local unit. Indicates the location of a dense mesh element falling into a local cell. Indicates the first Location in the dense feature grid of each sampling frame The eigenvector at that location.

[0051] Therefore, semantic memory Three-dimensional memory With validity mask They correspond one-to-one in both the same time location and the local spatial location.

[0052] In this embodiment, the frozen space-depth encoder is at 224. Processing the same historical frames at 224 resolution, outputting 16 frames per frame from a specified intermediate layer. 16 A total of 256 768-dimensional dense space tokens; each 4 4. Dense regions are averaged into a single local unit, resulting in 16 3D spatial tokens per frame. After concatenating up to 32 frames, 512 tokens are obtained. 768's three-dimensional memory Since the 3D branch extracts implicit 3D features only from RGB images, this embodiment does not require depth sensor output, explicit point cloud, or explicit 3D map as necessary inputs.

[0053] (III) Problems to be solved and gating residual integration scheme While the above alignment ensures a one-to-one correspondence between geometric evidence and semantic tokens, incorporating 3D evidence into the action model under a fixed budget still faces two key issues: First, if 3D memory is directly input into the action network as a second set of independent historical tokens alongside semantic memory, it will increase the context length and attention computation, and cause the interface size to grow with the task duration; Second, if geometric features are superimposed on semantic memory without constraints, it may overwhelm the pre-trained semantic representation, and in scenarios where 3D information is not required, it may actually damage the original strategy.

[0054] To address the aforementioned issues, this embodiment proposes a semantic budget-gated residual fusion scheme. This scheme writes 3D evidence into the corresponding positions of existing semantic tokens using controlled residuals, rather than appending new tokens. Specifically, it includes the following steps: First, construct the fused input and generate the 3D residuals and thresholds. For the... For each valid memory location, the corresponding semantic token is concatenated with the 3D memory token and then normalized:

[0055] in, Indicates the first The fused input is obtained by concatenating and normalizing a semantic token representing an effective memory location with a three-dimensional memory token. Indicates the first A semantic memory token for each valid memory location. Indicates the first A three-dimensional memory token with one effective memory location. Representation layer normalization operator.

[0056] The residual branch and the gated branch output the three-dimensional residual and the gate value, respectively:

[0057]

[0058] in, This represents the three-dimensional residual output by the residual branch. Indicates the first Fusion input of one effective memory location, Indicates residual branch, This indicates the gate value of the gated branch output. This represents the sigmoid activation function. Indicates a gated branch.

[0059] Second, semantic memory is selectively written using gated residuals. Fusion memory is defined as:

[0060] in, Indicates the first A fusion memory token for each valid memory location. Indicates the first A semantic memory token for each valid memory location. Indicates the first A validity mask for each valid memory location. This represents the three-dimensional residual output by the residual branch. Represents the residual scaling factor, validity mask The padding position is set to 0. This operation does not append a second set of memory tokens, and the token count before and after fusion remains unchanged from the context budget; this is because the code does not handle the 3D residual. Perform norm clipping, residual scaling factor It only serves a scaling function and does not constitute a strict upper limit on the amount of data that can be written.

[0061] In this embodiment, the fusion module concatenates the 1024-dimensional semantic token and the 768-dimensional three-dimensional memory token at each location, and after layer normalization, residual multilayer perceptron and gated multilayer perceptron, outputs a 1024-dimensional residual and a scalar gate value; the initial gate value is set to 0.1, and the residual scaling factor is... Set to 0.25, the two together constitute the nominal initial write coefficient of 0.025, which makes the 3D write intensity low in the early stage of training and smoothly start from a state close to the original strategy.

[0062] (iv) Semantic Budget Soft Constraint To limit 3D writes to the semantic budget during training, this embodiment further introduces three soft constraints. The effective memory position count is the larger of the sum of the effectiveness mask and 1:

[0063] in, Indicates the number of effective memory locations. An index representing the effective memory location. Indicates the first A validity mask for each valid memory location.

[0064] (1) Residual ratio constraint. The actual write volume and the relative write ratio are respectively:

[0065]

[0066] in, Indicates the first A semantic memory token for each valid memory location. Indicates the first A validity mask for each valid memory location. This represents the three-dimensional residual output by the residual branch. This indicates the gate value of the gated branch output. Indicates the first The relative write ratio of each effective memory location. Indicates the first The actual amount of data written to each effective memory location. This represents a small positive number that prevents the denominator from being zero.

[0067] Residual proportion loss penalty exceeding threshold Relative write:

[0068] in, Indicates the percentage loss of the residual. This represents the threshold for the relative proportion of the residual.

[0069] (2) Gating interval constraint. Let the average threshold value of valid tokens be:

[0070] in, This represents the average threshold value of valid tokens.

[0071] Gated interval loss constrains the average threshold to Inside:

[0072] in, This indicates the weight of the lower bound penalty for gating. Indicates the lower limit of the threshold. This indicates the weight of the gating limit penalty. This indicates the upper limit of the threshold.

[0073] (3) Semantic consistency constraint. For any memory sequence Define the mask mean:

[0074] in, This represents the mask mean of the memory sequence at the valid positions. Represents the first memory sequence A token for a valid memory location.

[0075] Semantic consistency loss constrains the semantic direction to avoid excessive drift before and after fusion:

[0076] in, This represents the semantic consistency loss. This represents the semantic cosine similarity threshold. This indicates the direction of the mask mean for fused memories. This indicates the direction of the mask mean for semantic memory before fusion. This indicates that the gradient operator is stopped. The auxiliary loss and the overall training objective are:

[0077]

[0078] ,

[0079]

[0080] in, Indicates the loss of the stream matching action. The weighting coefficient representing the residual proportional loss. Indicates auxiliary loss, This represents the actual action blocks in the training samples. Indicates the time variable for stream matching The corresponding noisy action state, The value range is from 0 to 1. Indicates the standard Gaussian distribution Gaussian noise in the mid-sample, This represents the current RGB image observation, language commands, robot state, and fused memory. The conditional context constituted This represents the conditional velocity field predicted by the stream matching action head based on the noisy action state and conditional context.

[0081] In this embodiment, the lower limit of the threshold mean =0.03, upper limit of the mean threshold =0.35, residual relative proportion threshold =0.35, semantic cosine similarity threshold =0.98; the weights for the lower gating bound, upper gating bound, residual ratio, and semantic consistency are 20, 0.05, 0.05, and 0.005, respectively. No penalty is applied within the threshold; a correction gradient is only provided when the 3D branch is completely inactive, there is no selective writing, the local residual is too large, or there is global semantic drift, thereby limiting the 3D writing to within the semantic budget without changing the token size.

[0082] (v) Modulation of Memory Conditioned Actions In the action network l Within each action block, the action is in a hidden state. As a query, fused memories are used as keys and values, and retrieved through masked cross-attention:

[0083] in, Indicates the first l The result obtained by masking cross-attention retrieval of the hidden state of actions in each action block. This represents the masked cross-attention operator. Indicates the fusion of memories, Indicates a validity mask.

[0084] The padding position is masked before the softmax. Linear mapping as scale parameter With offset parameter Modulate the action state as follows:

[0085] in, This represents the motion concealment state after scale-offset modulation. This represents the root mean square normalization operator.

[0086] The modulation module is located before the action branch feedforward layer, and the original current observation, language conditions, robot state, and flow matching action head remain unchanged. When the scale parameter and offset parameter are close to zero, the modulation module degenerates into the original normalized action representation, enabling the newly added memory branch to be smoothly trained from a state close to the original policy.

[0087] In this embodiment, the motion expert has the following number of hidden motion states. =20, dimension 1024; the hidden state of the action is used as the query, the fused memory is used as the key and value to perform mask cross attention, the stream matching head predicts 20 steps of action, and replans after the first 16 steps are executed.

[0088] (VI) Summary In summary, this embodiment first extracts semantic memory and implicit 3D memory in parallel from historical RGB observations under a fixed budget, aligning them token by token. Then, it selectively writes 3D evidence into semantic tokens using gated residuals, and restricts the writing intensity and range through soft constraints. Finally, it applies memory-conditional scale-offset modulation to the action hiding state. Since both the fused memory and semantic memory are 512... 1024, action-side memory cross-attention does not increase the number of tokens or the size of the attention graph, and its complexity remains constant. And fixed This ensures that the interface size does not grow with task duration; the additional computation is focused on freezing the 3D encoder, local pooling, and token-by-token fusion, making it suitable for modular integration into existing pre-trained vision-language-action policies. Indicates the number of hidden states for an action. Indicates a fixed memory token budget. Representing the dimensions of memory features, The order of computational complexity.

[0089] Example 2 The 3D-MAM benchmark is constructed as follows: Figure 2 As shown, this embodiment provides a benchmark construction and evaluation method for a robot's three-dimensional spatial memory capability. Its core idea is to set historical three-dimensional state variables that affect subsequent correct actions for each task round. In the pre-task stage, the strategy to be tested can obtain these variables from the observation. Then, in the subsequent decision-making stage, the variables are made unobservable by means of occlusion, movement, exchange, disassembly, or temporary cues. This makes the current observation insufficient to uniquely determine the correct action, and only strategies that truly maintain and utilize historical three-dimensional states can continue to succeed.

[0090] This embodiment constructs twelve desktop manipulation tasks on the ManiSkill 3.0.0b21 and SAPIEN 3.0.3 physics simulation platforms, employing a Franka Emika Panda seven-DOF robotic arm and parallel grippers. The tasks are divided into four categories, with three tasks in each category, based on the type of historical 3D state. Each task displays 3D information that determines the correct subsequent actions in an early stage, and this information is made unavailable for direct observation in later stages through occlusion, movement, exchange, disassembly, or temporary prompts. Table 1 lists the twelve tasks and their 3D memory requirements, where OV-3D represents occluded viewpoint 3D memory, SO-3D represents single-object 3D position memory, MA-3D represents multi-object absolute layout memory, and MR-3D represents multi-object relative position memory.

[0091] Expert trajectories are generated from task-specific key poses and sub-object sequences. The motion planner extends the discrete phase into dense joint trajectories that satisfy collision constraints. The original trajectory records at least one of the following: joint position and velocity, end-effector pose, gripper state, forward RGB, and wrist RGB. The construction methods for the four types of tasks are described in detail below.

[0092] (a) Occlusion viewpoint 3D memory task The historical 3D state variables for this type of task are the historical bindings between object identities and slots. During task generation, multiple objects with identities are randomly assigned slots, and the identity-slot bindings are recorded. Then, all slots are covered with opaque occluders of uniform appearance, ensuring that the occluders are arranged in the same or similar patterns across different rounds at the time of decision-making. Task instructions require the robot to uncover the corresponding occluders in identity order and restore the occlusion state after the operation.

[0093] Since the current observation no longer reveals the identity, different historical bindings correspond to different correct uncovering sequences, and the strategy must rely on the 3D binding information observed before occlusion. The phase state machine records the identity display, complete occlusion, sequential uncovering, occlusion reset, and termination check in sequence; the success condition jointly verifies the target identity sequence, the opened position, that non-target occluders were not opened incorrectly, the final pose of the occluders, and the completion of all necessary stages.

[0094] (ii) Single-object three-dimensional position memory task The historical 3D state variable for this type of task is the initial 3D position of a single object. During task generation, the same object is randomly placed in one of several candidate initial regions, and its identity, initial 3D position, and allowed pose are recorded. The robot then moves the object to a common temporary storage region. At different rounds of decision-making, the object appears to be in the same temporary storage position, but the correct return region is determined by the invisible historical initial position.

[0095] The phase state machine includes initial position registration, temporary transfer, stability confirmation, historical position recovery, and termination check; the automatic success condition requires that the correctly identified object falls into the historical target area, the position and attitude error is within the tolerance, the temporary storage phase has been completed, and the object has not fallen into other candidate areas.

[0096] (III) Multi-object absolute layout memory task The historical three-dimensional state variables for this type of task are the absolute historical positions of multiple objects. During task generation, different initial absolute positions are assigned to multiple identified objects, recording the identity, position, and pose of each object. Subsequently, the objects are moved sequentially into a shared disk or a temporary storage area adjacent to the disk, ensuring that different rounds have the same or similar object sets and temporary storage layouts at the time of decision. The strategy requires returning each object to its respective historical absolute position.

[0097] The phase state machine records disk placement, disk-side placement, temporary storage confirmation, object-by-object recovery, and global check; the automatic success condition verifies the identity, historical target position, and attitude of each object, confirms that all specified temporary storage phases are completed, and eliminates cross-matching between object identity and target position.

[0098] (iv) Multi-object relative position memory task The historical three-dimensional state variables for this type of task represent the relative historical structure between multiple objects, including identity-role binding, support relationships, left-right order, front-back relationships, spacing, pose, and height hierarchy. During task generation, multiple objects construct a bridge-like or hierarchical structure and record the aforementioned relationships; subsequently, the structure is completely disassembled, and the objects are placed in a shared temporary storage area. Different rounds at the decision point may have the same scattered observations, but the correct recovery action is determined by the historical structural relationships.

[0099] The phase state machine includes structure registration, complete disassembly, temporary storage confirmation, structure restoration, and stability check. Automatic success conditions require that the supported object is correctly identified, the supported object is located on the corresponding support surface, the left-right, front-back, and up-down relationships are correct, the spacing and posture are within tolerance, the structure is stable, and the disassembly phase has been confirmed to have occurred completely.

[0100] (v) Task set, division and evaluation indicators In one verification embodiment, each of the twelve tasks across the four categories used 70 successful expert trajectories corresponding to random seeds 0 to 69 as training examples, totaling 840 trajectories across the twelve tasks; the average sample trajectory length was approximately 639 steps. Each task used seeds 70 to 99 for 30 independent closed-loop evaluation rounds, totaling 360 evaluation rounds per round across the twelve tasks; training example and evaluation rounds were counted separately. The number of tasks and samples mentioned above are experimental configurations and do not constitute essential technical features of the benchmark generation mechanism.

[0101] The evaluation metric is the strict synthesis power:

[0102] in, Indicates the exact synthesis power. Indicates the number of successful rounds. Indicates the total number of rounds.

[0103] Success is only counted when the native info["success"] flag is true and the complete task objective is achieved; it is not replaced by manually relaxed distance thresholds or partial stage completion. The comparison method uses the same task partitioning, object randomization, action heads, and evaluation seeds, thus enabling the 3D memory capabilities of different strategies to be compared under the same object allocation, target location, and task randomization.

[0104] (vi) Machine-executable success predicate In this embodiment, the final physical state in the task environment is read upon task termination, and a machine-executable success predicate outputs the task success result. In the simulation environment, the reference-side adapter reads the object identifier, 3D position, quaternion or rotation matrix, contact or support state, and stage flags. Position conditions compare the Euclidean distance between the object's position and the target position; attitude conditions calculate the angular distance between the current rotation and the target rotation; and identity and relationship conditions are executed by discrete predicates in the task configuration. The environment updates the stage flags at each control step, and at the termination step, the applicable predicates are combined, and a success result and failure reason are output. Success predicates may include at least one of the following: object pose conditions, relative relationship conditions between objects, support relationship conditions, operation sequence conditions, target area occupancy conditions, and robot termination conditions, and logical combinations thereof. A success result is output when the logical combination is satisfied.

[0105] In this embodiment, historical three-dimensional state variables that affect the robot's subsequent correct actions are set; in the pre-task stage, the strategy under test is enabled to obtain historical three-dimensional state variables from observations, and in the subsequent decision-making stage, the historical three-dimensional state variables are hidden or the current observation is made ambiguous so that the current observation is insufficient to uniquely determine the correct action; the strategy under test is controlled to execute in a closed loop in the task environment; when the task terminates, the final physical state in the task environment is read; and the task success result is output according to the machine-executable success predicate acting on the final physical state.

[0106] Historical 3D state variables include at least one of the following: object identity, object initial position, object absolute layout, object relative position, support relationship, object orientation, and height level; the baseline task includes at least one of the following: occlusion viewpoint 3D memory, single object 3D position memory, multi-object absolute layout memory, and multi-object relative position memory.

[0107] The strategy to be tested is a pre-trained strategy or a strategy trained using the demonstration trajectory corresponding to the benchmark task; the evaluation process does not require retraining the strategy to be tested as a necessary step, and the task success rate is calculated based on the task success results of multiple closed-loop rounds.

[0108] The machine can execute successful predicates including at least one of the following, and logical combinations thereof: object pose condition, relative relationship condition between objects, support relationship condition, operation sequence condition, target area occupancy condition, and robot termination condition, and outputs a success result when the logical combination is satisfied.

[0109] Table 1. 3D-MAM Twelve Tasks and 3D Memory Requirements

[0110] The above embodiments are merely preferred embodiments of the present invention. Those skilled in the art can make adjustments or equivalent substitutions to the feature dimensions, sampling strategies, gating networks, loss thresholds, action temporal domains, benchmark task objects, and physical platforms without departing from the spirit and scope of the present invention and the claims.

[0111] Example 3 like Figure 3 As shown, the present invention provides a robot control and benchmark evaluation system for executing the robot control and benchmark evaluation method described in Embodiment 1, comprising: The observation acquisition module is used to acquire the robot's current and historical observations; The feature extraction module is used to extract semantic memory from historical observations. and three-dimensional memory representing the three-dimensional spatial state of a scene ; The association and memory enhancement module is used to build semantic memory. With three-dimensional memory The connections between time and local spatial location form a fusion memory enhanced by three-dimensional memory. ; The motion generation module is used to generate robot motions based on current observations and historical memories. The history update module is used to add new observations to the history after at least part of the robot's actions are performed; The benchmark construction and evaluation module is used to obtain historical three-dimensional state variables from historical observations, and to construct and evaluate the benchmark of the robot's three-dimensional memory ability based on the historical three-dimensional state variables and the current observations.

[0112] In this embodiment, the association and memory enhancement module writes spatial features into the corresponding token positions of existing semantic features, so that the number of memory tokens and feature dimensions remain unchanged before and after fusion; the action generation module does not receive independent three-dimensional historical token sequences that are parallel to semantic features.

[0113] The benchmark construction and evaluation module is configured to set historical three-dimensional state variables, hide the historical three-dimensional state variables or make the current observation ambiguous in the subsequent decision-making stage, control the closed-loop execution of the strategy to be tested, read the final physical state, and output the task success result according to the machine-executable success predicate.

[0114] In this embodiment, the robot control and benchmark evaluation system, in order to achieve the principles and beneficial effects of Embodiment 1, includes hardware structures and / or software modules corresponding to the execution of various functions. Those skilled in the art should readily recognize that, in conjunction with the illustrative units and algorithm steps described in the embodiments disclosed herein, this invention can be implemented in hardware and / or a combination of hardware and computer software. Whether a function is executed by hardware or computer software depends on the specific application and design constraints of the technical solution. Different methods can be used to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0115] Example 4 The present invention provides an electronic device, including a processor and a memory storing a computer program. When the computer program is executed by the processor, the electronic device performs the robot manipulation and benchmark evaluation method described in Embodiment 1.

[0116] In this embodiment, the electronic device may include: a processor, a memory, a bus, and a communication interface. The processor, the communication interface, and the memory are connected via the bus. The memory stores a computer program that can run on the processor. When the processor runs the computer program, it executes some or all of the steps of the robot manipulation and benchmark evaluation method provided in Embodiment 1 of this application.

[0117] Example 5 The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the robot manipulation and benchmark evaluation method described in Embodiment 1.

[0118] This embodiment provides a computer-readable storage medium storing a computer program that, when executed, implements some or all of the steps of the robot manipulation and benchmark evaluation method as described in Embodiment 1.

[0119] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable and reproducible read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer. The readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. The readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC), or they can exist as discrete components in the electronic device.

[0120] The present invention can be provided as a method, apparatus, or computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) including computer-usable program code. Described with reference to the flowcharts and / or block diagrams of methods, apparatuses, and computer program products according to embodiments of the present invention, it should be understood that each flowchart and / or block diagram, and combinations thereof, can be implemented by computer program instructions. These computer program instructions can be provided to a computer-readable storage medium that operates in a particular manner on a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing device, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means implemented in the flowchart. Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes in the flowchart. These computer program instructions may also be accreted to a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing one or more processes and / or boxes in the flowchart. Figure 1 The steps of the function specified in one or more boxes.

[0121] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A method for robot control and benchmark evaluation, characterized in that, Includes the following steps: S1. Prepare current input and history cache: Obtain the robot's current observations and historical observations; whereby historical observations include RGB image sequences acquired in chronological order; S2, Fixed Budget Time Sampling: Sampling with a fixed memory token budget. The following steps are performed: RGB image sequences are sampled according to a fixed budget; the sampling results are arranged in temporal and local grid order; and a validity mask is used. Exclude the padding positions on the right; where each sampling frame retains A two-dimensional local token; S3. Parallel Extraction of Semantic Memory and 3D Memory: Based on the sampling results at a fixed budget time, semantic memory is extracted from the sampled frames using a 2D visual encoder and a memory encoder. Simultaneously, a frozen spatial-depth encoder is used to extract dense implicit 3D features from the same sampled frames, and local average pooling is used to obtain 3D memories that correspond one-to-one with the semantic tokens. ; S4. Fusion of token-by-token alignment and gated residuals: integrating semantic memory and three-dimensional memory Semantic memory tokens are concatenated sequentially along the temporal and local spatial dimensions, and the three-dimensional memory is then gated using semantic budgeting residuals. Selective writing of semantic memory , to obtain fused memories Furthermore, the number of semantic memory tokens and the feature dimension remain unchanged before and after fusion; S5. Conditioned Action Modulation: This technique involves hiding actions and then fusing them into memory through masked cross-attention retrieval. The search results are projected into scale parameters and offset parameters and the action hiding state is modulated. Then, the action block is predicted by the flow matching action head. S6, Closed-loop update: Execute at least some of the actions in the action block, add the new observations obtained after execution to the history cache, and enter the next control cycle; S7. Benchmark Construction and Evaluation: Construct an evaluation task with historical three-dimensional state variables, so that the policy to be tested executing S1 to S6 runs in a closed loop in the evaluation task, and evaluate the policy to be tested's ability to retain and utilize three-dimensional memory based on historical three-dimensional state variables and current observations.

2. The robot control and benchmark evaluation method according to claim 1, characterized in that, S3 includes: Based on the sampling results at a fixed budget time, local semantic features are output using a frozen 2D visual encoder. ; Local semantic features The frames are stitched together in chronological order, and the frame order and local grid position codes are superimposed. ; Based on the superposition results, a semantic memory is generated using a memory encoder. ; Simultaneously, a frozen spatial-depth encoder is used to extract implicit 3D features from the same sampled frames and output a dense feature mesh; Divide the dense feature grid into... Each two-dimensional local token corresponds to a non-overlapping region, and average pooling is performed on each non-overlapping region to obtain a three-dimensional memory that corresponds to each semantic token. .

3. The robot control and benchmark evaluation method according to claim 1, characterized in that, S4 includes: Establish semantic memory according to the order of historical observation. and three-dimensional memory The correspondence in the time dimension, and the three-dimensional memory Pooling as a semantic memory Three-dimensional memory tokens corresponding to local regions are used to construct local spatial location correspondences; Based on the established correspondence, the three-dimensional memory token is written into the corresponding semantic memory. Token location; Based on the write results, and according to semantic memory and three-dimensional memory Generate three-dimensional residuals and thresholds; Threshold and validity mask The residual scaling factor is applied to the 3D residual, and the result is fused with the corresponding semantic feature token to obtain the fused memory. Furthermore, the number of tokens and feature dimensions remain unchanged before and after fusion, forming a three-dimensional memory-enhanced historical memory.

4. The robot control and benchmark evaluation method according to claim 3, characterized in that, Three-dimensional memory Selective writing of semantic memory The expression for the objective function at time is as follows: ， in, Describe the objective function. Indicates the loss of the stream matching action. Indicates auxiliary loss, This represents the actual action blocks in the training samples. Indicates the time variable for stream matching The corresponding noisy action state, The value range is from 0 to 1. Indicates the standard Gaussian distribution Gaussian noise in the mid-sample, This represents the current RGB image observation, language commands, robot state, and fused memory. The conditional context constituted This indicates that the flow matching action head predicts the conditional velocity field based on the noisy action state and conditional context. Let E represent the target velocity field, and let E represent the training samples, the flow matching time variable τ, and the Gaussian noise. Take the expected value. The weighting coefficient representing the residual proportional loss. Indicates the percentage loss of the residual. Indicates the loss within the gated region. The weight coefficients representing the semantic consistency loss. This represents the semantic consistency loss. This represents the semantic cosine similarity threshold. The cosine similarity in the direction of the semantic memory mean before and after fusion is represented. This indicates that a stopping gradient is applied to the mean direction of the semantic memory before fusion. This indicates the weight of the lower bound penalty for gating. Indicates the lower limit of the threshold. This represents the average threshold value of valid tokens. This indicates the weight of the gating limit penalty. Indicates the upper limit of the threshold. Indicates the number of effective memory locations. An index representing the effective memory location. Indicates the first A validity mask for each valid memory location. Indicates the first The relative write ratio of each effective memory location. This represents the threshold for the relative proportion of the residual. This indicates the direction of the mask mean for fused memories. This indicates that the gradient operator is stopped.

5. The robot control and benchmark evaluation method according to claim 1, characterized in that, S5 includes: Using the hidden state of the action as a query, the fused memory will be used. As keys and values, perform masked cross-attention retrieval and fusion memory. ; The search results are projected onto scale and offset parameters; Based on the projection results, the action hidden state is modulated, and the action block is predicted by the stream matching action head.

6. The robot control and benchmark evaluation method according to claim 1, characterized in that, S7 includes: Construct an evaluation task with historical three-dimensional state variables; In the preliminary stage of the task, the test policy that executes S1 to S6 obtains historical three-dimensional state variables from historical observations, where the historical three-dimensional state variables are used to influence the correct actions of the robot. After setting and recording the historical three-dimensional state variables, the strategy to be evaluated in the decision-making stage makes action decisions based on the current observations in the evaluation task, hiding the historical three-dimensional state variables or making the current observations ambiguous, so that the current observations are insufficient to uniquely determine the correct action. Control the closed-loop execution of the strategy under test within the task environment; Read the final physical state of the task environment when the task terminates; Based on the final physical state, the machine can execute a successful predicate to output the task success result.

7. A robot control and benchmark evaluation system, used to execute the robot control and benchmark evaluation method according to any one of claims 1-6, characterized in that, include: The observation acquisition module is used to acquire the robot's current and historical observations; The feature extraction module is used to extract semantic memory from historical observations. and three-dimensional memory representing the three-dimensional spatial state of a scene ; The association and memory enhancement module is used to build semantic memory. With three-dimensional memory The connections between time and local spatial location form a fusion memory enhanced by three-dimensional memory. ; The motion generation module is used to generate robot motions based on current observations and historical memories. The history update module is used to add new observations to the history after at least part of the robot's actions are performed; The benchmark construction and evaluation module is used to obtain historical three-dimensional state variables from historical observations, and to construct and evaluate the benchmark of the robot's three-dimensional memory ability based on the historical three-dimensional state variables and the current observations.

8. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the computer program is executed by the processor, the electronic device performs the robot manipulation and benchmark evaluation method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the robot manipulation and benchmark evaluation method according to any one of claims 1 to 6.

Images