VLA robot control system based on three-dimensional space and instance understanding enhancement

Abstract

This invention provides a VLA robot control system based on enhanced 3D spatial and instance understanding, belonging to the fields of computer vision and robot control technology. In this system, a multi-view spatial fusion module and a spatial conditional geometry aggregation module elevate the visual features of multi-view images to a unified global memory, and introduce explicit multi-view consistency constraints in all modalities, enabling the model to establish globally consistent geometric features. An object-centric 3D instance module directly initializes the state detector in continuous 3D space, extracting high-level 3D instance feature vectors of physical entities, achieving accurate perception of the object's 3D boundaries and geometry. A coordinate-driven 3D self-supervised prediction module retains the pre-trained self-supervised predictor and reuses it for feature regression under coordinate conditions, synthesizing occluded geometric features from partially visible perspectives, thus maintaining coherent reasoning ability even under severe occlusion.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and robot control technology, and relates to the establishment of a neural network model and system for three-dimensional perception and robot control using computer vision and deep learning technology. Specifically, it relates to a vision-language-action model (VLA) robot control system based on three-dimensional space and instance understanding enhancement. Background Technology

[0002] 3D perception and general robot control are the core foundations of intelligent systems such as intelligent manufacturing, smart home services, and autonomous mobile robots. To achieve this goal, in recent years, with the rapid development of large language models and large-scale multimodal datasets, Visual Language Algorithm (VLA) has become a key paradigm for general robot policy learning. By utilizing large-scale internet-based visual-language pre-trained weights, VLA has demonstrated remarkable operational capabilities on different tasks and various robot forms, highlighting its enormous practical value in the field of general robotics. Some existing advanced models (such as the pi0 and pi0.5 models based on stream matching or discrete action tagging) can perform complex long-field-of-view reasoning tasks and output continuous low-level control actions by fusing pre-trained visual-language models with motion expert decoders and jointly training with heterogeneous data from different robots.

[0003] However, developing general-purpose robot neural network models capable of high-precision operation in the real, complex, and uncertain physical world still faces many challenges. Despite significant progress in Virtual Laser Analog (VLA), current mainstream VLAs primarily rely on two-dimensional visual input for operation and lack true three-dimensional scene understanding. This deficiency hinders their deployment in real-world environments where spatial reasoning is crucial, such as untrained stranger families. Existing technologies mainly suffer from the following three limitations:

[0004] First, the primary challenge of current VLA lies in its weak ability to extract 3D spatial positions and the lack of multi-view consistency constraints. Existing VLA architectures typically encode single-view observations independently, or simply stitch and fuse multi-view features along the channel dimension without explicitly reconstructing 3D coordinates or enforcing cross-view geometric consistency. Due to the lack of a metric-level 3D reference coordinate system, the policy model cannot align its decisions with the real physical space, resulting in its inability to perform accurate 6-DOF pose inference.

[0005] Second, they lack the ability to deeply understand 3D instances. Standard VLAs only treat objects as two-dimensional image regions on a plane for reasoning, failing to acquire information about the object's volume, 3D boundaries, or fine-grained geometry. Therefore, these models struggle significantly when faced with tasks requiring precise spatial interaction (such as grasping specific details of an object or precise relative placement based on the 3D structure of another object) because they cannot reliably distinguish which pixels belong to which specific object instance in 3D space.

[0006] Third, the model's reasoning ability is extremely fragile when faced with occlusion. In real-world robotic operation scenarios, even slight changes in perspective can lead to the occlusion of critical parts of task-related objects. Because pure 2D models lack the ability to infer invisible geometry and maintain coherent spatial memory, existing VLAs can only generate actions based on incomplete or even misleading visual cues when faced with occlusion, resulting in a sharp decline in the success rate of grasping and manipulation.

[0007] While existing pure 3D scene understanding and perception models (such as point cloud-based 3D detection networks) offer powerful spatial reasoning capabilities, directly integrating them into VLA-based operational frameworks faces architectural incompatibility challenges. Furthermore, traditional 3D perception models heavily rely on the high cost of dense instance-level annotation. Forcibly modifying the underlying structure of existing VLAs to introduce 3D perception often forces the system to discard the visual-language pre-training prior knowledge that endows VLAs with strong generalization and language alignment capabilities. How to introduce powerful 3D reasoning capabilities to robots without disrupting existing visual-language model priors is a pressing issue that needs to be addressed. Summary of the Invention

[0008] To address the limitations of existing technologies, this invention provides a Vision-Language-Action (VLA) robot control system based on enhanced 3D spatial and instance understanding. This system designs a plug-and-play 3D enhancement framework (i.e., a 3D VLA framework) that injects robust 3D reasoning capabilities into the pre-trained VLA without requiring additional manual 3D intensive annotation or discarding the prior knowledge of the original visual language model. To address the first challenge, this invention proposes a multi-view spatial fusion module and a spatial conditional geometry aggregation mechanism. By enhancing the visual features of multi-view images to a unified global memory and introducing explicit multi-view consistency constraints across all modalities, the model can establish globally consistent geometric features. To address the second challenge, this invention designs an object-centric 3D instance module. This module does not rely on 2D masks but directly initializes the state detector in continuous 3D space, extracting high-level 3D instance feature vectors of physical entities, achieving accurate perception of the object's 3D boundaries and geometry. To address the third challenge, this invention proposes a coordinate-driven 3D self-supervised prediction module. This module retains the self-supervised predictor, which is usually discarded after pre-training, and reuses it for feature regression under coordinate conditions. It synthesizes occluded geometric features from partially visible perspectives, thus maintaining coherent reasoning ability even under severe occlusion.

[0009] The technical solution of the present invention is as follows:

[0010] A VLA robot control system based on 3D space and instance understanding enhancement, the system is equipped with a multi-view camera sensor and a robotic arm actuator, the VLA robot control system specifically includes:

[0011] 1) Multimodal data input and basic feature extraction module;

[0012] This module receives multi-view image data collected by the robot's multi-view camera sensor at the current moment, as well as natural language task instructions provided by the user; it uses a multi-view image feature encoder and a language instruction encoder to extract the visual features of the images, the textual semantic features of the language instructions, and the proprioceptive state features of the robot, respectively.

[0013] 2) Multi-view spatial fusion module, used to perform multi-view consistency feature enhancement processing to generate a unified visual feature representation with three-dimensional spatial attributes;

[0014] This module flattens the extracted image visual features in the spatial dimension, combines them with the corresponding non-projected 3D spatial coordinate grid and 3D position encoding, and establishes a cross-view geometric correspondence through a feature attention mechanism; it uses the 3D spatial coordinates to spatially align and index the image visual features, and outputs a global memory feature that has global view consistency and is associated with absolute physical coordinates.

[0015] 3) An object-centered 3D instance module, used to perform 3D instance feature extraction and dynamic calibration processing;

[0016] This module initializes multiple state detectors containing semantic states and 3D reference points in a continuous 3D space; using the features of the global memory bank, it extracts the texture and semantic information of specific instances through multi-scale feature sampling and cross-attention mechanism; and introduces an occlusion perception gating mechanism and a bounded coordinate refinement strategy to denoise and spatially calibrate the extracted features, outputting a clean and accurate 3D instance feature vector.

[0017] 4) A coordinate-driven 3D self-supervised prediction module is used to perform geometric feature completion and occlusion inference processing;

[0018] This module constructs a coordinate-driven 3D feature regression predictor; it utilizes a position and viewpoint conditional hybrid attention mechanism, combines absolute coordinate features with camera viewpoint encoding, and infers and synthesizes the geometric structure features of invisible or occluded areas based on the currently unoccluded visible context features, and outputs a 3D complete feature vector.

[0019] 5) Spatial condition geometry aggregation module and motion expert decoder module, used to perform feature relative spatial alignment and low-level control motion decoding output;

[0020] This module utilizes a spatial conditional geometry aggregation module to transform and project the 3D instance feature vector output by the object-centered 3D instance module and the 3D completion feature vector output by the coordinate-driven 3D self-supervised prediction module into a 3D relative position feature representation centered on the robot's end effector, based on the robot's current spatial pose. This 3D relative position feature is then injected (added) into the corresponding 3D instance feature vector and 3D completion feature vector, thereby aggregating and generating 3D instance-geometric aggregated features. These aggregated 3D instance-geometric aggregated features are then concatenated with the image visual features and text semantic features output by the multimodal data input and basic feature extraction modules to form a unified hybrid contextual cue sequence, which is input into the motion expert decoder. The motion expert decoder decodes, calculates, and outputs a continuous physical action sequence to control the robot's end effector based on the current task command and the fused hybrid contextual cue sequence.

[0021] Furthermore, the multimodal data input and basic feature extraction module specifically includes the following sub-modules:

[0022] 11) Image feature encoder module: This module acquires camera image data from multiple perspectives (such as wrist view, global view, etc.) around the robot control system in real time, and performs preprocessing such as cropping and scaling on the images; the preprocessed multi-view image data is input into the multi-view image feature encoder, which extracts the image visual features through the backbone network of the visual encoder and outputs the image visual features.

[0023] 12) Language instruction encoder module: This module synchronously acquires the natural language task instructions input by the user, inputs the text data into the language instruction encoder for feature encoding and feature embedding, and outputs the text semantic features of the language instructions.

[0024] 13) Proprioceptive state encoding and fusion module. This module synchronously acquires the robot's proprioceptive state input (including the joint angle of the robotic arm and the state of the end effector, etc.), and discretizes it into state features to characterize the current physical state starting point of the robot.

[0025] Furthermore, the multi-view spatial fusion module specifically includes the following sub-modules:

[0026] 21) A module for flattening two-dimensional features and associating them with a three-dimensional absolute spatial coordinate grid. This module receives the image visual features from the multimodal data input and the output of the basic feature extraction module. In order to construct a unified three-dimensional representation without introducing high three-dimensional voxelization computation costs, the traditional dense three-dimensional grid conversion scheme is abandoned. The two-dimensional image features from each viewpoint are flattened in the spatial dimension to form a disordered visual feature sequence.

[0027] 22) The nonlinear projection module from absolute coordinates to feature space: This module uses the unprojected original three-dimensional spatial coordinate data to perform nonlinear dimensionality upscaling and mapping through a coordinate projection network to calculate the position embedding features in absolute three-dimensional space; after layer normalization, the position embedding features are added to the flattened visual feature sequence to give the image visual features absolute physical space perception capability.

[0028] 23) Continuous 3D rotational position encoding injection module. This module uses a multi-layer Transformer encoder-based architecture to process the attention calculation within the coordinate-driven 3D self-supervised prediction module. To preserve fine-grained relative distance information for use by the subsequent self-supervised geometric completion module, continuous 3D relative position encoding is specifically introduced into the query and key vectors of the attention mechanism, forcing the network to learn strict Euclidean relative distance geometric priors.

[0029] 24) Global Geometric Consistency Self-Attention Calculation and Feature Output Module: This module interacts with the position embedding features calculated by the nonlinear projection module of the absolute coordinates into the feature space and the features added to the visual feature sequence. The network spontaneously establishes a cross-view geometric correspondence and outputs globally consistent global memory features. This feature eliminates the inherent scale and depth ambiguity of single-view 2D images, providing a unified metric space benchmark for subsequent 3D instance understanding.

[0030] Furthermore, the object-centered 3D instance module specifically includes the following sub-modules:

[0031] 31) 3D State Detector Generation Unit Module: This module initializes several 3D state detectors in the global 3D space. Each detector contains a high-dimensional semantic state vector and the physical coordinates of a 3D reference point. Prior to this, an upsampling decoding network is used to decode the global memory features generated by the multi-view spatial fusion module into a multi-scale semantic feature pyramid.

[0032] 32) Multi-scale feature sampling and interaction unit module: This module projects the detector's 3D reference points back to the 2D camera view planes using the camera intrinsic and extrinsic parameter matrices, and introduces a depth truncation normalization mechanism during the projection process to prevent numerical instability near the camera plane; using the projected 2D coordinates as the sampling center point, the detector's semantic state vector dynamically predicts the spatial 2D offset and attention weights, and extracts the corresponding object texture and semantic information from the multi-scale semantic feature pyramid through a deformable cross attention mechanism;

[0033] 33) Occlusion Perception Gating Unit Module: To filter geometric noise caused by occlusion or being in a blind spot, this module introduces an adaptive view validity gating mechanism. Features extracted from a specific viewpoint are nonlinearly evaluated using a multilayer perceptron, and a gating weight is generated by combining temperature parameters and an activation function to characterize the confidence level of the features at that viewpoint. The system uses the normalized gating weights to perform residual weighted summation on the features from all viewpoints, automatically filtering out low-quality camera viewpoint inputs, thereby updating the detector's semantic state vector.

[0034] 34) Bounded Coordinate Refinement Iterative Update Module: To correct the initial 3D reference point position error of the detector and ensure it accurately aligns with the geometric center of the real object, this module uses the updated semantic state vector to predict the compensation vector for the 3D position. To prevent spatial divergence errors such as coordinate deviations in the early stages of training, strict boundary constraints are imposed on the calculation of the compensation vector: nonlinear truncation is performed using the hyperbolic tangent activation function, and combined with a preset maximum physical step scaling factor, a bounded coordinate refinement offset is calculated to safely iteratively update the 3D reference point coordinates of the detector.

[0035] Furthermore, the coordinate-driven 3D self-supervised prediction module specifically includes the following sub-modules:

[0036] 41) Target Coordinate Sampling Unit Module: This module defines the mask coordinate set used for self-supervised distillation training, and the complete 3D coordinate set used to synthesize invisible geometric structures during the inference phase. During training, an asymmetric multi-view masking strategy forces the network to infer missing structures using cross-view geometric priors rather than simple spatial interpolation; unmasked regions form visible context features and their corresponding visible coordinate points. During inference, masking is disabled, and the predictor directly queries in continuous space using the completed coordinates.

[0037] 42) Hybrid Attention Unit Module with Position and Viewpoint Conditions: This module designs a 3D feature regression predictor for inferring unobserved features. This predictor needs to simultaneously encode absolute physical coordinates and specific camera viewpoint information. This unit injects continuous 3D absolute spatial position encoding and discrete camera viewpoint embedding features in parallel into the visible context features and the target query vector sequence, respectively. By separating identity feature injection and relative distance encoding, a hybrid attention operation with 3D relative position encoding is used in the cross-attention and self-attention layers to extract and aggregate 3D information across space.

[0038] 43) Exponential Moving Average Teacher Network Collapse Prevention Distillation and 3D Complete Feature Vector Output Module: To ensure the stability of self-supervised learning, a teacher network with the exact same structure as the predictor is constructed. However, the parameters of this teacher network do not participate in gradient backpropagation. Instead, during training, the weights of the main network are slowly and smoothly updated using an exponential moving average mechanism. Simultaneously, after continuous feature regression processing by multiple layers of position- and viewpoint-conditional hybrid attention unit modules, the predictor outputs a 3D complete feature vector corresponding to the target coordinates. This feature reconstructs the visual and geometric implicit representation of the occluded or invisible parts of the object, significantly enhancing the model's robustness under occlusion conditions.

[0039] Furthermore, the spatial condition geometry aggregation module and the action expert decoder module specifically include the following sub-modules:

[0040] 51) Spatial Conditional Coordinate Transformation Unit Module: This module receives the reference point coordinates of the 3D instance feature vector and the spatial coordinates corresponding to the 3D completion feature vector, and simultaneously acquires the 3D spatial pose of the robot end effector in the absolute world coordinate system in real time. Through coordinate subtraction and coordinate system rotation transformation, it calculates the 3D relative physical offset of each object instance and geometric completion point relative to the robot end effector. Then, it uses a spatial encoding function to perform high-dimensional mapping on the 3D relative physical offset to generate continuous relative spatial position encoding, and transforms it into interactive availability embedding features through a linear layer. The interactive availability embedding features are used as relative geometric priors and are added element-wise to the corresponding 3D instance feature vector and 3D completion feature vector, so that both carry physical distance information centered on the robot end effector.

[0041] 52) Instance and Geometric Completion Aggregation Module under Spatial Conditions: This module constructs a multimodal feature aggregation layer, introduces an attention mechanism including a distance space penalty term and a gated routing mechanism based on uncertainty evaluation, and dynamically weights and fuses the 3D instance feature vectors injected with relative spatial location information with the 3D completion feature vectors for information complementarity; this module combines the originally discrete instance semantics with the local completion geometry structure to output high-density, blind-spot-free 3D instance-geometric aggregation features;

[0042] 53) Hybrid prompting input module of motion expert decoder: The motion expert decoder adopts a parameter architecture independent of the visual backbone network; it splices and integrates 3D instance-geometric aggregation features, textual semantic features of language instructions and image visual features on the feature channel to form a hybrid context prompting sequence with complete 3D entity cognition and spatial perception.

[0043] 54) Action sequence trajectory generation strategy module based on flow matching architecture: In this module, the motion expert decoder, under the global guidance of the hybrid context cue sequence, processes the received state features and a noise vector representing multiple consecutive time steps of action sequence through a unidirectional attention mechanism; through a multi-layer attention mechanism, it deeply captures the complete three-dimensional geometry of the target object, the relative spatial pose of the end effector and the target, and the physical mapping relationship between the specific operation intention; after a flow field integration derivation process of a set number of steps, the model restores the initial random distribution to a high-fidelity robot motion trajectory distribution, and finally outputs a set of low-level continuous robot action command sequences within a given time span in the future. The command sequence specifically covers parameters such as the joint pose of the robotic arm and the opening and closing state of the end effector, so as to achieve the smoothness of robot motion and the coherence of long-term tasks.

[0044] Furthermore, the VLA neural network model is optimized during its training phase based on a self-supervised signal and action imitation multi-task learning loss system, employing a two-stage progressive training process, specifically as follows:

[0045] Phase 1: Spatial Fusion and Instance Extraction Pre-training; This phase focuses on learning physical entity features with strict viewpoint consistency, and jointly training a multi-view spatial fusion module, an object-centered 3D instance module, and an action expert decoder.

[0046] Phase 2: 3D completion and geometric aggregation refinement training; upon entering the second phase, the parameters of the perception module trained in the first phase are forcibly frozen; based on this, the coordinate-driven 3D self-supervised prediction module and the spatial conditional geometric aggregation module are trained, and the motion expert decoder is continuously updated.

[0047] Compared with the prior art, the present invention has the following significant technical effects:

[0048] 1. Significantly enhances the robot's 3D spatial perception and multi-view consistent understanding capabilities. The multi-view spatial fusion module proposed in this invention abandons the simple 2D feature channel stacking of traditional VLA. By injecting globally consistent 3D absolute coordinates and relative distance encodings into the entire network architecture, it forces the model to understand the physical geometric relationships between multiple camera views. This provides the robot with a unified metric-level reference coordinate system, enabling the model to perform high-precision six-DOF pose reasoning when performing tasks such as spatial arrangement and precise grasping, completely overcoming the depth and scale misjudgment problems easily caused by planar vision.

[0049] 2. This invention overcomes the limitations of traditional 2D instance segmentation, achieving precise 3D instance-level interaction. The object-centric 3D instance module directly initializes and updates the physical entity detector in continuous 3D space, extracting 3D instance feature vectors unaffected by viewpoint switching and image plane overlap. This mechanism, combined with the spatial conditional geometry aggregation module, converts the absolute 3D position of the instance into a relative offset centered on the robot's physical actuator. This allows the robot to directly understand the spatial interaction relationship between the object instance and itself, thereby accurately distinguishing and manipulating closely arranged or similarly textured object instances in extremely complex desktop or indoor scenes.

[0050] 3. Significantly improved robustness in complex occlusion environments. Addressing the frequent issues of blind spots and object occlusion in real-world physical environments, this invention innovatively introduces a coordinate-driven 3D self-supervised prediction mechanism. This mechanism, in conjunction with the 3D self-supervised prediction module, reuses the feature completion predictor from the mask self-supervised task during the inference phase. The model can automatically infer and synthesize the complete 3D structural semantic features of invisible or occluded parts of objects based on known locally visible geometric context. This greatly reduces motion jitter or erroneous trajectory generation caused by missing cues when encountering visual occlusion, ensuring a high success rate for complex tasks such as long-field-of-view, multi-stage cleaning and tidying, thereby achieving high-precision, highly generalizable dexterous maneuver control of robots in the physical world.

[0051] 4. It perfectly preserves the strong generalization prior of the visual-language large model and supports efficient end-to-end learning and high-frequency deployment. The framework of this invention adopts a plug-and-play, non-intrusive design concept. All 3D enhancement modules are integrated in the form of parallel feature injection or side-channel prediction, without modifying the core attention mechanism of the original visual-language model. It fully retains the general semantic knowledge and zero-shot generalization ability trained on massive amounts of internet text and image data. Furthermore, by using a hybrid two-stage training strategy, combining the first stage of discrete action feature pre-training with massive amounts of data with the second stage of continuous action post-training based on action expert decoders, this invention significantly improves the accuracy of continuous action generation and real-time control inference speed of the robot while maintaining strong cross-domain cognitive generalization capabilities. This enables the model to be directly deployed in completely unfamiliar real-world home operating environments. Attached Figure Description

[0052] Figure 1 This is a comparison diagram showing the differences between the VLA robot control system based on three-dimensional space and instance understanding enhancement of the present invention and the traditional system;

[0053] Figure 2 This is a schematic diagram of the overall framework and execution logic of a VLA robot control system based on three-dimensional space and instance understanding enhancement according to the present invention;

[0054] Figure 3 This is a schematic diagram of the execution logic of the object-centered 3D instance module (left) and the spatial conditional geometry aggregation module (right) in this invention. Detailed Implementation

[0055] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0056] This invention proposes a VLA robot control system based on enhanced 3D spatial and instance understanding. Addressing three core pain points of existing mainstream VLA systems in general robot control applications—namely, weak 3D spatial position extraction capabilities and lack of multi-view consistency constraints, insufficient deep understanding of 3D physical instance boundaries and geometry, and extreme vulnerability to inference when faced with complex perspective changes and occlusion by the robotic arm itself in the real physical world—this invention innovatively designs four core enhancement components: a multi-view spatial fusion module, an object-centered 3D instance module, a coordinate-driven 3D self-supervised prediction module, and a spatial conditional geometry aggregation module. For the differences between the overall architecture of this invention and existing technologies, please refer to [link to relevant documentation]. Figure 1 For the overall architecture and execution logic, please refer to [link / reference]. Figure 2 .

[0057] The core concept of this invention lies in designing a plug-and-play 3D vision enhancement framework that seamlessly integrates robust and accurate 3D spatial geometric reasoning capabilities into a VLA pre-trained on massive datasets. This solution does not require drastically altering the internal structure of the underlying visual language model, thus preserving the powerful language alignment and logical reasoning prior knowledge learned by the model from billions of data points on the internet.

[0058] This invention relates to a VLA robot control system based on enhanced 3D spatial and instance understanding. The system deploys a Vision-Language-Motion Neural Network (VLA) model, equipped with multi-view camera sensors and a robotic arm actuator. The VLA neural network model includes a multi-view image feature encoder, a language command encoder, a multi-view spatial fusion module, an object-centered 3D instance module, a coordinate-driven 3D self-supervised prediction module, a spatial conditional geometry aggregation module, and a motion expert decoder. During the training phase, the VLA neural network model is optimized based on a self-supervised signal and motion imitation multi-task learning loss system. The VLA robot control system specifically includes:

[0059] S1. Multimodal Data Input and Basic Feature Extraction Module: This module acquires image data collected by the robot's multi-view cameras, natural language commands issued by the user, and the robot's proprioceptive state input in real time; it then uses a pre-built multimodal feature encoder to parse the data and extract image visual features, text semantic features, and state features representing the robot's starting point.

[0060] S2. Multi-view spatial fusion module: This module fuses the multi-view image visual features extracted in module S1 with three-dimensional absolute spatial coordinates and position codes. It establishes a spatial geometric correspondence across cameras through a feature attention mechanism and outputs a global memory feature that eliminates multi-view differences and has global view geometric consistency.

[0061] S3. Object-Centric 3D Instance Feature Extraction and Dynamic Calibration Module: This module initializes the state detector in continuous 3D space. Utilizing the global memory features built in module S2, it extracts feature information of specific objects through dynamic sampling and cross-attention mechanisms. This module employs an automatic pseudo-labeling alignment strategy based on 3D space: it automatically labels multi-view images using an externally pre-trained general vision-language segmentation model and back-projects the 2D segmentation results into 3D space to generate instance-level pseudo-labels, serving as supervision signals during the training phase to achieve instance-level semantic perception in 3D space. Finally, it combines occlusion-aware gating and coordinate refinement strategies for denoising, outputting clean and accurate 3D instance feature vectors. For a schematic diagram of the module architecture and execution logic, please refer to [link to module architecture diagram]. Figure 3 Left sub-diagram.

[0062] S4. Coordinate-driven 3D self-supervised prediction and occlusion geometry completion module: This module addresses the information loss caused by robotic arm occlusion or blind spots by constructing a coordinate-condition-based 3D feature regression predictor; utilizing the contextual features of the visible parts, it directly infers and synthesizes the object geometry of the invisible or severely occluded areas in the latent feature space, and outputs a 3D completion feature vector.

[0063] S5. Spatial Conditional Geometry Aggregation and Motion Expert Decoder Module: This module converts the 3D instance feature vectors from module S3 and the 3D completion feature vectors from module S4 into 3D relative position features centered at the robot's end effector, and aggregates them to generate 3D instance-geometric aggregated features (see the schematic diagram of this module's architecture and execution logic). Figure 3 (Right subgraph); Finally, the aggregated features are concatenated with the basic multimodal features of module S1 to form a unified hybrid contextual cue sequence, which is then input into the motion expert decoder to decode and output a continuous physical motion sequence that directly controls the robot's actuators.

[0064] In this embodiment, the robot control system is specifically embodied as a multimodal general-purpose intelligent mobile robot platform for unstructured physical environments such as complex and unfamiliar home services and intelligent manufacturing. The robot control system is equipped with various configurations of external environment and body perception sensors: a monocular RGB camera sensor for close-range precision operation is installed at the end of the robotic arm (wrist) and in a clear global field of view visible from the desktop. These sensors provide a wide range of observational inputs, which may include overlapping and occlusion of viewpoints. Furthermore, the actuators include a robotic arm with multiple degrees of freedom and an end effector.

[0065] This embodiment also provides a specific implementation method for pre-building and training the aforementioned VLA neural network model (i.e., 3D VLA), combined with the VLA robot control system based on three-dimensional space and instance understanding enhancement of the present invention, as follows:

[0066] S1. Multimodal data input and basic feature extraction module, which specifically includes the following sub-modules:

[0067] S1.1 Image Feature Encoder Module: In this embodiment, to avoid losing the powerful prior world knowledge provided by the pre-trained large model, the visual branch of an industry-leading large-scale pre-trained visual language model (such as the Vision Transformer (ViT) in the PaliGemma model with a total of 2 billion parameters) is selected as the core image feature encoder. Raw RGB image data collected in real-time from multi-view camera sensors around the robot control system is acquired. Necessary preprocessing operations (including size cropping, resolution scaling, and normalization) are performed on the images. The preprocessed multi-view image tensors are input into the backbone network of the vision transformer, dividing the image into non-overlapping two-dimensional sub-blocks. After linear projection and multi-layer self-attention mechanisms, high-dimensional image visual features are extracted.

[0068] S1.2 Language Instruction Encoder Module: This module synchronously acquires natural language task instructions (such as "Please put the blue mug on the table into the dishwasher") issued by the human operator in the form of voice or text. This embodiment uses a causal Transformer large language model (e.g., the Gemma architecture) with billions of parameters as the backbone network of the language instruction encoder. The natural language data is segmented into discrete features by a dedicated text segmenter, and then mapped to continuous-dimensional embedding vectors using the word embedding matrix of the large-scale language model. These embedding vectors are further semantically refined within the language instruction encoder through a bidirectional attention layer, outputting textual semantic features that accurately represent the boundaries of the intent.

[0069] S1.3 Proprioceptive State Encoding and Fusion Module: In addition to external commands, this module synchronously acquires the robot's proprioceptive input to enable the strategy model to understand its current state. This input includes the joint angles of the multi-degree-of-freedom robotic arm and the opening and closing width of the end effector. These continuous floating-point physical quantities are digitized and discretized into specific state features, representing the robot's current proprioceptive physical starting point as an independent feature sequence.

[0070] S2. Multi-view spatial fusion module, used to perform multi-view consistency feature enhancement processing, specifically includes the following sub-modules:

[0071] S2.1 Module for Flattening 2D Features and Associating with a 3D Absolute Spatial Coordinate Grid: This module first flattens the multi-view visual features extracted by the image feature encoder's construction and execution module along the spatial dimension, converting the original 3D feature map tensor (number of channels × height × width) into an unordered 2D feature sequence matrix (sequence length × number of channels, where the sequence length equals the product of height and width multiplied by the number of viewpoints). Next, to give this sequence, which has lost its spatial arrangement structure, a real physical world meaning is given, using the calibrated intrinsic and extrinsic parameter matrices of each camera, a 3D absolute spatial coordinate grid (containing real X, Y, Z coordinate points) that has not undergone perspective projection scaling is directly calculated and assigned.

[0072] S2.2 Nonlinear Projection Module for Absolute Coordinates to Feature Space: This module inputs the generated 3D spatial coordinate grid into a lightweight coordinate projection neural network. This network is configured as a multilayer perceptron, and the specific mathematical operations are as follows: First, the 3D physical coordinate vector is multiplied by a first weight matrix and a first bias term is added for linear dimensionality increase. Then, it is fed into a Gaussian error linear unit activation function for nonlinear transformation to enhance feature representation. Next, it is multiplied by a second weight matrix and a second bias term is added. Finally, layer normalization is performed. Through these operations, the low-dimensional 3D absolute coordinates are converted into a 3D positional code that is completely consistent with the number of image visual feature channels. This code is then added element-wise to the flattened image visual features, thereby achieving rigorous absolute spatial domain alignment of the visual features.

[0073] S2.3 Continuous 3D Rotational Position Encoding Injection Module: To ensure that feature fusion relies not only on absolute coordinates but also allows the network to clearly perceive the physical relative distance between any two feature points, this invention further introduces continuous 3D rotational position encoding as an example of 3D relative position encoding in the attention mechanism calculation stage of the coordinate-driven 3D self-supervised prediction module (S4). Specifically, this module defines a formula for calculating the attenuation of continuous phase angles for each component (X, Y, Z) of the spatial coordinates, concatenates these independent continuous embeddings, and applies them specifically to the query and key matrices of the attention mechanism. This operation introduces a strong Euclidean relative distance prior to the network.

[0074] S2.4 Global Geometric Consistency Self-Attention Calculation and Feature Output Module: This module feeds the feature sequence, which is the element-wise sum of the aforementioned 3D positional encoding and the flattened image visual features, into a feature fusion center composed of multi-layer Transformer encoders. Driven by the attention mechanism, this encoder can automatically find and aggregate feature points that, although their appearance differs greatly in different cameras, actually belong to the same physical structure or are in adjacent locations in the 3D coordinate system, across the physical field of view boundaries of different cameras. After multiple multi-head self-attention calculations and feedforward layer processing, the final output is a global memory feature that integrates rich texture semantics and fully possesses global perspective 3D consistency.

[0075] S3. Object-centered 3D instance module, used to perform 3D instance feature extraction and dynamic calibration processing, specifically includes the following sub-modules:

[0076] S3.1 3D State Detector Generation Unit Module: Unlike traditional systems that rely on pre-generated 2D proposal boxes, this module directly focuses on the continuous 3D world. The system pre-instantiates a large number of state detectors within the task's 3D space. Each detector contains two key components: a high-dimensional feature vector representing the individual's high-level semantic state, and a 3D reference point representing the individual's centroid location in 3D space. Simultaneously, a transposed convolutional network is used to deconvolve and decode the global memory features obtained from module S2, restoring spatial details and generating multi-scale semantic pyramid features with multiple receptive field scales.

[0077] S3.2 Multi-scale Feature Sampling and Interaction Unit Module: This module requires the 3D reference points of the 3D detector to be reprojected onto the 2D observation planes of each camera in order for the 3D detector to acquire its corresponding visual features. When performing projection calculations using the camera's extrinsic and intrinsic parameter matrices, if the 3D points are very close to the camera's imaging physical cross-section (i.e., depth Z is close to 0), the perspective projection denominator becomes extremely small, leading to numerical explosion. To address this issue, the system designs a depth truncation mechanism, forcibly adding a very small positive stability constant threshold to the projection denominator, ensuring absolute projection safety. Using these safe 2D coordinate points, after projection and normalization, as anchor points, the detector's semantic state vector dynamically predicts the sampling offset and sampling attention weight distribution on the multi-scale semantic pyramid features through a linear layer. A deformable cross-attention mechanism is employed to extract features only from a few key pixel regions activated by attention weights near the anchor points.

[0078] S3.3 Occlusion Perception Gating Unit Module: An object is often not simultaneously visible to all of the robot's cameras. Forcibly fusing sampled features from all viewpoints would severely contaminate the object's representation due to noise outside the field of view. Therefore, this module introduces an adaptive view validity gating mechanism. For each detector's features extracted from a specific viewpoint of its projection, a multilayer perceptron performs nonlinear evaluation, and the output is divided by a learnable temperature parameter controlling sharpness. Finally, a gating weight between 0 and 1 is generated using a sigmoid activation function. This weight represents the confidence level of the feature at that viewpoint. The system uses these normalized gating weights to perform residual weighted summation on the features from all viewpoints, automatically filtering out low-quality camera viewpoints that are facing away from the object or occluded, and updating the detector's semantic state.

[0079] S3.4 Bounded Coordinate Refinement Iterative Update Module: Since the initial 3D reference points of the detector are randomly or uniformly generated, to ensure they accurately align with the geometric center of the real object, the module predicts a 3D position compensation vector using the updated semantic state vector after each attention network layer. To prevent severe coordinate deviations from the scene during early training when network parameters oscillate wildly, strict boundary constraints are imposed on the calculation of the compensation vector: the predicted coordinate offset is nonlinearly truncated using a hyperbolic tangent activation function (the output is strictly constrained to between -1 and 1), and then multiplied by a predefined maximum physical step size (0.3). This safe and bounded compensation value is added back to the 3D reference point of the previous layer. Through multi-level sampling, gating fusion, and coordinate refinement iterations within the module, a clean 3D instance feature vector, stripped of environmental background and with extremely accurate spatial positioning, is output.

[0080] S4. Coordinate-driven 3D self-supervised prediction module, used to perform geometric feature completion and occlusion inference processing, specifically including the following sub-modules:

[0081] S4.1 Target Coordinate Sampling Unit Module: During the non-deployment training phase of the model, to force the network to learn to infer physical occlusion, the module performs asymmetric random feature masking on the image features from multiple viewpoints entering the attention network. The unmasked parts constitute the visible context features; the masked parts correspond to the mask coordinates to be learned and reconstructed. Based on this, in order to extract occlusion completion coordinates that are realistically meaningful for the deployment phase, the occluded or invisible areas of the object are located online in 3D space according to the mask boundaries, and a set of completed 3D coordinates for inference is directly generated within this area using a completion model (e.g., PointAttn). A dynamic clustering algorithm based on physical density is used to remove flying points and isolated noise. Then, a set of 3D coordinates that are truly in the visual blind spot and farthest from the visible points on the surface is extracted. After the farthest point sampling algorithm, n completed 3D coordinate points are uniformly distributed for each instance (e.g., 5 completed points per instance).

[0082] S4.2 Hybrid Attention Unit Module with Position and Viewpoint Conditions: This module constructs a specially designed 3D feature regression predictor. To accurately predict unseen parts based on the visible context, this predictor employs a hybrid 3D attention mechanism. Unlike ordinary self-attention, this mechanism first generates a 3D position code expanded by a continuous Fourier transform to characterize the 3D absolute spatial frequency characteristics, and simultaneously generates a discrete categorical camera viewpoint embedding code. These two are then added together and injected into the target query vector representing the invisible region and the context feature vector representing the visible region, respectively. In the single-layer module of the predictor, two steps are performed sequentially: the first step is a cross-attention stage, where the target query vector, through the mechanism described above with relative rotation position code injection, crosses spatial distances to extract useful edge and texture features from the visible context; the second step is a self-attention stage, where each target query vector exchanges its inferred feature fragments. Through this high-frequency interaction, the predictor ensures that the reconstructed occluded parts possess physical geometric coherence.

[0083] S4.3 Exponential Moving Average Teacher Network Anti-Collapse Distillation and Output 3D Completed Feature Vector Module: Since coordinate completion is an open task of regressing floating-point vectors in a continuous high-dimensional space, if only a single error loss is used, the network is prone to outputting a global average value to avoid feature collapse. To ensure the stability of self-supervised learning, a teacher network with the exact same structure as the predictor is constructed. However, the parameters of this teacher network do not participate in gradient backpropagation. Instead, during training, the weight updates of the main network are slowly and smoothly integrated using an exponential moving average mechanism. The unoccluded features predicted by the predictor and the target features output by the teacher network are calculated using multi-dimensional differences: First, a smooth L1 regression loss is applied to align the two in terms of absolute numerical magnitude; second, a cosine similarity penalty loss independent of vector length is applied to force the network to learn the correct high-dimensional semantic direction; finally, a variance regularization penalty term is forcibly calculated and introduced into the total loss. By constraining the standard deviation of each channel dimension to be greater than a certain set threshold, the path of outputting homogeneous and mediocre features is completely blocked. After the above deduction and optimization, this module can finally stably output a hidden structure 3D completion feature vector with high fidelity.

[0084] S5. Spatial Conditional Geometry Aggregation Module and Action Expert Decoder Module, used to perform feature relative spatial alignment and low-level control action decoding output, specifically includes the following sub-modules:

[0085] S5.1 Spatial Conditional Coordinate Transformation Unit Module: In the real physical world, low-level continuous actions such as grasping, placing, and wiping rely on the relative pose transformation between the robotic arm's end effector and the target object. To prevent the downstream motion decoder from memorizing complex absolute global coordinate transformation matrices, this module actively performs spatial relative coordinate correction. It reads the centroid coordinates of the 3D instance feature vector extracted by module S3 and the spatial coordinates of the 3D completion feature vector predicted by module S4, subtracting the physical 3D coordinates of the robot's end effector in the current frame from these coordinates to obtain the 3D relative spatial offset with the end effector as the origin. Furthermore, simple 3D differences, due to their flat numerical values, are difficult for neural networks to capture millimeter-level operational deviations. This invention applies a continuous frequency basis expansion of sine and cosine to this relative spatial offset (i.e., substituting the offset vector into sine and cosine formulas with different harmonic exponents). After capturing high-frequency spatial topological changes through multi-band expansion, it is fed into a multilayer perceptron to project onto a channel dimension equal in width to the instance and completion features, and then summed and injected back into the features themselves. This gives all 3D object information passed to the subsequent action decoding module a strong sense of spatial accessibility for operation.

[0086] S5.2 Instance and Geometric Completion Aggregation Module under Spatial Conditions: This module unconditionally fuses the 3D completion feature vector generated by module S4 into the high-confidence 3D instance features of module S3. However, this can introduce redundant interference when there is no severe occlusion. Therefore, an uncertainty quantification evaluation system based on the instance confidence of the detector output is designed. The instance confidence of each pixel calculated together with the instance mask in S3 is processed by the Sigmoid activation function, and then subtracted from the result by 1 to obtain the visual uncertainty score of the instance (a score close to 1 indicates severe occlusion or blurriness, and close to 0 indicates clear visibility). Simultaneously, based on 3D Euclidean distance, the nearest neighbor 3D completion feature point group is retrieved near each instance location, and aggregated through an attention mechanism including a distance spatial penalty term to obtain the local geometric context adjacent to the instance. Subsequently, the 3D instance features, local geometric context, and the aforementioned uncertainty score are concatenated at the channel level and fed into a gate control multilayer perceptron to calculate a control gate vector with dimensions equal to the number of channels. Crucially, the final layer of this gated multilayer perceptron is intentionally given a large negative constant bias (e.g., -3.0). This design ensures that the value of the gate vector approaches 0 during the initial training phase and when uncertainty is extremely low, maintaining the identity mapping of the original instance features. Only when the network perceives high uncertainty and strong correlations exist between the locally completed features is the gate coefficient dynamically amplified to absorb a large proportion of the completed geometric details. This fusion process ultimately outputs a smooth 3D instance-geometric aggregation feature with strong 3D geometric cognition.

[0087] The S5.3 Action Expert Decoder Hybrid Prompt Input Module: This module is equipped with an action expert decoder independent of the visual backbone network. It integrates the 3D instance-geometric aggregation features produced by module S5.2 with the image visual features and text semantic command features extracted by module S1 on the feature channels to form a unified hybrid contextual prompt sequence.

[0088] S5.4 Action Sequence Trajectory Generation Strategy Module Based on Flow Matching Architecture: The action expert decoder module uses a continuous flow matching architecture to generate action sequence trajectories. During inference, the input of the action expert decoder simultaneously receives robot state features and a noise vector to be processed, generated by a pure Gaussian normal distribution, representing action sequences with multiple consecutive time steps in the future. Crucially, the system employs a one-way attention masking mechanism to connect the aforementioned hybrid contextual cue sequence with the action expert decoder: during the processing of the action expert decoder, the input action noise features can cross-reference the preceding hybrid contextual cue sequence; conversely, all cue sequences generated by visual and linguistic features must never cross-reference any action features. Inside the action expert decoder, the time step information of the flow matching is processed by a multilayer perceptron using the Swish activation function, transformed into scaling and translation parameters, and flexibly modulated and injected into each self-attention transformation layer of the decoder using an adaptive layer normalization mechanism (such as AdaLN-Zero or adaptive RMSnorm). Ultimately, the action expert decoder does not directly output a single-step action, but predicts a flow field velocity vector that can drive the current noise state towards the target's true action trajectory distribution. By combining an ordinary differential equation solver with a few smooth integral denoising iterations, a precise continuous motion trajectory is deterministically reconstructed from random noise. This sequence, containing the joint poses and end effector states of the robotic arm across multiple future frames, is directly converted into a target reference signal for the underlying controller, driving the robot to smoothly execute intricate and complex physical interactions.

[0089] The following are the specific training strategies, core structural parameters, loss function, and hyperparameter settings for training the VLA neural network model in this embodiment:

[0090] (1) Hyperparameter configuration of core network module:

[0091] To maximize parameter efficiency, the backbone network weights of the large-scale basic vision-language model are kept completely frozen throughout the training process. The system only updates the gradients of the newly introduced multi-view spatial fusion module, object-centered 3D instance module, coordinate-driven 3D self-supervised prediction module, spatial conditional geometry aggregation module, and action expert decoder module.

[0092] For the object-centric 3D instance module, the system initializes 300 state detectors to ensure comprehensive coverage of complex scenes; then, through feature matching and uncertainty filtering mechanisms, it adaptively extracts up to 16 high-confidence instance features for each scene.

[0093] For the coordinate-driven 3D self-supervised prediction module, in order to balance the consumption of computing resources and geometric fidelity, the network is set to process a maximum of 6 occluded object instances in each observation, and each object instance is uniformly sampled with exactly 5 3D completion coordinate query points.

[0094] (2) Optimizer configuration and learning rate scheduling:

[0095] This embodiment uses the AdamW optimizer to update model parameters. The gradient clipping norm is set to 0.5, and the weight decay coefficient is set to... The global effective batch size is set to 64.

[0096] The learning rate scheduling employs a cosine annealing decay strategy, specifically including a 1200-step warm-up phase, to allow the learning rate to smoothly climb to its peak. Then it enters a decay phase of 3800 steps, during which the learning rate gradually decreases to a minimum. This ensures extremely high convergence stability of the model during training on heterogeneous data.

[0097] (3) Combination of loss function and weight coefficients:

[0098] To match the numerical scale of heterogeneous geometric properties and high-dimensional features, the final optimized total loss function of the system is composed of the following joint task loss weights, with the weight coefficients of each sub-loss being rigorously set empirically:

[0099] (3.1) Joint geometric alignment and instance-aware multi-task loss: For the extraction constraints of 2D to 3D instance boundaries, a classification evaluation loss coefficient is set. Two-dimensional detection box regression loss coefficient Generalized intersection-exchange ratio loss coefficient Mask cross-entropy loss coefficient DICE pixel overlap loss coefficient And the most crucial absolute three-dimensional position coordinate regression constraint loss coefficient. .

[0100] (3.2) Composite self-supervised distillation inference loss: For the coordinate-driven 3D self-supervised prediction module, in order to strictly prevent the homogenization collapse of feature extraction and maintain the consistency of high-dimensional orientation, a feature reconstruction smoothing L1 loss coefficient for numerical alignment is set. 1.0, Semantic Direction Cosine Alignment Loss Coefficient Anti-collapse variance regularization penalty loss coefficient Furthermore, the update parameter for the teacher network exponential moving average momentum used to stabilize the target features was set to m=0.999.

[0101] (3.3) Action flow matching and language instruction constraint loss: For downstream control output, calculate the mean square error between the flow field velocity vector generated by the real high-frequency action trajectory and the prediction vector of the expert decoder, while retaining the text cross-entropy loss used for text semantic task alignment.

[0102] (4) Two-stage progressive training steps:

[0103] To ensure the stability of the optimization and prevent continuous high-dimensional feature regression tasks from dominating or causing early learning phase collapse, the training process is decoupled into a progressive two-stage strategy. The entire training process is highly efficient, deployed on a single compute node equipped with four NVIDIA A800 GPUs, and takes only about 14 hours in total.

[0104] Phase 1: Spatial Fusion and Instance Extraction Pre-training.

[0105] This phase focuses on learning the physical entity features with strict viewpoint consistency, jointly training a multi-view spatial fusion module, an object-centric 3D instance module, and a motion expert decoder. The training iterations in this phase are set at 5000 steps, using a batch size of 64, with a total time consumption of approximately 5 hours. This phase lays the foundation for accurate 2D-to-3D geometric anchoring of the model.

[0106] Phase Two: 3D Completion and Geometric Aggregation Refinement Training.

[0107] In the second phase, the parameters of the perception module trained in the first phase are forcibly frozen. Based on this, a coordinate-driven 3D self-supervised prediction module and a spatial conditional geometry aggregation module are trained, and the action expert decoder is continuously updated. This phase also performs 5000 iterations, using a batch size of 64, and takes approximately 9 hours.

[0108] To verify the effectiveness of this invention, the model underwent rigorous validation on a benchmark set containing large-scale, complex physical interaction environments. The testing phase was expanded to eight NVIDIA A800 GPUs to handle high-frequency physical environment simulation rendering and evaluation. Qualitative and quantitative comparisons show that in constrained physical scenes where the target object is severely occluded or the viewpoint changes drastically, the baseline visual-language-action model often completely loses its grasping and localization capabilities. However, the system of this invention successfully completes the geometric blind spot features through the aforementioned enhancement modules, achieving accurate spatial alignment. On the two challenging benchmarks of LIBERO-Plus and RoboTwin2.0, this embodiment achieved state-of-the-art performance in both grasping success rate and cross-scene generalization testing.

[0109] The core of this invention is the construction of a VLA robot control system based on enhanced 3D spatial and instance understanding, aiming to achieve high-precision, high-stability, and occlusion-resistant general robot manipulation. Based on existing advanced VLA architectures (such as pi0.5), this system innovatively designs four plug-and-play parameter-efficient functional modules. These modules include: a multi-view spatial fusion module for establishing 3D feature representations with global viewpoint geometric consistency; an object-centered 3D instance module for achieving accurate instance-level feature stripping and localization in continuous space; a coordinate-driven 3D self-supervised prediction module for inferring the complete physical structure of the target when vision is limited or occluded; and a spatial conditional geometry aggregation module for dynamically aligning 3D perception information with the relative pose of the actuator. Through the synergistic effect of these modules, this invention can significantly improve the accuracy, real-time performance, and robustness of robots performing complex control actions in unstructured physical environments. Experiments show that this invention demonstrates strong application value and generalization potential in various real-world complex manipulation tasks.

[0110] Another embodiment of the present invention provides an intelligent robot that integrates the above-mentioned robot control system including a VLA neural network model based on three-dimensional space and instance understanding enhancement.

[0111] Another embodiment of the present invention provides a computer device including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for implementing the functions of various modules of the system of the present invention.

[0112] Another embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the functions of each module of the system of the present invention.

[0113] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A vision-language-action model robot control system based on three-dimensional space and instance understanding enhancement, the system being equipped with a multi-view camera sensor and a robotic arm actuator, characterized in that, The vision-language-motion model robot control system includes: 1) Multimodal data input and basic feature extraction module; This module receives multi-view image data collected by the robot's multi-view camera sensor at the current moment, as well as natural language task instructions provided by the user; it uses a multi-view image feature encoder and a language instruction encoder to extract the visual features of the images, the textual semantic features of the language instructions, and the proprioceptive state features of the robot, respectively. 2) Multi-view spatial fusion module, used to perform multi-view consistency feature enhancement processing to generate a unified visual feature representation with three-dimensional spatial attributes; This module flattens the extracted image visual features in the spatial dimension, combines them with the corresponding non-projected 3D spatial coordinate grid and 3D position encoding, and establishes a cross-view geometric correspondence through a feature attention mechanism; it uses the 3D spatial coordinates to spatially align and index the image visual features, and outputs a global memory feature that has global view consistency and is associated with absolute physical coordinates. 3) An object-centered 3D instance module, used to perform 3D instance feature extraction and dynamic calibration processing; This module initializes multiple state detectors containing semantic states and 3D reference points in a continuous 3D space; utilizes the features of the global memory bank to extract the texture and semantic information of specific instances through multi-scale feature sampling and cross-attention mechanism; and introduces an occlusion perception gating mechanism and a bounded coordinate refinement strategy to denoise and spatially calibrate the extracted features, outputting a 3D instance feature vector. 4) A coordinate-driven 3D self-supervised prediction module is used to perform geometric feature completion and occlusion inference processing; This module constructs a coordinate-driven 3D feature regression predictor; it utilizes a position and viewpoint conditional hybrid attention mechanism, combines absolute coordinate features with camera viewpoint encoding, and infers and synthesizes the geometric structure features of invisible or occluded areas based on the currently unoccluded visible context features, and outputs a 3D complete feature vector. 5) Spatial condition geometry aggregation module and motion expert decoder module, used to perform feature relative spatial alignment and low-level control motion decoding output; This module utilizes a spatial conditional geometry aggregation module to transform and project the 3D instance feature vector output by the object-centered 3D instance module and the 3D completion feature vector output by the coordinate-driven 3D self-supervised prediction module into a 3D relative position feature representation centered on the robot's end effector, based on the robot's current spatial pose. This 3D relative position feature is then injected into the corresponding 3D instance feature vector and 3D completion feature vector, thereby aggregating and generating 3D instance-geometric aggregated features. These aggregated 3D instance-geometric aggregated features are then concatenated with the image visual features and text semantic features output by the multimodal data input and basic feature extraction modules to form a unified hybrid contextual cue sequence, which is input into the motion expert decoder. The motion expert decoder decodes, calculates, and outputs a continuous physical action sequence to control the robot's end effector based on the current task command and the fused hybrid contextual cue sequence.

2. The system as described in claim 1, characterized in that, The multimodal data input and basic feature extraction module specifically includes the following sub-modules: 11) Image feature encoder module: This module acquires real-time camera image data from multiple perspectives around the robot control system and performs preprocessing such as cropping and scaling on the images; the preprocessed multi-view image data is input into the multi-view image feature encoder, which extracts the image visual features through the backbone network of the visual encoder and outputs the image visual features. 12) Language instruction encoder module: This module synchronously acquires the natural language task instructions input by the user, inputs the text data into the language instruction encoder for feature encoding and feature embedding, and outputs the text semantic features of the language instructions. 13) Proprioceptive state encoding and fusion module: This module synchronously acquires the robot's proprioceptive state input and discretizes it into state features to characterize the current physical state starting point of the robot.

3. The system as described in claim 1, characterized in that, The multi-view spatial fusion module, specifically Includes the following sub-modules: 21) A module for flattening two-dimensional features and associating them with a three-dimensional absolute spatial coordinate grid. This module receives the image visual features from the multimodal data input and the output of the basic feature extraction module, constructs a unified three-dimensional representation, and flattens the two-dimensional image features from each perspective in the spatial dimension to form a disordered visual feature sequence. 22) Nonlinear projection module from absolute coordinates to feature space: This module uses the unprojected original 3D spatial coordinate data to perform nonlinear dimensionality upscaling and mapping through a coordinate projection network to calculate the position embedding features in the absolute 3D space. The embedded features at this location are normalized by the layer and added to the flattened visual feature sequence to perform absolute spatial domain alignment of the image visual features; 23) Continuous 3D rotational position encoding injection module. This module uses a multi-layer Transformer encoder-based architecture to process the attention calculation within the coordinate-driven 3D self-supervised prediction module. Continuous 3D relative position encoding is introduced into the query vector and key vector of the attention mechanism to force the network to learn Euclidean relative distance geometric prior. 24) Global geometric consistency self-attention calculation and feature output module. This module performs attention interaction between the position embedding features calculated by the nonlinear projection module of the absolute coordinates to the feature space and the features added to the visual feature sequence. The network spontaneously establishes a geometric correspondence across perspectives and outputs globally consistent global memory features.

4. The system as described in claim 1, characterized in that, The object-centered 3D instance module specifically includes the following sub-modules: 31) Three-dimensional state detector generation unit module. This module initializes several three-dimensional state detectors in the global three-dimensional space. Each detector contains a high-dimensional semantic state vector and a three-dimensional reference point physical coordinate. Before that, the global memory features generated by the multi-view space fusion module are decoded into a multi-scale semantic feature pyramid using an upsampling decoding network. 32) Multi-scale feature sampling and interaction unit module. This module projects the detector's three-dimensional reference points back to the two-dimensional camera view planes using the camera intrinsic and extrinsic parameter matrices. During the projection process, a depth truncation normalization mechanism is introduced. The projected two-dimensional coordinates are used as the sampling center point. The detector's semantic state vector dynamically predicts the spatial two-dimensional offset and attention weights. The corresponding object texture and semantic information are extracted from the multi-scale semantic feature pyramid through a deformable cross-attention mechanism. 33) Occlusion perception gating unit module. This module introduces an adaptive view validity gating mechanism. It uses a multilayer perceptron to perform nonlinear evaluation on features extracted from a specific viewpoint, and combines temperature parameters and activation functions to generate gating weights to characterize the confidence of the features at that viewpoint. The system uses normalized gating weights to perform residual weighted summation on the features of all viewpoints, automatically filtering out low-quality camera viewpoint inputs, thereby updating the semantic state vector of the detector. 34) Bounded coordinate refinement iterative update module: This module uses the updated semantic state vector to predict the compensation vector of the three-dimensional position; it applies boundary constraints to the calculation of the compensation vector, performs nonlinear truncation through the hyperbolic tangent activation function, and calculates the bounded coordinate refinement offset by combining the preset maximum physical step scaling factor, and performs safe iterative update of the coordinates of the detector's three-dimensional reference point.

5. The system as described in claim 1, characterized in that, The coordinate-driven 3D self-supervised prediction module specifically includes the following sub-modules: 41) Target coordinate sampling unit module: This module is used for the mask coordinate set for self-supervised distillation training, and for synthesizing the complete 3D coordinate set of invisible geometric structures during the inference phase. During the training phase, an asymmetric multi-view masking strategy forces the network to infer missing structures using cross-view geometric priors rather than spatial interpolation. Unmasked regions form visible context features and their corresponding visible coordinate points. During the inference phase, masking operations are turned off, and the predictor directly uses the complete coordinates to query in continuous space. 42) Hybrid attention unit module with position and viewpoint conditions. This module designs a three-dimensional feature regression predictor for inferring unobserved features. This predictor needs to encode absolute physical coordinates and specific camera viewpoint information at the same time. This unit injects the continuous three-dimensional absolute spatial position encoding and discrete camera viewpoint embedding features in parallel into the visible context features and the target query vector sequence, respectively. By separating identity feature injection and relative distance encoding, hybrid attention operations with 3D relative position encoding are used in cross-attention and self-attention layers to extract and aggregate 3D information across space. 43) The module for preventing collapse distillation of the teacher network and outputting the 3D complete feature vector by exponential moving average: This module constructs a teacher network with the same structure as the predictor, but the parameters of this teacher network do not participate in gradient backpropagation. Instead, during the training process, the weights of the main network are slowly and smoothly updated by using the exponential moving average mechanism. At the same time, after continuous feature regression processing by the multi-layer position and viewpoint conditional attention unit module, the predictor outputs the 3D complete feature vector corresponding to the target coordinates.

6. The system as described in claim 1, characterized in that, The spatial condition geometry aggregation module and the motion expert decoder module specifically include the following sub-modules: 51) Spatial Conditional Coordinate Transformation Unit Module: This module receives the reference point coordinates of the 3D instance feature vector and the spatial coordinates corresponding to the 3D completion feature vector, and simultaneously acquires the 3D spatial pose of the robot end effector in the absolute world coordinate system in real time; through coordinate subtraction and coordinate system rotation transformation, it calculates the 3D relative physical offset of each object instance and geometric completion point relative to the robot end effector; then, it uses a spatial encoding function to perform high-dimensional mapping on the 3D relative physical offset to generate continuous relative spatial position encoding, and transforms it into interactive availability embedding features through a linear layer; The interactive availability embedding features are used as relative geometric priors and are added element-wise to the corresponding 3D instance feature vector and 3D completion feature vector, so that both carry physical distance information centered on the robot end effector. 52) Instance and geometric completion aggregation module under spatial conditions: This module constructs a multimodal feature aggregation layer, introduces an attention mechanism including a distance space penalty term and a gated routing mechanism based on uncertainty evaluation, and dynamically weights and fuses the 3D instance feature vectors injected with relative spatial location information with the 3D completion feature vectors for information complementarity; This module combines the originally discrete instance semantics with the local completion geometric structure to output 3D instance-geometric aggregation features; 53) The hybrid prompt input module of the motion expert decoder adopts a parameter architecture independent of the visual backbone network; it splices and integrates the 3D instance-geometric aggregation features, the textual semantic features of language instructions and the image visual features on the feature channel to form a hybrid context prompt sequence with complete 3D entity cognition and spatial perception. 54) Action sequence trajectory generation strategy module based on flow matching architecture. In this module, the motion expert decoder, under the global guidance of the hybrid context cue sequence, processes the received state features and a noise vector representing multiple consecutive time steps of action sequence in the future through a one-way attention mechanism; through a multi-layer attention mechanism, it deeply captures the complete three-dimensional geometry of the target object, the relative spatial pose of the end effector and the target, and the physical mapping relationship between the specific operation intention; after a flow field integration derivation process of a set number of steps, the model restores the initial random distribution to a high-fidelity robot motion trajectory distribution, and finally outputs a set of low-level continuous robot action command sequences containing a given time span in the future. The command sequence covers the joint pose of the robotic arm and the opening and closing state parameters of the end effector.

7. The system as described in claim 1, characterized in that, The vision-language-action model is optimized during its training phase based on a self-supervised signal and action imitation multi-task learning loss system, employing a two-stage progressive training process, specifically as follows: The first stage is pre-training for spatial fusion and instance extraction. This stage focuses on learning physical entity features with strict viewpoint consistency, and jointly training a multi-view spatial fusion module, an object-centered 3D instance module, and an action expert decoder. The second stage involves training for 3D completion and geometric aggregation refinement. Upon entering the second stage, the parameters of the perception module that were trained in the first stage are forcibly frozen; on this basis, the coordinate-driven 3D self-supervised prediction module and the spatial conditional geometry aggregation module are trained, and the action expert decoder is continuously updated.

Images