A somatic intelligent robot, a robot motion control system and method

Abstract

The application discloses a body-integrated intelligent robot, a robot motion control system and a method, the body-integrated intelligent robot is provided with an execution component, a moving module and a robot motion control system, the robot motion control system comprises: a multi-modal input encoder, which is used for collecting and processing multi-modal data to obtain multi-modal features and fusing to obtain a multi-modal feature token sequence; an action expert network, which is used for mapping the fused multi-modal feature token sequence into an abstract robot action sequence block; and an execution controller, which is used for converting the abstract robot action sequence block into a control instruction executable by bottom-layer hardware. The application can realize direct mapping from environmental perception, semantic understanding to mobile control and execution of specified task actions, thereby improving task understanding, response speed and execution completion degree of the robot in a complex environment.

Description

Technical Field

[0001] This invention relates to the field of multimodal artificial intelligence control technology, specifically to an embodied intelligent robot, a robot motion intelligent control system based on a vision-language-action (VLA) model applied to a mobile embodied intelligent robot, and its control method. Background Technology

[0002] Embossed intelligent robots are interdisciplinary products that integrate machine vision, natural language processing, robotics and machine learning. They rely on a closed-loop architecture of "perception-decision-action" to achieve interaction in the physical world. However, traditional control technologies still face bottlenecks such as insufficient generalization ability, high complexity and low degree of autonomy.

[0003] In terms of motion control, traditional robot control methods are mostly based on trajectory planning and feedback control, which can achieve stable operation in regular, low-dynamic environments, but they are difficult to meet the requirements of embodied robots for real-time performance, flexibility, and generalization in unstructured environments. For example, when dealing with multi-degree-of-freedom cooperative movements, existing control systems often suffer from problems such as unreasonable decomposition of control signals and unnatural motion transitions, thereby reducing the smoothness and safety of robot task execution.

[0004] Furthermore, the lack of an efficient coordination mechanism between existing robot action decisions and low-level control leads to a disconnect between upper-level strategies and lower-level execution, resulting in response delays or energy waste. At the same time, the action generation process is mostly based on single-modal or weakly fused perceptual inputs, lacking full utilization of multimodal information, and often fails to achieve human-like natural performance in complex tasks (such as human-robot collaboration, dynamic obstacle avoidance, and manipulation tasks).

[0005] In recent years, the development of multimodal large models has promoted the unified integration of perception, reasoning, and action generation. Visual-Language-Action (VLA) models, through end-to-end learning, directly map visual, sensor data, and language instructions into low-level control commands, demonstrating strong generalization and transfer capabilities. However, a complete technical system for embodied intelligent robot task execution is still lacking. Systematic solutions covering key aspects such as efficient multimodal data fusion, task-driven action generation, and real-time closed-loop control have not yet been formed.

[0006] To address this, this application proposes an embodied intelligent robot, an embodied robot motion intelligent control system and method based on the VLA model, which, while ensuring low energy consumption and high response speed, can achieve efficient perception and fusion of multimodal inputs, task-driven motion generation, and stable closed-loop control, thereby breaking through the bottlenecks of existing technologies and improving the operational capability and adaptability of embodied intelligent robots in complex physical environments, thus solving the aforementioned technical problems. Summary of the Invention

[0007] The main objective of this invention is to provide an embodied intelligent robot, a robot motion control system, and a method. By using end-to-end multimodal perception and motion prediction, and omitting the explicit path planning module, it achieves direct mapping from environmental perception and semantic understanding to motion control and execution of specified task actions. This improves the robot's task understanding, response speed, and execution completion in complex environments, thereby solving the technical problems mentioned in the background art.

[0008] The present invention solves the above-mentioned technical problems by adopting the following technical solutions: An embodied intelligent robot has an execution component for performing a specified motion task and a motion module for controlling movement. It also has a sensor component for collecting and receiving information and a robot motion control system for controlling the operation of the execution component and the motion module based on the received information.

[0009] Preferably, a robot motion control system, installed on the embodied intelligent robot, comprises a multimodal input encoder, a motion expert network, and an execution controller, including: The multimodal input encoder is used to acquire and process multimodal data from sensor components and user task commands to obtain multimodal features and fuse them to obtain a multimodal feature token sequence: The motion expert network, deployed on the inference module within the system controller's computing unit, employs a Transformer architecture to map the fused multimodal feature token sequence into abstract robot motion sequence blocks. The execution controller is used to convert the abstract robot motion sequence blocks output by the motion expert network into control instructions that can be executed by the underlying hardware. Preferably, the multimodal input encoder includes: A visual acquisition module used to acquire RGB images and depth information of the environment; A lidar module used to collect three-dimensional spatial structure information of the environment; A language input module for receiving natural language task instructions from users; A state sensing module used to acquire robot pose, speed, and running status in real time.

[0010] Preferably, the multimodal input encoder further includes: The server computing unit is used to perform time synchronization and feature alignment processing within the computing unit after receiving the collected multimodal data through the bus interface; The visual encoder, depth information processor, language encoder, and state encoder are used to extract visual features, depth information features, language features, and state features respectively, and encode them into a multimodal feature token sequence, which is then fed into the action expert network for decision-making.

[0011] Preferably, the action expert network includes: an input layer, an output layer, and multiple alternating layers of cross-attention and self-attention; After the multimodal feature token sequence is input through the input layer, the self-attention layer models the temporal and contextual dependencies within a single modality, and then the interaction and information fusion between multimodal features are realized through the cross-attention layer. After multiple iterations, a unified representation is obtained, and finally the output layer generates a unified action sequence that includes chassis motion and the specified task action execution device.

[0012] Wherein: the cross-attention layer is used for cross-attention, and the self-attention layer uses a causal attention mask to ensure that each action token can only pay attention to historical tokens within the block, avoiding future action dependencies.

[0013] Preferably, after the server computing unit performs time synchronization and feature alignment processing, for a specified length of The temporal sequence of the visual encoder, depth information processor, language encoder, and state encoder is used to map multimodal data to a unified dimension. ,have:

[0014] in, This is a data feature matrix mapped to a unified dimension. , , , These represent image, speech, IMU, and radar data, respectively. Used to represent data mapping operation matrices express dimensional real number field, Indicates the dimension of the data. Indicates time; Add learnable positional encoding Modal identification coding ,have:

[0015] in, Let be the new data feature matrix at time t; The final unified encoded multimodal feature token sequence for:

[0016] in, Indicates a token representing past actions. A vector representation of the first position in the sequence.

[0017] Preferably, in the output layer and the multiple alternating cross-attention layers, gating cross-attention interacts with the action context, specifically including: (1) Calculate key-value pairs: Pre-trained vision-language model, used to generate hidden states for current / historical observations, has ,in, This represents the output representation of the visual-language input sequence after passing through a VLM encoder, and the key is obtained through linear head computation. AND value ,have:

[0018]

[0019] in, , The projection matrix representing the key and the projection matrix representing the value; (2) Computational motion experts and cross-attention: in the first Layer, motion experts get the query When performing multi-head cross-attention on memory, we have:

[0020] in, For confidence level or time decay mask, This is a contextual representation, that is, cross-modal information aggregated through an attention mechanism. To represent the dimension of the key vector, To represent the attention weight matrix, To represent the spatial dimension of the matrix, it is an L×N real matrix, where L is the number of tokens queried in the current layer of the action expert network and N is the number of multimodal tokens output by the VLM; By employing gated fusion and residual planning, we have:

[0021] in, For the first Layer Action Expert Token Sequence The weight matrix is ​​a learnable matrix. It is a non-linear activation function. For the first l Hidden intermediate states in the layer This indicates that layer normalization is applied to avoid gradient explosion or vanishing. This is element-wise multiplication, used for gating or feature selection: (3) Action experts in the token sequence Perform causal self-attention, so that , , Construct an upper triangular invisible mask ,have:

[0022] in, Represented as the first in the attention mask , One element; Further calculation of attention is as follows:

[0023] in, This is represented as the output of the attention mechanism. It is represented as a key vector.

[0024] Preferably, the motion expert network employs an asynchronous reasoning mechanism, which uses the results of the previous reasoning cycle to maintain the continuous operation of the robot while generating new motion sequences.

[0025] Preferably, the execution controller includes: The motion control unit is used to convert linear velocity and angular velocity commands into control signals to drive the motion module and control the robot's movement. The execution control unit executes the start and stop of components and the drive operations, including grasping, rotating, and positioning, of designated task targets according to control signals; The state feedback unit is used to collect state feedback information in real time and return it to the multimodal input encoder and motion expert network for motion correction and the next round of inference input.

[0026] A robot motion control method, implemented based on any of the aforementioned robot motion control systems, includes: The sensor components collect RGB image information, 3D spatial information, voice commands, and the robot's current position, speed, and status information. The robot motion control system processes the collected information data to obtain control signals for the operation of the right and wrong control execution components and the moving module. The execution control unit drives the execution components and the moving module to perform corresponding operations based on control signals.

[0027] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0028] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0029] As can be seen from the above technical solution, the present invention provides an embodied intelligent robot, a robot motion control system, and a method. Compared with the prior art, the present invention has the following advantages: 1. This invention achieves direct mapping from environmental perception and semantic understanding to motion control and execution of specified tasks through end-to-end multimodal perception and action prediction, omitting the explicit path planning module. This improves the robot's task understanding, response speed, and execution completion in complex environments.

[0030] 2. Through the real-time perception of the multimodal input encoder, the end-to-end decision-making and asynchronous reasoning mechanism of the motion expert network, and the closed-loop execution control of the execution controller, the embodied intelligent robot can achieve efficient and stable autonomous task execution in complex dynamic environments.

[0031] 3. This invention combines attention-based multimodal feature fusion technology with causal mask-driven long-term modeling method, which can effectively overcome the perception blind spots and action delays caused by a single sensor in traditional operations, and significantly improve the target detection accuracy and action response speed in complex scenes.

[0032] It should be understood that the descriptions in this section are not intended to identify key or essential features of embodiments of the invention, nor are they intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Of course, implementing any product of the invention does not necessarily require achieving all of the advantages described above simultaneously. Attached Figure Description

[0033] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the system data processing flow of the present invention; Figure 2 This is a structural example diagram of a cleaning robot in an embodiment of the present invention; Figure 3 This is a schematic diagram of the Transform architecture of the motion expert network of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] In this embodiment, a cleaning robot is used as an example; see details below. Figures 1 to 3 .

[0036] like Figure 1 As shown. The embodied intelligent robot, robot motion control system, and method proposed in this embodiment of the invention specifically refer to: an embodied cleaning robot, a cleaning robot motion control system, and a method, wherein: A. A holographic cleaning robot, which has cleaning components for cleaning designated targets and a motion module for controlling movement, and is also equipped with sensor components for collecting and receiving information and a cleaning robot motion control system for controlling the operation of the cleaning components and the motion module based on the received information.

[0037] B. The cleaning robot motion control system, installed on the embodied cleaning robot, consists of a multimodal input encoder, a motion expert network, and an execution controller. The three interact through data and control signals to achieve end-to-end mapping from perception to action, omitting the explicit path planning module. Specifically, through end-to-end multimodal perception and motion prediction, the explicit path planning module is omitted, achieving direct mapping from environmental perception and semantic understanding to motion control and cleaning actions. This improves the robot's task understanding, response speed, and execution completion in complex environments, thereby driving the corresponding drive components of the embodied cleaning robot to perform corresponding operations.

[0038] C. A motion control method for a cleaning robot, implemented based on the aforementioned motion control system for a cleaning robot, including: The sensor components collect RGB image information, 3D spatial information, voice commands, and the robot's current position, speed, and status information. The cleaning robot's motion control system processes the collected information data to obtain control signals for the operation of the cleaning components and the moving module, thus controlling whether the cleaning is correct or not. The cleaning control unit drives the cleaning components and the moving module to perform corresponding operations based on control signals.

[0039] Furthermore, the core technical architecture of the aforementioned motion control system for the cleaning robot comprises three main components: a multimodal input encoder, a motion expert network, and an execution controller, such as... Figure 3 As shown.

[0040] In one specific embodiment, the embodied cleaning robot employs, for example... Figure 2 The structure shown is such that: (1) The multimodal input encoder is responsible for processing data from heterogeneous sensors and user commands, performing unified processing and encoding. The specific data processing method can be adjusted according to the robot's configuration. For example, the visual encoder can use a pre-trained visual-language model (such as SigLIP) to encode RGB images into visual feature tokens; the depth information processor can use radar point cloud data to convert it into a bird's-eye view (BEV) representation and then extract features or directly tokenize it to retain three-dimensional spatial information, or it can use the RGB-D information of the depth camera; the language encoder directly encodes the user's natural language commands into language feature tokens; and the state encoder encodes the robot's current position, speed, state, and other information into state tokens. The obtained results are provided to the motion expert module.

[0041] Furthermore, in the specific implementation process, the multimodal input encoder specifically includes: a binocular camera installed at the front of the robot, used as a visual acquisition unit to acquire RGB images and depth information of the environment; a LiDAR installed around the chassis, used as a LiDAR unit to acquire three-dimensional spatial structure information of the environment; a language input unit for receiving natural language task commands from the user; and a state sensing unit for acquiring the robot's pose, speed, and running status in real time, which specifically includes an inertial measurement unit (IMU), an encoder, and a collision sensor installed at the center of the robot.

[0042] The data collected by the aforementioned sensors is transmitted to the server computing unit via a bus interface. Within the computing unit, time synchronization and feature alignment are performed. Features are then extracted by the visual encoder, depth information processor, language encoder, and state encoder, and uniformly encoded into a multimodal feature token sequence, which is then sent to the action expert module for decision-making.

[0043] (2) The motion expert network, deployed on the inference module within the system controller computing unit, adopts a Transformer architecture to generate future robot motion sequence blocks based on multimodal information. This module receives the fused multimodal token sequence as input and processes the multimodal information through alternating cross-attention (CA layer) and self-attention (SA layer) mechanisms, directly outputting a unified motion sequence containing chassis motion (linear velocity, angular velocity) and cleaning device control. The CA layer cross-attention is responsible for cross-attention (i.e., the interaction between VLM key-value pairs and motion features), while the SA layer allows motion tokens in the motion expert network to self-attention with each other. At the same time, the SA layer uses a causal attention mask to ensure that each motion token can only pay attention to historical tokens within the block, avoiding future motion dependencies. This structure eliminates the need for explicit path planning or intermediate representations, achieving end-to-end mapping of perceived motion.

[0044] Based on the Transformer architecture, the input is a sequence of multimodal tokens. First, a self-attention layer (SA layer) models the temporal and contextual dependencies within a single modality. Then, a cross-attention layer (CA layer) enables the interaction and information fusion between multimodal features. After multiple iterations, the model obtains a unified representation, and the output layer generates action sequences, thus achieving end-to-end decision-making driven by multimodal perception. Specific steps include: L1. Input Given a length The temporal sequence and multimodal observations (image / speech / IMU / radar, etc.) are mapped to a unified dimension through their respective encoders. :

[0045] in, This is a data feature matrix mapped to a unified dimension. , , , These represent image, speech, IMU, and radar data, respectively. Used to represent data mapping operation matrices express dimensional real number field, Indicates the dimension of the data. Indicates time; And incorporate learnable positional coding Modal identification coding :

[0046] in, Let be the new data feature matrix at time t; The input sequence for the motion expert is:

[0047] in Indicates a token representing past actions. A vector representation of the first position in the sequence.

[0048] L2. Calculate key-value pairs The pre-trained visual-language model generates hidden states based on current / historical observations. .in, This represents the output representation of the visual-language input sequence after passing through a VLM encoder, and the key is obtained through linear head computation. AND value ,have:

[0049]

[0050] in, , The projection matrix representing the key and the projection matrix representing the value are usually... .

[0051] L3. Computational Motion Expert and Cross Attention In the Layer, motion experts get the query When performing multi-head cross-attention on memory, we have:

[0052] in, For confidence level or time decay mask, This is a contextual representation, that is, cross-modal information aggregated through an attention mechanism. To represent the dimension of the key vector, To represent the attention weight matrix, To represent the spatial dimension of the matrix, it is an L×N real matrix, where L is the number of tokens queried in the current layer of the action expert network and N is the number of multimodal tokens output by the VLM; By employing gated fusion and residual planning, we have:

[0053] in, For the first Layer Action Expert Token Sequence The weight matrix is ​​a learnable matrix. It is a non-linear activation function. For the first lHidden intermediate states in the layer This indicates that layer normalization is applied to avoid gradient explosion or vanishing. This is element-wise multiplication, used for gating or feature selection. L4. Calculate the causal mask in self-attention. Action experts in the token sequence Perform causal self-attention to ensure that you only see the "past and present" and do not reveal future actions.

[0054] make , , Construct an upper triangular "invisible" mask ,have:

[0055] in, Represented as the first in the attention mask , One element; Further calculation of attention is as follows:

[0056] in, This is represented as the output of the attention mechanism. It is represented as a key vector.

[0057] At this point, gating cross-attention interacts with the action context; causal masking ensures the autoregressiveness and temporal causal legitimacy of action generation; the combination of the two enables multimodal knowledge and temporal control to be coupled in a closed loop within a unified Transformer for end-to-end robot decision-making.

[0058] Furthermore, it's important to note that to improve system response speed and execution smoothness, motion expert networks employ an asynchronous inference mechanism. This means that while generating new action sequences, the execution controller can use the results of the previous inference cycle to maintain continuous robot operation, thereby reducing the waiting latency between perception, inference, and execution, and achieving relatively continuous and smooth motion control. Specifically, at each time step, the robot client can receive new observations and send them to the policy server for inference. After performing inference, the policy server generates new action blocks and returns them to the robot client. The robot client can still continue executing the current action block while waiting for new action blocks, avoiding complete pauses and idle time.

[0059] In one specific embodiment, an example of data fitting during the asynchronous inference execution process is as follows:

[0060] At t=0, the motion expert network begins to generate predicted action blocks (right turn + maintain speed), and the execution controller immediately issues instructions. The robot has already executed the action while waiting for the inference results of the next cycle, ensuring continuous operation.

[0061] At each time step, sensor feedback (obstacle distance, yaw angle, channel width, etc.) is sent back to the motion expert network for subsequent motion optimization.

[0062] Abstract actions (such as "turn 5° to the right and maintain speed") are mapped by the controller to low-level motor speeds / robotic arm joint angles, realizing the mapping from high-level decision-making to physical execution.

[0063] (3) The execution controller is used to receive the abstract motion sequence output by the motion expert network and convert it into specific control signals of the chassis drive motor and the upper limb robotic arm. At the same time, it collects real-time sensor feedback and transmits it back to the multimodal input module and the motion expert network to support closed-loop optimization and online strategy update.

[0064] In summary, through the real-time perception of the multimodal input encoder, the end-to-end decision-making and asynchronous reasoning mechanism of the motion expert network, and the closed-loop execution control of the execution controller, the cleaning robot can achieve efficient and stable autonomous cleaning operations in complex dynamic environments.

[0065] Therefore, this application combines attention-based multimodal feature fusion technology with causal mask-driven long-term modeling method to effectively overcome the perception blind spots and action delays caused by a single sensor in traditional operations, significantly improving target detection accuracy and action response speed in complex scenarios. At the same time, by combining asynchronous reasoning and feedback adaptive control technology, it ensures that the robot can maintain continuous and smooth execution even in the presence of communication delays or large computational overhead, thus possessing higher robustness and task completion efficiency.

[0066] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the method described above.

[0067] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the method described above.

[0068] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the embodied intelligent robots, robot motion control systems, and methods described above.

[0069] It is understood that the system provided in the embodiments of the present invention corresponds to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0070] This application also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other via the communication bus. Memory, used to store computer programs; The processor, when executing the program stored in the memory, implements the aforementioned embodied intelligent robot, robot motion control system, and method.

[0071] The communication bus mentioned in the above-mentioned electronic devices can be a standard bus for interconnecting peripheral components or an extended industrial standard structure bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0072] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0073] The memory may include random access memory or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0074] The processors mentioned above can be general-purpose processors, including central processing units, network processors, etc.; they can also be digital signal processors, application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0075] It should also be noted that electronic devices include terminal devices, which can also be called terminals, user equipment, mobile stations, mobile terminals, etc. Terminal devices can be mobile phones, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality terminal devices, augmented reality terminal devices, wireless terminals in industrial control, wireless terminals in autonomous driving, wireless terminals in remote surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, and so on. The embodiments of this application do not limit the specific technologies or device forms used in the terminal devices.

[0076] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, an optical medium, or a semiconductor medium, etc.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0078] Furthermore, it should be noted that if any directional indication (such as up, down, left, right, front, back, etc.) is involved in the embodiments of the present invention, the directional indication is only used to explain the relative positional relationship and movement of each component in a specific posture. If the specific posture changes, the directional indication will also change accordingly.

[0079] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the meaning of "and / or" throughout the text includes three parallel solutions; for example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied simultaneously. Furthermore, in the embodiments of this invention, "multiple" refers to two or more. Moreover, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

Claims

1. A robot motion control system, installed on an embodied intelligent robot, the embodied intelligent robot having an actuator for performing a specified motion task and a motion module for controlling movement, and further having a sensor assembly for collecting and receiving information and a robot motion control system for controlling the operation of the actuator and the motion module based on the received information, the robot motion control system comprising a multimodal input encoder, a motion expert network, and an actuator controller, characterized in that, include: The multimodal input encoder is used to acquire and process multimodal data from sensor components and user task commands to obtain multimodal features and fuse them to obtain a multimodal feature token sequence: The motion expert network, deployed on the inference module within the system controller's computing unit, employs a Transformer architecture to map the fused multimodal feature token sequence into abstract robot motion sequence blocks. The execution controller is used to convert the abstract robot motion sequence blocks output by the motion expert network into control instructions that can be executed by the underlying hardware. The multimodal input encoder includes: The server computing unit is used to perform time synchronization and feature alignment processing within the computing unit after receiving the collected multimodal data through the bus interface; The visual encoder, depth information processor, language encoder, and state encoder are used to extract visual features, depth information features, language features, and state features respectively, and encode them into a multimodal feature token sequence, which is then fed into the action expert network for decision-making. After the server computing unit performs time synchronization and feature alignment processing, for a specified length of The temporal sequence of the visual encoder, depth information processor, language encoder, and state encoder is used to map multimodal data to a unified dimension. ,have: in, Map all sensor data to a data feature matrix with a unified dimension. , , , These represent the image, speech, IMU, and radar data at time t, respectively. Used to represent data mapping operation matrices express dimensional real number field, Indicates the dimension of the data. Indicates time; Add learnable positional encoding Modal identification coding ,have: in, Let be the new data feature matrix at time t; The final unified encoded multimodal feature token sequence for: in, Indicates a token representing past actions. A vector representation of the first position in the sequence.

2. The robot motion control system as described in claim 1, characterized in that, The multimodal input encoder also includes: A visual acquisition module used to acquire RGB images and depth information of the environment; A lidar module used to collect three-dimensional spatial structure information of the environment; A language input module for receiving natural language task instructions from users; A state sensing module used to acquire robot pose, speed, and running status in real time.

3. The robot motion control system as described in claim 2, characterized in that, The motion expert network includes: an input layer, an output layer, and multiple alternating layers of cross-attention and self-attention; After the multimodal feature token sequence is input through the input layer, the self-attention layer models the temporal and contextual dependencies within a single modality, and then the interaction and information fusion between multimodal features are realized through the cross-attention layer. After multiple iterations, a unified representation is obtained, and finally the output layer generates a unified action sequence that includes chassis motion and specified task action execution device. Wherein: the cross-attention layer is used for cross-attention, and the self-attention layer uses a causal attention mask to ensure that each action token can only pay attention to historical tokens within the block, avoiding future action dependencies.

4. The robot motion control system as described in claim 3, characterized in that, In the output layer and multiple alternating cross-attention layers, interaction with the action context is achieved through gated cross-attention, specifically including: (1) Calculate key-value pairs: Pre-trained vision-language model, used to generate hidden states for current / historical observations, has ,in, This represents the output representation of the visual-language input sequence after passing through a VLM encoder, and the key is obtained through linear head computation. AND value ,have: in, , The projection matrix representing the key and the projection matrix representing the value; (2) Computational motion experts and cross-attention: in the first Layer, motion experts get the query When performing multi-head cross-attention on memory, we have: in, For confidence level or time decay mask, This is a contextual representation, that is, cross-modal information aggregated through an attention mechanism. To represent the dimension of the key vector, To represent the attention weight matrix, To represent the spatial dimension of the matrix, it is an L×N real matrix, where L is the number of tokens queried in the current layer of the action expert network and N is the number of multimodal tokens output by the VLM; By employing gated fusion and residual planning, we have: in, For the first Layer Action Expert Token Sequence The weight matrix is ​​a learnable matrix. It is a non-linear activation function. For the first l The intermediate hidden state in the layer, This indicates that layer normalization is applied to avoid gradient explosion or vanishing. This is element-wise multiplication, used for gating or feature selection: (3) Action experts in the token sequence Perform causal self-attention, so that , , Construct an upper triangular invisible mask ,have: in, Represented as the first in the attention mask , One element; Further calculation of attention is as follows: in, This is represented as the output of the attention mechanism. It is represented as a key vector.

5. The robot motion control system as described in claim 3, characterized in that, The motion expert network employs an asynchronous reasoning mechanism, which uses the results of the previous reasoning cycle to maintain the robot's continuous operation while generating new motion sequences.

6. The robot motion control system as described in claim 1, characterized in that, The execution controller includes: The motion control unit is used to convert linear velocity and angular velocity commands into control signals to drive the motion module and control the robot's movement. The execution control unit executes the start and stop of components and the drive operations, including grasping, rotating, and positioning, of designated task targets according to control signals; The state feedback unit is used to collect state feedback information in real time and return it to the multimodal input encoder and motion expert network for motion correction and the next round of inference input.

7. A robot motion control method, characterized in that, Based on the robot motion control system as described in any one of claims 1-6, including: The sensor components collect RGB image information, 3D spatial information, voice commands, and the robot's current position, speed, and status information. The robot motion control system processes the collected information data to obtain control signals for the operation of the right and wrong control execution components and the moving module. The execution control unit of the execution controller drives the execution components and the moving module to perform corresponding operations based on control signals.

Images