Control method and system for modular intelligent robot based on neural network

Abstract

The application relates to a control method and system of a modular intelligent robot based on a neural network. The method comprises the following steps: querying an active node, obtaining a hardware configuration list, loading an end-to-end neural network model corresponding to each independent hardware based on the hardware configuration list, and obtaining a neural network model instance; converging original sensor data streams of the modular intelligent robot, and performing time synchronization alignment on the sensor data streams to obtain a multi-modal perception data packet; inputting the multi-modal perception data packet into an encoder of the end-to-end neural network model instance to obtain a high-dimensional feature vector; inputting the high-dimensional feature vector into a decision network of the end-to-end neural network model instance to obtain an action instruction set; and solving the action instruction set into a motion trajectory to obtain a smooth motion trajectory. The method can support plug-and-play of modules and flexible replacement of algorithm nodes, and can flexibly and independently control independent modules of the modular intelligent robot.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control, and in particular relates to a control method and system for a modular intelligent robot based on neural networks. Background Technology

[0002] With the rapid development of embodied intelligent robot technology, high-performance humanoid robot platforms have emerged. These platforms typically have highly integrated sensors and actuators, enabling them to complete complex motion and operational tasks, thus becoming the mainstream choice for universities and research institutions to verify algorithms.

[0003] In the traditional technological approach, research institutions or learners who need to conduct research related to embodied intelligence typically purchase such complete machine platforms directly. The supplier provides the complete hardware system and the underlying control interface. After obtaining the robot, the user needs to start from debugging the underlying drivers and setting up the communication environment, and gradually work upwards to kinematic modeling, controller design, and finally implement and verify their own advanced intelligent algorithms.

[0004] However, current methods of developing complete system platforms, or traditional entry-level approaches, have several significant drawbacks. First, integrated design leads to extremely high hardware costs, far exceeding the budgets of typical laboratories and individual learners. Second, in pursuit of versatility, complete systems often integrate redundant functions that users do not need, resulting in wasted resources and high maintenance costs. More importantly, the long development chain from the hardware level to algorithm verification creates a steep learning curve, requiring researchers to spend months on engineering implementation rather than focusing on core algorithm innovation. Furthermore, fixed system architectures are difficult to match with the rapid iteration and flexible combination required for cutting-edge AI algorithm research. Summary of the Invention

[0005] Therefore, it is necessary to provide a control method and system for a modular intelligent robot based on neural networks that can independently control each module of the modular intelligent robot, supports plug-and-play functionality of modules and flexible replacement of algorithm nodes, and adapts to the needs of rapid algorithm iteration and verification, in order to address the aforementioned technical problems.

[0006] Firstly, this application provides a control method for a modular intelligent robot based on a neural network, comprising:

[0007] Based on ROS2 asynchronous communication, active nodes are queried to obtain a hardware configuration list. Based on the hardware configuration list, the end-to-end neural network model corresponding to each independent hardware is loaded to obtain a neural network model instance.

[0008] The raw sensor data streams from the various independent hardware components of the modular intelligent robot are aggregated, and the sensor data streams are time-synchronized and aligned to obtain a multimodal perception data package.

[0009] The multimodal sensing data packets are input into the encoder of the end-to-end neural network model instance to perform state feature encoding and obtain high-dimensional feature vectors.

[0010] The high-dimensional feature vector is input into the decision network of the end-to-end neural network model instance to make action policy decisions and obtain the action instruction set;

[0011] The motion instruction set is solved into a motion trajectory to obtain a smooth motion trajectory; the smooth motion trajectory is used to instruct each actuator of each independent hardware to adjust its operating parameters.

[0012] Furthermore, the end-to-end neural network model is constructed using the following method;

[0013] The precise 3D model and hardware parameter data of the modular intelligent robot are obtained, and a high-fidelity simulation environment is constructed based on the precise 3D model and hardware parameter data; the high-fidelity simulation environment includes a virtual robot model and a training scene;

[0014] Based on a high-fidelity simulation environment, a reinforcement learning framework is used to generate policy actions and record them as actions. After the virtual robot model executes the policy actions, the state data of the virtual robot model and the state data of the training scene are recorded as states, thus obtaining a simulation interaction experience buffer.

[0015] Based on the simulation interaction experience buffer, with the goal of maximizing the expected value of accumulated rewards, the end-to-end neural network framework is iteratively trained to obtain the neural network policy model.

[0016] The neural network policy model is subjected to domain randomization reinforcement training to obtain a generalized neural network policy model.

[0017] The generalized neural network strategy model is lightweighted to obtain an end-to-end neural network model.

[0018] Furthermore, the end-to-end neural network model can be constructed using the following methods:

[0019] Based on motion capture data playback or optimal control algorithms, the robot motion trajectory executed by the virtual robot model in a high-fidelity simulation environment is recorded, and the state data of the virtual robot model and the state data of the training scene are also recorded to obtain a demonstration dataset.

[0020] Based on the demonstration dataset, with the goal of minimizing the loss function value, the end-to-end neural network framework is iteratively trained to obtain the neural network policy model.

[0021] The neural network policy model is subjected to domain randomization reinforcement training to obtain a generalized neural network policy model.

[0022] The generalized neural network strategy model is lightweighted to obtain an end-to-end neural network model.

[0023] Furthermore, the neural network policy model is subjected to domain randomization reinforcement training to obtain a generalized neural network policy model, including:

[0024] By setting random variation ranges for each configuration parameter in the high-fidelity simulation environment, a domain randomization configuration table is obtained.

[0025] Based on the domain randomized configuration table, a set of random configuration parameters are sampled and injected into the high-fidelity simulation environment to obtain a simulation environment instance;

[0026] Based on the simulation interaction experience buffer or demonstration dataset, the neural network policy model is iteratively trained through simulation environment instances to obtain an updated neural network policy model.

[0027] Based on the validation environment set, the performance of the updated neural network policy model is evaluated to obtain the generalized neural network policy model.

[0028] Furthermore, the formula for calculating the expected value of accumulated rewards is as follows:

[0029]

[0030] in, To accumulate the expected value of rewards, For mathematical expectation, The state at time step t, As a strategy, It is the instant reward for the k-th step in the future. This is the discount factor.

[0031] Furthermore, the end-to-end neural network models corresponding to each independent hardware component in the hardware configuration list are loaded to obtain neural network model instances, including:

[0032] Based on the hardware configuration list, the encoding of each independent hardware component is parsed to obtain the hardware configuration query key;

[0033] Based on the hardware configuration query key, a matching search is performed in the pre-trained model library to obtain candidate pre-trained model files;

[0034] Based on the hardware configuration list, an adaptability check is performed on the candidate pre-trained model files to obtain the adaptability check results;

[0035] Based on the results of the adaptive check, candidate pre-trained model files are configured to obtain the initialized neural network;

[0036] The input and output ports of the initialized neural network are bound to the original sensor data stream to obtain a neural network model instance.

[0037] Secondly, this application also provides a control system for a modular intelligent robot based on a neural network, comprising:

[0038] The instance module is used to query active nodes based on ROS2 asynchronous communication, obtain a hardware configuration list, and load the end-to-end neural network model corresponding to each independent hardware based on the hardware configuration list to obtain a neural network model instance.

[0039] The synchronization module is used to aggregate the raw sensor data streams from the various independent hardware components of the modular intelligent robot and to perform time synchronization and alignment of the sensor data streams to obtain multimodal perception data packets.

[0040] The feature module is used to input multimodal sensing data packets into the encoder of the end-to-end neural network model instance to perform state feature encoding and obtain high-dimensional feature vectors.

[0041] The decision module is used to input high-dimensional feature vectors into the decision network of the end-to-end neural network model instance to make action strategy decisions and obtain action instruction sets;

[0042] The motion module is used to convert the set of motion instructions into motion trajectories to obtain smooth motion trajectories; the smooth motion trajectories are used to instruct the actuators of each independent hardware to adjust their operating parameters.

[0043] Thirdly, this application also provides a computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement any step of the method provided in the first aspect of this application.

[0044] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any step of the method provided in the first aspect of this application.

[0045] The aforementioned control method and system for a modular intelligent robot based on neural networks utilizes ROS2 asynchronous communication to query active nodes, obtain a hardware configuration list, and load the end-to-end neural network model corresponding to each independent hardware component based on the hardware configuration list, thus obtaining a neural network model instance. It aggregates the raw sensor data streams from each independent hardware component of the modular intelligent robot and performs time synchronization and alignment on the sensor data streams to obtain a multimodal perception data packet. The multimodal perception data packet is input into the encoder of the end-to-end neural network model instance for state feature encoding, resulting in a high-dimensional feature vector. The high-dimensional feature vector is then input into the decision network of the end-to-end neural network model instance for action strategy decision-making, resulting in an action instruction set. The action instruction set is then decomposed into a motion trajectory, resulting in a smooth motion trajectory. The smooth motion trajectory is used to instruct the actuators of each independent hardware component to adjust their operating parameters. The robot's control model can be broken down into standardized functional modules that can be freely disassembled and combined. These modules can be purchased and combined as needed, fundamentally avoiding the need to pay for redundant functions and significantly reducing the initial purchase cost and subsequent iteration and upgrade costs. ROS2 is used to achieve communication decoupling, supporting plug-and-play modules and flexible replacement of algorithm nodes. End-to-end learning is used as the core algorithm framework, enabling it to be co-designed with modular hardware. This naturally adapts to the rapid iteration and verification requirements of cutting-edge algorithms, effectively improving the control accuracy and model iteration speed of modular robots. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 A schematic diagram of the flow of a control method for a modular intelligent robot based on a neural network, provided in an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of the structure of a control system for a modular intelligent robot based on a neural network, provided in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] In one embodiment, such as Figure 1As shown, a control method for a modular intelligent robot based on a neural network is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the terminal can be a computing unit of independent hardware within the modular intelligent robot, or it can be an independent terminal. In this embodiment, the method includes the following steps:

[0051] Step 101: Based on ROS2 asynchronous communication, query active nodes to obtain the hardware configuration list, and based on the hardware configuration list, load the end-to-end neural network model corresponding to each independent hardware to obtain a neural network model instance.

[0052] ROS2 asynchronous communication (Robot Operating System 2) is a non-blocking message passing mechanism. Under this mechanism, the sending node can continue to execute subsequent tasks without waiting for confirmation from the receiving node after sending data, improving responsiveness and resource utilization.

[0053] An active node refers to a software process that is currently running and participating in communication in ROS2. Each node is typically responsible for a specific function, such as driving a sensor or an actuator.

[0054] A hardware configuration manifest is a structured list or configuration file that describes information about all connected and operational individual hardware units in a current modular intelligent robot system. This manifest typically includes a unique identifier for each hardware component, its model number, communication interface, and the identifier of its corresponding pre-trained model.

[0055] An end-to-end neural network model is a deep learning model that directly maps from raw input to output. It is a parameter file of a neural network trained for specific hardware or hardware combinations to perform a specific task.

[0056] A neural network model instance refers to a software object that, after loading an end-to-end neural network model file stored on disk into the terminal memory and completing initialization, can be directly invoked for inference. It includes the model's structure, weights, and runtime context.

[0057] The terminal actively discovers all running nodes in the current network through the services or tools provided by ROS2. It communicates with active nodes to obtain detailed attribute information of the hardware modules represented by each node, summarizes and organizes this information to generate a complete hardware configuration list. The terminal reads the hardware configuration list, searches and matches it against a pre-built model repository based on the identification information of each hardware component, finds the corresponding end-to-end neural network model file, loads the model file into memory using a deep learning framework, and initializes it, thereby creating a runnable neural network model instance for each hardware component or hardware combination.

[0058] Step 102: Aggregate the raw sensor data streams of each independent hardware component of the modular intelligent robot, and perform time synchronization and alignment of the sensor data streams to obtain a multimodal perception data packet.

[0059] Raw sensor data streams refer to the unprocessed sequence of measurement data continuously and in real-time acquired from the robot's individual sensor hardware. Raw sensor data streams exist in the form of digital signals or simple codes.

[0060] Time synchronization alignment is a data processing technique that aims to adjust data from different sensors, which have different timestamps due to slight differences in the time of acquisition, to the same unified time reference, so that the data are logically regarded as being observed at the same time.

[0061] A multimodal sensing data packet is a time-aligned, structured collection of data. It encapsulates synchronized data from multiple types of sensors at the same time window or point in time, forming a comprehensive and consistent snapshot of the robot's own state and its surrounding environment.

[0062] The terminal uses ROS2's topic subscription mechanism to simultaneously listen for and receive raw sensor data streams published by all sensor nodes. Each received data frame is appended with a precise timestamp. A time synchronization algorithm is used for processing. The terminal can select a master clock and align the data from all other sensors to integer multiples of that master clock's time points using interpolation or nearest neighbor lookup. The various sensor data aligned to the same time point are then assembled into a complete multimodal sensing data packet according to a predefined structure.

[0063] Step 103: Input the multimodal sensing data packet into the encoder of the end-to-end neural network model instance to perform state feature encoding and obtain a high-dimensional feature vector.

[0064] Specifically, an encoder is a sub-network or component in an end-to-end neural network model, consisting of structures such as convolutional neural networks, recurrent neural networks, or transformers, and is specifically responsible for extracting low-dimensional, abstract, and information-rich feature representations from high-dimensional, redundant raw input data.

[0065] State feature encoding refers to the computational process performed by the encoder on the input multimodal sensing data packets. This process aims to understand and compress the raw data, transforming it into abstract features that can effectively characterize the robot's current state and the state of its external environment.

[0066] A high-dimensional feature vector is a fixed-length numerical vector. Although its dimensionality may still be high, it is much more concise and abstract compared to the original sensor data. High-dimensional feature vectors condense all the state information crucial for decision-making at the current moment.

[0067] The terminal performs necessary preprocessing on the multimodal sensing data packets to conform to the encoder's input format requirements. The processed data packets are then fed into the encoder portion of the loaded neural network model instance. The encoder network performs a series of nonlinear transformations on the input data according to its pre-trained weights and structure. These transformations may include convolution operations to extract spatial features, pooling operations to reduce dimensionality, or self-attention mechanisms to capture long-range dependencies. After layers of computation, in the final layer of the encoder, the original multimodal data is transformed and compressed into a high-dimensional feature vector representing the current integrated state.

[0068] Step 104: Input the high-dimensional feature vector into the decision network of the end-to-end neural network model instance to make action policy decisions and obtain the action instruction set.

[0069] Specifically, the decision network is another sub-network or component in an end-to-end neural network model, located after the encoder. It consists of structures such as fully connected layers, and its function is to calculate the optimal or suboptimal action to be performed in the current state based on the state features provided by the encoder.

[0070] Action policy decision-making refers to the process by which a decision network selects an action or an action probability distribution from all possible action spaces based on the input state characteristics through internal computation. This process embodies the policy learned by the model.

[0071] A motion instruction set is a specific set of low-level control commands used to directly drive the robot's actuators. For example, for a wheeled robot, it might be a set of target velocities for the left and right wheels; for a robotic arm, it might be target angles or torques for each joint.

[0072] The terminal takes a high-dimensional feature vector as input and passes it to the decision network part of the neural network model instance. The decision network performs further calculations on the feature vector. Depending on the task, the network's output layer may output a specific action value or the probability of each action. Based on the output of the decision network, the terminal parses and generates the corresponding action instruction set. For example, if the network outputs the target angle of a joint, then the set of angles is the action instruction set; if the output is an action probability, it may be necessary to determine the action to be executed by sampling or taking the maximum value, and then convert it into a specific instruction.

[0073] Step 105: The motion instruction set is solved into a motion trajectory to obtain a smooth motion trajectory; wherein, the smooth motion trajectory is used to instruct each actuator of each independent hardware to adjust the operating parameters.

[0074] Solving refers to the process of converting an abstract, discrete, or target-point-based set of motion instructions into a continuous, time-varying, physically realizable sequence of motion paths and postures through mathematical models and algorithms.

[0075] A motion trajectory refers to a continuous function or discrete sequence of motion parameters such as position, attitude, velocity, and acceleration of a robot or its actuator in space that change over time.

[0076] A smooth motion trajectory refers to the motion trajectory obtained after optimization through a solution process. It has characteristics such as continuity, differentiability, and no abrupt changes, which can avoid impact, vibration, or loss of synchronization of the actuator, ensuring smooth and accurate motion.

[0077] An actuator is a device in a robot that is responsible for converting electrical signals or control commands into physical motion, such as a servo motor, a stepper motor, or a hydraulic cylinder.

[0078] Operating parameters refer to the specific control signals that directly drive the actuator, such as the duty cycle of pulse width modulation, target current, target position / speed / torque, etc.

[0079] The terminal is based on a set of motion instructions. Since the instructions may be discrete-time targets, directly sending them to the actuators would result in discontinuous motion. Therefore, trajectory planning is required between these instruction points. For example, polynomial interpolation, trapezoidal velocity planning, or more advanced algorithms can be used to generate a smooth motion trajectory where position, velocity, and acceleration change continuously from the current state to the target state. For complex machines, the generated smooth end-effector trajectory also needs to be decomposed into independent trajectories for each joint through inverse kinematics calculations. Dynamics also need to be considered, including torque feedforward calculations. The trajectories of the joints or drive wheels are converted into operating parameters that each actuator can understand, resulting in a smooth motion trajectory. Optionally, the target angle can be converted into the number of pulses in the motor encoder, or the target velocity can be converted into a current command in the motor driver.

[0080] The control method for a modular intelligent robot based on neural networks provided in this embodiment utilizes ROS2 asynchronous communication to query active nodes, obtain a hardware configuration list, and load the end-to-end neural network model corresponding to each independent hardware component based on the hardware configuration list to obtain a neural network model instance. It aggregates the raw sensor data streams from each independent hardware component of the modular intelligent robot and performs time synchronization and alignment on the sensor data streams to obtain a multimodal perception data packet. The multimodal perception data packet is input into the encoder of the end-to-end neural network model instance for state feature encoding to obtain a high-dimensional feature vector. The high-dimensional feature vector is input into the decision network of the end-to-end neural network model instance for action strategy decision-making to obtain an action instruction set. The action instruction set is then calculated into a motion trajectory to obtain a smooth motion trajectory. The smooth motion trajectory is used to instruct the actuators of each independent hardware component to adjust their operating parameters. Through the above methods, the control model of intelligent robots can be broken down into standardized functional modules that can be freely disassembled and combined. These modules can be purchased and combined as needed, fundamentally avoiding the need to pay for redundant functions and significantly reducing the initial purchase cost and subsequent iteration and upgrade costs. ROS2 is used to achieve communication decoupling, supporting plug-and-play modules and flexible replacement of algorithm nodes. End-to-end learning is used as the core algorithm framework, enabling it to be co-designed with modular hardware. This naturally adapts to the rapid iteration and verification requirements of cutting-edge algorithms, effectively improving the control accuracy and model iteration speed of modular robots.

[0081] In one embodiment, the end-to-end neural network model is constructed using the following method;

[0082] Step 201: Obtain the precise 3D model and hardware parameter data of the modular intelligent robot, and construct a high-fidelity simulation environment based on the precise 3D model and hardware parameter data; wherein, the high-fidelity simulation environment includes a virtual robot model and a training scene.

[0083] Specifically, a precise 3D model refers to a digital model created in a computer that perfectly matches the geometry and structural dimensions of the physical robot. It defines the shape, assembly relationships, and kinematic chains of each module of the robot.

[0084] Hardware parameter data refers to data describing the physical characteristics of a robot, mainly including actuator parameters, such as the torque-speed curve of a motor and the gear reduction ratio; sensor parameters, such as the noise model of an IMU, the intrinsic parameters and distortion coefficients of a camera; and mass attributes, such as the mass of each component, the position of the center of mass, the inertia tensor, and the friction coefficient and damping coefficient of the joints.

[0085] A high-fidelity simulation environment is a virtual computing environment that highly simulates the physical laws and sensory information of the real world. Driven by a physics engine, it can calculate the dynamic effects of collisions, friction, gravity, and other forces between virtual objects.

[0086] A virtual robot model is a digital avatar defined by a precise 3D model and hardware parameter data, capable of controlled movement in a simulation environment. It possesses physical properties such as mass, inertia, and joint constraints.

[0087] A training scenario is a virtual world constructed for a specific training task, which includes terrain, obstacles, target objects, lighting conditions, and other possible environmental elements.

[0088] The terminal exports a precise 3D model of the modular robot from its design files. Simultaneously, it collects and organizes all necessary hardware parameter data from hardware specifications, actual measurements, or system identification experiments. Within the selected simulation platform, the precise 3D model file is imported, and the hardware parameter data is configured onto the model one by one, including setting the mass, inertia matrix, and friction coefficient for each joint; setting the force / torque limits and transmission ratio for the motor model; and setting the focal length and resolution for the camera model. In the simulation platform, built-in tools are used to build training scenes, including setting the ground physical materials, placing and configuring various obstacles and target objects, defining lighting and textures, and simulating disturbances such as wind and water flow. The configured virtual robot model is then placed into the training scene, forming a complete, interactive, high-fidelity simulation environment.

[0089] Step 202: Based on the high-fidelity simulation environment, a policy action is generated through a reinforcement learning framework, and the policy action is recorded as an action. After the virtual robot model executes the policy action, the state data of the virtual robot model and the state data of the training scene are recorded as states to obtain the simulation interaction experience buffer.

[0090] Reinforcement learning frameworks are a set of algorithms and tools used to train agents to learn optimal behavioral policies through interaction with their environment. Their core elements include the agent, environment, state, action, and reward.

[0091] The policy action is the suggested action output by the neural network policy model being trained, based on the currently observed state. In the early stages of training, this action is usually random or suboptimal.

[0092] State data is a collection of all information at a specific moment that describes the virtual robot model itself and the state of the training scene. It typically includes the robot's joint angles, angular velocities, body posture, speed, and information such as the relative positions and distances of relevant targets in the scene.

[0093] The simulation interaction experience buffer is a data structure used to store training data. Each data entry records a complete interaction experience, typically stored as a tuple (state, action, reward, next state, whether to end).

[0094] The terminal initiates a reinforcement learning training loop within a pre-constructed high-fidelity simulation environment. The neural network policy model to be trained is used as the agent. At each time step, the agent receives current state data from the simulation environment and calculates a policy action accordingly. This action instruction is then sent to the virtual robot model in the simulation environment for execution. The simulation physics engine calculates the changes in the virtual robot model and the training scene after the action is executed, generating new state data. Based on a preset reward function, the immediate reward obtained in this step is calculated. The experience of this interaction is packaged into a tuple, including the state before the action, the executed policy action, the reward obtained, the new state after the action, and a flag indicating whether the round has ended. This tuple is stored in the simulation interaction experience buffer.

[0095] Step 203: Based on the simulation interaction experience buffer, with the goal of maximizing the expected value of accumulated rewards, the end-to-end neural network framework is iteratively trained to obtain the neural network policy model.

[0096] Among them, the expected value of accumulated rewards is the core optimization objective in reinforcement learning. It refers to the mathematical expectation of the sum of all discounted rewards that an agent can obtain from a certain state, following the current strategy and interacting with the environment.

[0097] An end-to-end neural network framework refers to the specific neural network architecture and its accompanying training algorithm used to build and train policy models, defining the mapping relationship from state input to action output.

[0098] A neural network policy model is a neural network whose parameters have converged after iterative training. It achieves a stable mapping from observed states to decision actions and represents the behavioral policy learned by the agent.

[0099] The terminal randomly samples a small batch of historical interaction experience data from the simulation interaction experience buffer. Based on the specific reinforcement learning algorithm used, it calculates the loss function of the current policy using the sampled data. For example, in the actor-critic algorithm, the advantage function is calculated, and the policy loss and value loss are constructed. The gradient of the loss function with respect to the neural network parameters is calculated using the backpropagation algorithm, and the optimizer is used to update the network parameters along the gradient descent direction to maximize the expected value of the accumulated reward. This process is repeated, continuously updating the buffer with new interaction experience and sampling from the buffer to update the network, iterating multiple times until the policy performance tends to stabilize and reach its optimum. The resulting network is the trained neural network policy model.

[0100] Step 204: Perform domain randomization reinforcement training on the neural network policy model to obtain a generalized neural network policy model.

[0101] Specifically, domain randomization is a training technique designed to improve a model's ability to transfer from simulation environments to the real world. Its core principle is to actively and randomly change various parameters of the simulation environment during simulation training, forcing the policy model to learn to adapt to these changes rather than relying excessively on a fixed simulation setting.

[0102] Domain randomized reinforcement training is a training process that introduces domain randomization techniques on top of the standard reinforcement learning training loop. Before each training round or each step, a new set of simulation parameters is randomly sampled from a preset range.

[0103] Generalized neural network policy models are policy models obtained after domain randomization reinforcement training. They are not sensitive to subtle changes in simulation parameters and have stronger robustness and adaptability. Therefore, they are more likely to work directly in the real physical world that is unseen and differs from the simulation.

[0104] The terminal determines the simulation parameters and their range of variation that need to be randomized during training. Common parameters include dynamic parameters, appearance and perception parameters, and environmental parameters. A randomization step is added to the training loop. At the beginning of each interactive loop, a new set of parameter values ​​is sampled uniformly or according to a certain distribution from the above parameter space and dynamically injected into the current high-fidelity simulation environment instance. The neural network policy model continues to interact and learn in this constantly changing simulation environment. The model must learn to complete the target task under diverse physical and perception conditions, thereby learning the core rules of the task itself. The policy performance is periodically tested on another set of fixed validation environments different from the training environment. When the model can stably complete the task in multiple randomized environments and performs well on the validation set, the training ends, and a generalized neural network policy model is obtained.

[0105] Step 205: Lightweight the generalized neural network strategy model to obtain an end-to-end neural network model.

[0106] Specifically, lightweight processing is a collective term for a series of techniques aimed at reducing the computational cost, parameter size, and memory footprint of neural network models in order to improve their inference speed on resource-constrained edge devices. The core objective is to reduce the demand for computing resources while maintaining model performance as much as possible.

[0107] An end-to-end neural network model refers to a neural network model file that has been trained through simulation, domain randomization, and lightweighting, and can be deployed and run on actual robot hardware.

[0108] The terminal analyzes the structure and parameters of the generalized neural network strategy model and selects appropriate lightweighting techniques. These include identifying and removing unimportant connections or entire neurons to generate a sparser, smaller network; converting network weights and activation values ​​from high-precision floating-point numbers to low-precision formats, significantly reducing model storage space and accelerating computation; using a large, high-performance teacher model to guide the training of a small student model, allowing the student model to learn the teacher model's behavior and achieve near-large model performance with a small model; and redesigning or searching for a lighter network structure using more efficient network modules and transferring knowledge from the original model. Selected lightweighting operations are performed, for example, iterative pruning and fine-tuning, or training the model to simulate quantization effects to maintain accuracy and post-training quantization, and fine-tuning the lightweight model in a simulation environment using a small amount of data to recover some accuracy. The performance and inference speed of the lightweight model are rigorously tested on a validation set to ensure it meets the requirements of actual deployment. The processed model, meeting performance and resource requirements, is converted into a format suitable for the target deployment framework, resulting in an end-to-end neural network model file for model loading and instantiation.

[0109] In one embodiment, the end-to-end neural network model may be constructed using the following method:

[0110] Step 301: Based on motion capture data playback or optimal control algorithm, record the robot motion trajectory executed by the virtual robot model in a high-fidelity simulation environment, and record the state data of the virtual robot model and the state data of the training scene to obtain the demonstration dataset.

[0111] Among them, motion capture data playback refers to the process of importing pre-recorded three-dimensional spatial coordinate sequence data about the movement of a body or object from the real world or other sources into a simulation environment, and driving the joints of a virtual robot model to reproduce the series of movements.

[0112] Optimal control algorithms are a class of mathematical methods used to solve for the optimal behavioral strategy of a dynamic system. In this embodiment, it refers to calculating a series of theoretically optimal joint control command sequences in a simulation environment, based on a known robot dynamics model and a clearly defined task objective.

[0113] A robot's motion trajectory refers to a series of precise joint angles, angular velocities, or torques generated by motion capture data playback or optimal control algorithms, driving a virtual robot model to complete a specific task. It represents the expert or optimal sequence of actions for completing the task.

[0114] The state data of a virtual robot model is the set of physical quantities it exhibits at every moment as it moves along the robot's trajectory. This typically includes the robot's position, orientation, linear velocity, angular velocity, and the angles, angular velocities, and forces acting on all its joints.

[0115] The state data of the training scene is a collection of task-related environmental information during the movement of the virtual robot model. This includes the position of target objects, the distance to obstacles, and the inclination of the ground.

[0116] The demonstration dataset is a collection of multiple state-action pairs. Each pair records the state at a specific moment, including the state data of the virtual robot model and the state data of the training scene, as well as the action that should be executed in that state, derived from the expert trajectory—that is, the corresponding control command in the robot's motion trajectory. The demonstration dataset serves as a training sample library for subsequent imitation learning.

[0117] If motion capture data playback is used, the terminal imports the pre-collected and processed motion capture data file into the simulation environment, and maps the data onto the skeletal joints of the virtual robot model through motion redirection technology to drive its movement, thereby obtaining a robot motion trajectory.

[0118] If an optimal control algorithm is used, the terminal sets explicit task objectives and constraints in the simulation environment and runs the algorithm. Based on the dynamics of the current model and scene, the algorithm calculates a series of optimal control commands from the starting point to the target point, forming a robot motion trajectory.

[0119] In a high-fidelity simulation environment, the terminal controls the virtual robot model to move strictly according to the aforementioned robot motion trajectory. The simulation engine runs synchronously at a high frequency, accurately calculating the physical effects of each step. At each time step of the playback, the terminal synchronously records the current environmental observations, namely the state data of the virtual robot model and the state data of the training scene, which together constitute the state at that moment; and the specific control commands, i.e., actions, being executed at that moment from the expert trajectory. The terminal organizes and denoises all the state-action pairs recorded throughout the playback process, and stores them in chronological order or randomly shuffled to form a structured demonstration dataset. The demonstration dataset describes what actions the expert took in what state.

[0120] Step 302: Based on the demonstration dataset, the end-to-end neural network framework is iteratively trained with the goal of minimizing the loss function value to obtain the neural network policy model.

[0121] Specifically, in machine learning, the loss function is used to quantify the difference between the model's predicted output and the true target value. The smaller the loss function, the more accurate the model's prediction. The loss function measures the difference between the action predicted by the neural network and the action recorded by experts in a demonstration dataset.

[0122] An end-to-end neural network framework refers to a neural network architecture used for imitation learning. Its input is a state, and its output is an action. An end-to-end neural network framework can be structurally similar to the framework in step 203, but the training objectives and data sources differ.

[0123] Neural network policy models attempt to mimic expert behavior by learning the mapping relationship between states and actions in a sample dataset, thereby acquiring the basic ability to complete specific tasks.

[0124] The terminal divides the demonstration dataset into training and validation sets. An initial model for the end-to-end neural network framework is prepared. A batch of state data is extracted from the dataset and input into the neural network. The network outputs its predicted actions, which are then compared to the corresponding actions of the real expert in the dataset. The loss value of the current batch of predictions is calculated based on the selected loss function. The loss value represents the degree of inaccuracy in the network's imitation of the expert. Using the backpropagation algorithm, the terminal calculates the gradient of the loss value with respect to all parameters of the neural network. The gradient indicates how to fine-tune the parameters to reduce the loss value. The network parameters are updated using an optimization algorithm along the reverse direction of the gradient. This process is repeated, training the network multiple times using a large number of samples from the demonstration dataset. Each training round makes the network's predicted actions closer to the expert's actions, i.e., the loss function value continuously decreases. The terminal periodically evaluates the model performance on the validation set. Training stops when the loss on the validation set no longer decreases significantly, or when the model successfully reproduces expert behavior in a simulation environment. The resulting model is the neural network policy model.

[0125] Step 303: Perform domain randomization reinforcement training on the neural network policy model to obtain a generalized neural network policy model.

[0126] Specifically, domain randomized reinforcement training is an advanced imitation learning fine-tuning process that incorporates domain randomization techniques. Its core is to continue using expert demonstration data to train and refine the policy model under constantly changing simulation environment parameters, but the training method is specifically designed for dynamic changes.

[0127] The generalized neural network policy model is a policy model obtained after domain randomization reinforcement training. Based on mimicking the basic skills of experts, it further enhances its robustness and generalization ability by dealing with various perturbations and changes in the randomized environment, enabling it to handle complex situations beyond the scope of the original demonstration data.

[0128] The terminal sets the environmental parameters to be randomized and their range of variation. At the beginning of each training cycle, it samples a new set of environmental physical and visual parameters from a preset random distribution and generates a dynamically generated randomized environment instance based on these parameters. The terminal randomly samples a batch of state-expert action pairs from the expert demonstration dataset. For each sampled expert state, it performs state alignment using the physics engine of the current randomized environment instance. The joint configuration recorded by the expert in that state is placed in the current randomized physics environment, and zero control is applied or the configuration is held for a moment. Then, the actual observed state fed back by the simulation environment is read. The new observed state and the original expert action constitute an aligned training sample. The aligned new state is input into the current neural network policy model, and the model outputs a predicted action. The difference between the predicted action and the original expert action is calculated. To address the possibility that the model's own policy may enter state regions not covered by expert data when interacting with the environment, the current policy can be run in the current randomized environment for a period of time to collect a series of new states. For these new states, the expert-level actions to be performed in these new states are obtained by running an optimal control algorithm once or by backpropagating through the dynamic model, thus generating new state-action pairs and supplementing them into the training data. The gradient of the loss function with respect to the model parameters is calculated using the backpropagation algorithm, and the model parameters are updated using the optimizer. The goal of the update is to ensure that the model outputs actions consistent with expert demonstrations when faced with aligned states generated by any randomized environment. This process is repeated iteratively, exposing the model to a vast array of different dynamic environments. At the end of training, the model learns to strip away the state appearance differences caused by changes in physical parameters, capturing the invariant intent related to the task objective in the expert's behavior. This results in a generalized neural network policy model that can effectively mimic expert behavior even under unseen physical conditions.

[0129] Step 304: Lightweight the generalized neural network strategy model to obtain an end-to-end neural network model.

[0130] Lightweight processing is a series of techniques designed to reduce the complexity, size, and computational requirements of neural network models so that they can be efficiently deployed on resource-constrained embedded hardware or mobile processors. The core objective is to improve inference speed and reduce power consumption and memory usage while minimizing performance loss.

[0131] An end-to-end neural network model refers to a neural network model file that has undergone initial training through imitation learning, fine-tuning through domain randomization imitation learning to achieve strong generalization ability, and optimization through lightweight processing, and can be directly deployed on real modular robot hardware.

[0132] The terminal analyzes the generalized neural network strategy model, evaluating its computational graph structure, number of parameters, computation time of each layer, and memory usage. Its benchmark performance is tested on a reserved validation environment set as a comparison standard before and after lightweighting. Based on the model characteristics and deployment platform, one or more lightweighting techniques are selected and applied sequentially. After applying operations such as pruning or quantization, model performance typically decreases slightly. The lightweight model is fine-tuned using a small amount of data in a domain-randomized simulation environment to adapt it to its new, simplified structure or numerical precision, restoring performance under diverse conditions as much as possible. The fine-tuned lightweight model is tested in a randomized validation environment to ensure that its generalization ability and task success rate do not decrease significantly, while inference speed meets real-time requirements. The model that meets the requirements is converted to a format supported by the target deployment framework to obtain an end-to-end neural network model.

[0133] In one embodiment, the neural network policy model is subjected to domain randomization reinforcement training to obtain a generalized neural network policy model, including:

[0134] Step 401: Set random variation ranges for each configuration parameter in the high-fidelity simulation environment to obtain a domain randomization configuration table.

[0135] Among them, the high-fidelity simulation environment is a virtual computing platform that can highly simulate the physical laws and sensory information of the real world, and its behavior is controlled by a large number of configurable parameters.

[0136] Each configuration parameter refers to all adjustable variables that define the specific attributes of the simulation environment. The configuration parameters cover a wide range of dynamic attributes, such as the mass of the robot body, inertia tensor, joint friction and damping, actuator force / torque limit and delay, and ground friction coefficient; perception attributes such as camera focal length, distortion, image noise, lidar ranging noise and offset; and scene attributes such as terrain height field, obstacle size and position, illumination angle and intensity, and object surface texture.

[0137] The random variation range is a numerical range specified for each configuration parameter, defining the boundary by which the parameter can be randomly adjusted during training.

[0138] A domain randomization configuration table is a structured list or configuration file that lists all parameters that need to be randomized and specifies the range of random variations for each parameter. It serves as a blueprint for performing domain randomization.

[0139] The terminal acquires all configurable parameters in a high-fidelity simulation environment that may affect the robot's behavior, perception, or task-related aspects. Based on prior knowledge of real-world uncertainties or experimental testing, a reasonable range of random variation is defined for each selected parameter, covering as many real-world variations as possible. The configuration parameters and their corresponding ranges are compiled into a clear domain randomization configuration table. This table will be used to generate the specific randomized environment in each subsequent training step.

[0140] Step 402: Based on the domain randomized configuration table, sample a set of random configuration parameters and inject the random configuration parameters into the high-fidelity simulation environment to obtain a simulation environment instance.

[0141] Specifically, sampling a set of random configuration parameters means independently and randomly generating a specific value for all listed parameters based on the statistical distribution defined for each parameter in the domain randomization configuration table.

[0142] Injection refers to dynamically setting or applying the specific parameter values ​​obtained from sampling to the underlying engine of a high-fidelity simulation environment, thereby instantly changing the physical and visual properties of that environment.

[0143] A simulation environment instance is a specific simulation environment initialized at a particular moment, assigned a set of specific parameter values ​​sampled from a domain randomized configuration table. Each sampling and injection generates an instance with unique parameter configuration, whose physical rules and visual appearance differ from other instances.

[0144] Each time a new training environment needs to be created, the terminal reads the domain randomization configuration table. For each parameter in the table, a specific value is generated by a random number generator based on its specified distribution and range. For example, the sampled values ​​might be a ground friction coefficient of 0.8, a left leg mass of 1.1 times its nominal value, and a camera Gaussian noise standard deviation of 2 pixels. The terminal then starts or resets a basic high-fidelity simulation environment and sets all the specific parameter values ​​obtained in the previous step into the environment one by one through the application programming interface provided by the simulation engine. After the parameters are injected and take effect, a simulation environment instance with unique physical and visual characteristics is ready. The virtual robot model will be placed in this instance, ready for interaction or training.

[0145] Step 403: Based on the simulation interaction experience buffer or demonstration dataset, the neural network policy model is iteratively trained through simulation environment instances to obtain an updated neural network policy model.

[0146] Specifically, the simulation interaction experience buffer is a dataset that stores historical interaction data. Each data entry typically includes a state, action, reward, next state, and end flag. It originates from the agent's experience of autonomously exploring and learning through trial and error in a simulation environment.

[0147] The demonstration dataset is a collection of state-action pairs generated by expert demonstrations to teach the model specific skills.

[0148] Neural network policy models are deep learning models that are to be trained or optimized, mapping environmental states to actions.

[0149] Updating a neural network policy model refers to obtaining a new policy model version with better expected performance after one or more rounds of parameter optimization based on specific data sources and objectives in the current simulation environment instance.

[0150] Step 403 is divided into two core paths due to the different data sources and training methods it relies on.

[0151] In this model, based on the simulation interaction experience buffer, the current neural network policy model is deployed to a simulation environment instance. The model interacts with the environment, selects actions according to its policy, and obtains new state and reward signals after execution. The experience of this interaction is stored in the simulation interaction experience buffer. The terminal randomly samples a small batch of historical experience data from the simulation interaction experience buffer and uses a reinforcement learning algorithm to calculate the policy gradient or value loss based on the sampled experience data. The core objective is to maximize the expected cumulative reward in the future. Through backpropagation, the parameters of the neural network policy model are updated along the direction of increasing the expected reward. The steps are repeated, with each training iteration performed in a new simulation environment instance with randomized parameters. By trying different instances, the model learns a robust policy that can obtain high rewards under various parameter configurations.

[0152] In this approach, a batch of expert-demonstrated state-action pairs are sampled from the demonstration dataset. Since the demonstration dataset is typically recorded under a set of nominal environmental parameters, while the parameters of the current simulation instance are randomized, the original state observations in the expert data may not match the physical reality of the current instance. For example, the joint configurations in the expert states are substituted into the dynamics of the current instance to calculate the corresponding true state observations under the current random physics. These aligned new states are then used to form new training sample pairs with the original expert actions. The terminal inputs the aligned states into a neural network policy model, which outputs predicted actions. The difference between the predicted actions and the expert actions is calculated, with the core objective being to minimize this difference. The model parameters are updated along the direction of reducing imitation error using a backpropagation algorithm. This forces the model to learn that regardless of how randomly the environmental physical parameters change, when a state corresponding to the expert's intention is observed, the expert-demonstrated action should be output. As the model is updated, its behavior may deviate from the expert and enter a state region not covered in the dataset. In this case, the current model is run in randomized instances to collect new states. The actions for the new states are recalculated using the optimal control algorithm, and the new state-action pairs are added to the training set to continuously improve the model.

[0153] Step 404: Based on the validation environment set, evaluate the performance of the updated neural network policy model to obtain the generalized neural network policy model.

[0154] The validation environment set is a predefined set of simulation environments with fixed parameters. These environments are typically designed to be more challenging and diverse than the training environment, and their parameter configurations have never appeared during training. They are used to objectively evaluate the model's performance under unseen conditions.

[0155] Model performance evaluation involves running the updated neural network policy model for multiple rounds in each environment of the validation environment set and recording quantitative results for a series of performance metrics.

[0156] A generalized neural network policy model refers to an updated neural network policy model that performs exceptionally well on a validation environment set, meaning that all performance metrics meet or exceed preset thresholds. It is considered to have successfully passed the generalization ability test and can adapt to new environmental conditions beyond what was seen during training, hence the name "generalized model."

[0157] The terminal selects a set of fixed-parameter validation environments. These environments cover a variety of reasonable challenges the task might encounter. The updated neural network policy model is deployed and run in each validation environment. Multiple rounds with random seeds are run in each validation environment to obtain stable performance statistics. The terminal collects all run results, calculates the mean and standard deviation of preset performance metrics, and compares the metrics to a success threshold. If the model's evaluation results on the vast majority of validation environments meet the requirements, the model is considered to have good generalization ability and is confirmed as the final generalized neural network policy model. If the requirements are not met, the parameter range of the domain randomization configuration table, the training algorithm, or more training rounds are performed for re-evaluation.

[0158] In one embodiment, the formula for calculating the expected value of the accumulated reward is:

[0159]

[0160] in, To accumulate the expected value of rewards, For mathematical expectation, For the state at time step t, As a strategy, It is the instant reward for the k-th step in the future. This is the discount factor.

[0161] Specifically, the state value function represents the expected sum of cumulative rewards that an agent can obtain starting from a specific state under a given policy, quantifying the long-term value of that state.

[0162] The mathematical expectation represents the average of the results of the expression within parentheses over all possible trajectories in which the agent continuously interacts with the environment while following the policy. It considers the randomness of the environmental dynamics and the potential randomness of the policy itself.

[0163] The state at time step t refers to the complete description of the environment and itself as perceived by the agent at time t, and is the basis for the agent to make decisions.

[0164] The action at time step t refers to the operation that the agent selects and executes based on the current state and policy at time t.

[0165] A policy is the behavioral guideline for an agent. It is a mapping function from the state space to the action space, which determines what action the agent should take in any given state.

[0166] The immediate reward at the k-th future step refers to the single, immediate benefit or cost signal fed back by the environment after the agent performs an action, starting from the current time t, at the k-th future time step. It is the environment's evaluation of the agent's single-step behavior.

[0167] The discount factor is a constant between 0 and 1, used to exponentially decay future rewards, giving higher weight to near-term rewards and lower weight to long-term rewards.

[0168] It explicitly requires the agent to optimize the sum of all future rewards starting from the current state, rather than just the immediate reward. This guides the agent to make long-term plans, potentially sacrificing short-term gains for greater long-term rewards.

[0169] In one embodiment, the end-to-end neural network model corresponding to each independent hardware component in the hardware configuration list is loaded to obtain a neural network model instance, including:

[0170] Step 601: Based on the hardware configuration list, parse and encode each independent hardware component to obtain the hardware configuration query key.

[0171] The hardware configuration list is a structured list that describes the specific information of all the individual hardware units contained in the currently assembled modular robot. Each record typically includes the type, model, version, unique identifier, and key performance parameters of the hardware.

[0172] Each independent hardware component refers to a detachable physical functional module that makes up the robot as a whole, such as a complete wheel and leg assembly, a robotic arm module with joint motors, or an independent depth camera module.

[0173] Parsing and encoding refers to the process of analyzing and understanding the text or structured information in a hardware configuration list and encoding it into a standardized, machine-readable format.

[0174] The hardware configuration lookup key is a specifically formatted string or hash value that serves as a fingerprint or index uniquely identifying the current complete hardware configuration. It encapsulates the entire robot's modular structure information for rapid retrieval within the model library.

[0175] The terminal obtains a hardware configuration list describing the current robot's structure, iterates through each individual hardware record in the list, extracts key feature information, and converts all non-standard descriptions into predefined standard enumeration values. To ensure that the same hardware combination always generates the same key, the descriptions of all hardware modules need to be sorted according to a fixed rule. All sorted and normalized hardware feature information is then concatenated into a string according to a predetermined format. For efficiency and consistency, a hash value can be calculated for this string as the hardware configuration query key. The hardware configuration query key uniquely represents the current hardware combination.

[0176] Step 602: Based on the hardware configuration query key, perform a matching search in the pre-trained model library to obtain candidate pre-trained model files.

[0177] Specifically, a pre-trained model library is a centrally stored database or file system directory containing a large number of pre-trained end-to-end neural network model files. Each model file is associated with one or more hardware configuration lookup keys, indicating which hardware configuration(s) the model was trained and optimized for.

[0178] Matching retrieval refers to the process of using hardware configuration query keys as search conditions to find related model files in a pre-trained model library.

[0179] Candidate pre-trained model files are one or more neural network model parameter files that meet the requirements, found through matching retrieval. These candidate pre-trained model files are potential intelligent control models that can be loaded for the current robot.

[0180] The terminal sends the hardware configuration query key as a query request to the management interface of the pre-trained model library. The management system of the model library receives the query key, searches in its index, and returns a list of metadata of all model files that match the query key, including file path, version number, training task description, performance score, etc.

[0181] Step 603: Based on the hardware configuration list, perform an adaptability check on the candidate pre-trained model files to obtain the adaptability check results.

[0182] Specifically, adaptability checks refer to a series of automated verifications performed on candidate pre-trained model files to determine whether they can run seamlessly, securely, and efficiently on current hardware configurations. The checks primarily focus on interface compatibility and capability matching.

[0183] The fitness test results are a structured report summarizing the pass / fail status of each test. Results are typically Boolean values ​​or a detailed enumeration of states, and may include specific diagnostic information.

[0184] The terminal reads the accompanying metadata configuration file from the candidate pre-trained model file. This metadata configuration file declares the hardware specifications required for model training. The hardware requirements declared in the model metadata are then compared item by item with the current hardware configuration list.

[0185] Does the model require sensor inputs that are completely consistent with the raw sensor data streams actually provided by the current hardware? Does the model output motion command dimension completely match the number and type of actuators in the current robot? Are the motion command values ​​that the model may output within the safe physical limits of the current hardware actuators to prevent hardware damage? Check whether the tasks claimed to be supported by the model are consistent with the user's intentions. Figure 1The terminal summarizes the conclusions of all checks. If all critical checks pass, the result is fully adapted. If there are non-critical mismatches or situations requiring warnings, the result is conditionally adapted with accompanying notes. If there are critical mismatches, the result is not adapted.

[0186] Step 604: Based on the adaptability check results, configure the candidate pre-trained model file to obtain the initialized neural network.

[0187] Configuration refers to the process of making necessary adjustments or setting parameters for candidate pre-trained model files based on the adaptability check results, so that they can adapt to the current operating environment, including modifying the model structure, adjusting parameters, or setting runtime flags.

[0188] Initializing a neural network refers to the process where, after the candidate pre-trained model file has completed adaptive configuration, it is loaded into memory by the deep learning framework, where weight reloading and computation graph construction are completed, and the software object state is prepared for forward inference.

[0189] The terminal analyzes the adaptability check results. If the result is fully adapted, the model file is loaded directly without modification. If the result is conditionally adapted, the corresponding adaptation operation is performed. For example, if only the image input size is different, an image scaling layer is automatically added to the front end of the model. If the force range of certain joints needs to be scaled, a fixed scaling coefficient matrix is ​​added to the output layer after loading the model, and the configured and adjusted model file is loaded using a deep learning framework. The computational graph structure of the model is instantiated, and the pre-trained weight parameters are populated into the network, creating a neural network object in memory.

[0190] Step 605: Bind the input and output ports of the initialized neural network to the original sensor data stream to obtain a neural network model instance.

[0191] The input port refers to the logical interface for initializing the neural network to receive data. Each port corresponds to a specific tensor in the network's input layer, which is expected to receive data with fixed dimensions and semantics.

[0192] The output port refers to the logical interface through which the neural network sends data. Each port corresponds to a specific tensor in the network's output layer, generating data with fixed dimensions and semantics.

[0193] Raw sensor data streams are sequences of raw data continuously and in real time collected from the robot’s various physical sensors and published in the form of topic messages in systems such as ROS2.

[0194] Binding refers to establishing a defined, real-time data connection channel between the input / output ports of the initialized neural network and the raw sensor data stream and actuator control interface.

[0195] A neural network model instance is a functional entity that transforms from a static object into a running entity capable of processing perceived data and generating control commands in real time, once the neural network has initialized and bound its inputs and outputs to the real data stream. Neural network model instances are core components in the intelligent control loop of robots.

[0196] Based on its hardware configuration, the terminal determines which ROS2 topic each input port should subscribe to, and which ROS2 topic each output port's data should be published to. Within the ROS2 framework, a subscriber is created for the initialized neural network object, allowing it to subscribe to the sensor topics determined in the previous step. A publisher is also created to publish its output action command topics. After binding is complete, the terminal activates the neural network instance. The neural network instance then continuously listens to its subscribed sensor topics. Whenever a new, time-synchronized raw sensor data stream arrives, the data is automatically populated into the corresponding input port, triggering the neural network to perform a forward inference calculation. The calculated action command is automatically published from the output port to the corresponding control topic, thus forming a closed loop.

[0197] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0198] Based on the same inventive concept, this application also provides a control system for a neural network-based modular intelligent robot, which implements the control method for the aforementioned neural network-based modular intelligent robot. The solution provided by this system is similar to the implementation described in the above method. Therefore, the specific limitations of one or more embodiments of the control system for a neural network-based modular intelligent robot provided below can be found in the limitations of the control method for the neural network-based modular intelligent robot described above, and will not be repeated here.

[0199] In one exemplary embodiment, such as Figure 2 As shown, a control system 700 for a modular intelligent robot based on a neural network is provided, comprising:

[0200] Instance module 701 is used to query active nodes based on ROS2 asynchronous communication, obtain a hardware configuration list, and load the end-to-end neural network model corresponding to each independent hardware based on the hardware configuration list to obtain a neural network model instance.

[0201] The synchronization module 702 is used to aggregate the raw sensor data streams of the various independent hardware components of the modular intelligent robot and to perform time synchronization and alignment of the sensor data streams to obtain multimodal perception data packets.

[0202] Feature module 703 is used to input multimodal sensing data packets into the encoder of the end-to-end neural network model instance to perform state feature encoding and obtain high-dimensional feature vectors;

[0203] The decision module 704 is used to input high-dimensional feature vectors into the decision network of the end-to-end neural network model instance to make action strategy decisions and obtain action instruction sets;

[0204] The motion module 705 is used to solve the motion instruction set into a motion trajectory to obtain a smooth motion trajectory; wherein, the smooth motion trajectory is used to instruct each actuator of each independent hardware to adjust the operating parameters.

[0205] Furthermore, the control system 700 of the modular intelligent robot based on neural networks also includes a training module for;

[0206] The precise 3D model and hardware parameter data of the modular intelligent robot are obtained, and a high-fidelity simulation environment is constructed based on the precise 3D model and hardware parameter data; the high-fidelity simulation environment includes a virtual robot model and a training scene;

[0207] Based on a high-fidelity simulation environment, a reinforcement learning framework is used to generate policy actions and record them as actions. After the virtual robot model executes the policy actions, the state data of the virtual robot model and the state data of the training scene are recorded as states, thus obtaining a simulation interaction experience buffer.

[0208] Based on the simulation interaction experience buffer, with the goal of maximizing the expected value of accumulated rewards, the end-to-end neural network framework is iteratively trained to obtain the neural network policy model.

[0209] The neural network policy model is subjected to domain randomization reinforcement training to obtain a generalized neural network policy model.

[0210] The generalized neural network strategy model is lightweighted to obtain an end-to-end neural network model.

[0211] Furthermore, the training module is also used for:

[0212] Based on motion capture data playback or optimal control algorithms, the robot motion trajectory executed by the virtual robot model in a high-fidelity simulation environment is recorded, and the state data of the virtual robot model and the state data of the training scene are also recorded to obtain a demonstration dataset.

[0213] Based on the demonstration dataset, with the goal of minimizing the loss function value, the end-to-end neural network framework is iteratively trained to obtain the neural network policy model.

[0214] The neural network policy model is subjected to domain randomization reinforcement training to obtain a generalized neural network policy model.

[0215] The generalized neural network strategy model is lightweighted to obtain an end-to-end neural network model.

[0216] Furthermore, the training module is also used for:

[0217] By setting random variation ranges for each configuration parameter in the high-fidelity simulation environment, a domain randomization configuration table is obtained.

[0218] Based on the domain randomized configuration table, a set of random configuration parameters are sampled and injected into the high-fidelity simulation environment to obtain a simulation environment instance;

[0219] Based on the simulation interaction experience buffer or demonstration dataset, the neural network policy model is iteratively trained through simulation environment instances to obtain an updated neural network policy model.

[0220] Based on the validation environment set, the performance of the updated neural network policy model is evaluated to obtain the generalized neural network policy model.

[0221] The formula for calculating the expected value of accumulated rewards is as follows:

[0222]

[0223] in, To accumulate the expected value of rewards, For mathematical expectation, For the state at time step t, As a strategy, It is the instant reward for the k-th step in the future. This is the discount factor.

[0224] Furthermore, instance module 701 is also used for:

[0225] Based on the hardware configuration list, the encoding of each independent hardware component is parsed to obtain the hardware configuration query key;

[0226] Based on the hardware configuration query key, a matching search is performed in the pre-trained model library to obtain candidate pre-trained model files;

[0227] Based on the hardware configuration list, an adaptability check is performed on the candidate pre-trained model files to obtain the adaptability check results;

[0228] Based on the results of the adaptive check, candidate pre-trained model files are configured to obtain the initialized neural network;

[0229] The input and output ports of the initialized neural network are bound to the original sensor data stream to obtain a neural network model instance.

[0230] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a control method for a modular intelligent robot based on a neural network as described above.

[0231] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0232] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0233] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A control method for a modular intelligent robot based on neural networks, characterized in that, The method includes: Based on ROS2 asynchronous communication, active nodes are queried to obtain a hardware configuration list. Based on the hardware configuration list, the end-to-end neural network model corresponding to each independent hardware is loaded to obtain a neural network model instance. The raw sensor data streams of each independent hardware component of the modular intelligent robot are aggregated, and the sensor data streams are time-synchronized and aligned to obtain a multimodal perception data packet. The multimodal sensing data packet is input into the encoder of the end-to-end neural network model instance to perform state feature encoding and obtain a high-dimensional feature vector; The high-dimensional feature vector is input into the decision network of the end-to-end neural network model instance to make action strategy decisions and obtain an action instruction set; The motion instruction set is calculated into a motion trajectory to obtain a smooth motion trajectory; wherein, the smooth motion trajectory is used to instruct each actuator of each independent hardware to adjust its operating parameters.

2. The method according to claim 1, characterized in that, The end-to-end neural network model is constructed using the following method; The precise 3D model and hardware parameter data of the modular intelligent robot are obtained, and a high-fidelity simulation environment is constructed based on the precise 3D model and the hardware parameter data; wherein, the high-fidelity simulation environment includes a virtual robot model and a training scene; Based on the high-fidelity simulation environment, a strategy action is generated through a reinforcement learning framework, and the strategy action is recorded as an action. After the virtual robot model executes the strategy action, the state data of the virtual robot model and the state data of the training scene are recorded as states to obtain a simulation interaction experience buffer. Based on the simulation interaction experience buffer, with the goal of maximizing the expected value of accumulated rewards, the end-to-end neural network framework is iteratively trained to obtain the neural network policy model. The neural network policy model is subjected to domain randomization reinforcement training to obtain a generalized neural network policy model. The generalized neural network strategy model is lightweighted to obtain the end-to-end neural network model.

3. The method according to claim 2, characterized in that, The end-to-end neural network model may be constructed using the following methods: Based on motion capture data playback or optimal control algorithms, the robot motion trajectory executed by the virtual robot model in the high-fidelity simulation environment is recorded, and the state data of the virtual robot model and the state data of the training scene are recorded to obtain a demonstration dataset. Based on the demonstration dataset, with the goal of minimizing the loss function value, the end-to-end neural network framework is iteratively trained to obtain the neural network policy model; The neural network policy model is subjected to domain randomization reinforcement training to obtain a generalized neural network policy model. The generalized neural network strategy model is lightweighted to obtain the end-to-end neural network model.

4. The method according to claim 3, characterized in that, The step of performing domain randomization reinforcement training on the neural network policy model to obtain a generalized neural network policy model includes: A random variation range is set for each configuration parameter in the high-fidelity simulation environment to obtain a domain randomization configuration table; Based on the domain randomization configuration table, a set of random configuration parameters are sampled and injected into the high-fidelity simulation environment to obtain a simulation environment instance; Based on the simulation interaction experience buffer or the demonstration dataset, the neural network policy model is iteratively trained through the simulation environment instance to obtain an updated neural network policy model. Based on the validation environment set, the performance of the updated neural network strategy model is evaluated to obtain the generalized neural network strategy model.

5. The method according to claim 2, characterized in that, The formula for calculating the expected value of the accumulated reward is as follows: in, To accumulate the expected value of rewards, For mathematical expectation, For the state at time step t, As a strategy, It is the instant reward for the k-th step in the future. This is the discount factor.

6. The method according to claim 1, characterized in that, The process of loading the end-to-end neural network model corresponding to each independent hardware component in the hardware configuration list to obtain a neural network model instance includes: Based on the hardware configuration list, each of the independent hardware components is parsed and encoded to obtain a hardware configuration query key; Based on the hardware configuration query key, a matching search is performed in the pre-trained model library to obtain candidate pre-trained model files; Based on the hardware configuration list, an adaptability check is performed on the candidate pre-trained model files to obtain the adaptability check results; Based on the adaptive check results, the candidate pre-trained model file is configured to obtain the initialized neural network; The input and output ports of the initialized neural network are bound to the original sensor data stream to obtain the neural network model instance.

7. A control system for a modular intelligent robot based on a neural network, characterized in that, The system includes: The instance module is used to query active nodes based on ROS2 asynchronous communication, obtain a hardware configuration list, and load the end-to-end neural network model corresponding to each independent hardware based on the hardware configuration list to obtain a neural network model instance. The synchronization module is used to aggregate the raw sensor data streams of each independent hardware component of the modular intelligent robot and to perform time synchronization and alignment of the sensor data streams to obtain a multimodal perception data packet. The feature module is used to input the multimodal sensing data packet into the encoder of the end-to-end neural network model instance to perform state feature encoding and obtain a high-dimensional feature vector; The decision module is used to input the high-dimensional feature vector into the decision network of the end-to-end neural network model instance to make action strategy decisions and obtain an action instruction set; The motion module is used to solve the motion instruction set into a motion trajectory to obtain a smooth motion trajectory; wherein the smooth motion trajectory is used to instruct each actuator of each of the independent hardware to adjust its operating parameters.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images