Robot control method

Abstract

The application is suitable for the field of artificial intelligence technology, and provides a robot control method, which is applied to a man-machine collaborative system, the system comprising a first unit cluster, a second unit cluster, and a distributed memory pool; the method comprising: acquiring an environment image and limb information of a user; wherein the limb information comprises motion information and surface electromyogram signals of the user's limbs; based on the first unit cluster, performing perception processing on the environment image and the limb information to obtain terrain semantic parameters and motion intention information of the user, and storing the terrain semantic parameters and the motion intention information into the distributed memory pool; wherein the terrain semantic parameters represent mechanical arm control parameters adapted to the terrain; based on the second unit cluster, obtaining the terrain semantic parameters and the motion intention from the distributed memory pool, and controlling the mechanical arm of the robot to move according to the terrain semantic parameters and the motion intention information. The control efficiency and precision of the mechanical arm are improved.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a robot control method. Background Technology

[0002] Robots can fulfill the requirements of various work scenarios, including industrial, medical, and daily life, through human-robot collaboration. As robotics technology develops towards intelligence and humanization, human-robot collaborative operation with wheeled robots equipped with robotic arms has become a core technological requirement in the fields of intelligent manufacturing, medical rehabilitation, and service robots.

[0003] However, existing robotic arm collaborative systems still have many structural defects in perception, understanding, decision-making, and execution, which prevents robots from achieving accurate and efficient collaborative control according to user needs. Summary of the Invention

[0004] This application provides a robot control method that can improve the control efficiency and accuracy of robots.

[0005] In a first aspect, embodiments of this application provide a robot control method, which is applied to a human-machine collaborative system, the system comprising a first unit cluster, a second unit cluster, and a distributed memory pool; the method includes: Acquire environmental images and user limb information; wherein, the limb information includes the user's limb movement information and surface electromyography signals; Based on the first unit cluster, the environmental image and the limb information are processed for perception to obtain terrain semantic parameters and the user's motion intention information, and the terrain semantic parameters and the motion intention information are stored in the distributed memory pool; wherein, the terrain semantic parameters represent the robotic arm control parameters adapted to the terrain; Based on the second unit cluster, the terrain semantic parameters and the motion intention are obtained from the distributed memory pool, and the robot's robotic arm is controlled to move according to the terrain semantic parameters and the motion intention information.

[0006] Secondly, embodiments of this application provide a human-machine collaborative system, which includes a graphics processing unit, a neural network processing unit, a behavior processing unit, a microcontroller unit, and a distributed memory pool; The graphics processing unit is used to convert environmental images into three-dimensional point cloud data. The neural network processing unit is used to acquire the three-dimensional point cloud data from the distributed memory pool, identify terrain semantic parameters based on the three-dimensional point cloud data, and predict the user's movement intention information based on the user's limb information. The behavior processing unit is used to obtain the terrain semantic parameters and the motion intention information from the distributed memory pool, obtain the robotic arm control instructions based on the terrain semantic parameters and the motion intention information, and store the robotic arm control instructions in the distributed memory pool. The microcontroller unit is used to obtain the robotic arm control instructions from the distributed memory pool and control the robotic arm of the robot according to the robotic arm control instructions.

[0007] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0009] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the method described in the first aspect above.

[0010] In this embodiment, the human-machine collaborative system comprises a first unit cluster, a second unit cluster, and a distributed memory pool. The first unit cluster acts as the "brain" of the robot control, responsible for high-level perception and understanding tasks. By combining environmental images and limb information, it can make the robot's movement more adapted to the environment. Introducing surface electromyography (EMG) signals allows for the detection of movement intentions before the user's muscles exert force, effectively improving the efficiency and accuracy of robot control. The second unit cluster acts as the "cerebellum" of the robot control, responsible for low-level real-time motion control. Zero-copy data transfer between the "brain" and "cerebellum" is achieved through the distributed memory pool, enabling low-latency data interaction and improving the robot's control efficiency. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating the robot control method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the robot control method provided in an embodiment of this application; Figure 3 This is a schematic diagram of a joint cause-effect graph provided in an embodiment of this application; Figure 4 This is a flowchart illustrating the robot control method provided in an embodiment of this application; Figure 5 This is a flowchart illustrating the robot control method provided in an embodiment of this application; Figure 6 This is an architecture diagram of the human-machine collaborative system provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0013] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0014] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0015] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0017] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0018] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0019] The robot control method provided in this application can be applied to terminal devices such as embodied intelligence, embodied robots, collaborative robots, and robotic arms. This application does not impose any restrictions on the specific type of terminal device.

[0020] Before providing a further detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows: Surface electromyography (EMG) signals: weak bioelectrical signals generated during muscle contraction, voltage signals non-invasively collected from the skin surface, reflecting the nerve's driving activity on the muscle.

[0021] Interconnection channel: In a multi-level control system, the interconnection channel refers to the communication and control channel responsible for bidirectional data interaction, status synchronization, and command transmission between levels.

[0022] The cerebellum-cerebellum heterogeneous computing architecture is a computing framework that separates the decision-making (brain) and real-time control (cerebellum) of an intelligent system into two types of heterogeneous hardware and layers to work together. The core is the separation and coordination of high computing power thinking and high real-time action.

[0023] As robotics technology develops towards intelligence and humanization, human-robot collaborative operation with wheeled robots equipped with robotic arms has become a core technological requirement in intelligent manufacturing, medical rehabilitation, and service robotics. However, existing wheeled robot robotic arm collaborative systems still have many structural defects in the four stages of perception, understanding, decision-making, and execution, making it impossible for the system to achieve real-time, safe, and natural collaborative control in complex terrains and dynamic human-robot interaction scenarios.

[0024] Current mainstream collaborative control solutions for wheeled robot arms employ a layered software architecture. The perception layer runs on a general-purpose processor, while the control layer runs on a real-time controller. Data exchange between the two layers occurs through operating system middleware. This approach generally meets the requirements in structured environments, single-task scenarios, and low-dynamic human-machine interaction scenarios. However, it exposes systemic bottlenecks in unstructured terrain, multi-task concurrency, and high-dynamic human-machine collaborative scenarios. For example, inaccurate understanding of motion intent leads to trajectory prediction errors; lagging terrain semantic understanding prevents the robot arm's control parameters from being adapted in advance; a single prediction channel causes control instability under dynamic disturbances; and a distributed computing architecture results in excessively high end-to-end latency.

[0025] This application belongs to the technical field of robot control and artificial intelligence chip design. Specifically, it relates to a collaborative control system and method for a wheeled robot arm that integrates motion intention understanding of surface electromyography signals, real-time compilation of terrain semantic parameters, dual-channel residual decoupling predictive control, and a heterogeneous computing architecture of the cerebellum and cerebrum. It is applicable to scenarios requiring human-machine collaborative operation, such as industrial collaborative robots, rehabilitation exoskeletons, remote operation robots, and wheeled service robots.

[0026] According to one embodiment of this application, a robot control method is provided, which is applied to a human-machine collaborative system, the system including a first unit cluster, a second unit cluster, and a distributed memory pool.

[0027] Figure 1 A flowchart illustrating the robot control method, such as... Figure 1 As shown, a robot control method according to one embodiment of this application may include: Step 110: Acquire environmental images and user limb information; wherein, limb information includes user limb movement information and surface electromyography signals; Step 120: Based on the first unit cluster, perform perception processing on the environmental image and limb information to obtain terrain semantic parameters and user motion intention information, and store the terrain semantic parameters and motion intention information in a distributed memory pool; wherein, the terrain semantic parameters represent the control parameters of the robotic arm adapted to the terrain. Step 130: Based on the second unit cluster, obtain terrain semantic parameters and motion intentions from the distributed memory pool, and control the robot's robotic arm to move according to the terrain semantic parameters and motion intention information.

[0028] Steps 110-130 are described in detail below.

[0029] In step 110, the user and the robot can work together in the same space. The human-machine collaborative system is a comprehensive system that relies on the robot and the operator, i.e. the user, to complete the task. The system is equipped with a first unit cluster, a second unit cluster, and a distributed memory pool. The three components work together to support the overall operation of the robot control method.

[0030] The robot can be equipped with cameras to capture environmental images of its surroundings, or cameras can be installed in the space to capture environmental images. During human-robot collaboration, environmental images and user limb information can be acquired in real time. Limb information can include the user's limb movement information and surface electromyography (EMG) signals. Movement information can characterize limb orientation, movement speed, and trajectory. For example, the robot can acquire the orientation and angular velocity of the user's wrist, as well as the EMG signals of the wrist skin.

[0031] The environmental images in this embodiment may include color images and / or depth images. For example, color images and depth images acquired by a depth camera may be received.

[0032] Specifically, environmental images, captured by vision sensors mounted on the robot, reflect the appearance and terrain of the environment in which the robot operates, serving as the fundamental data source for environmental perception analysis. User limb information includes two categories: limb movement information and surface electromyography (EMG) signals. Limb movement information visually represents the user's current movement state, posture changes, and movement trends. EMG signals, electrical signals generated by muscle activity, can anticipate the user's movement tendencies before any limb movement is made. Combining these two types of information allows for a complete reconstruction of the user's limb behavior and potential movement intentions, providing multi-dimensional evidence for analyzing movement intentions. Acquiring environmental images and user limb information is the initial step in robot control methods, used to comprehensively collect environmental and human data in human-robot collaborative scenarios, providing raw input for subsequent perception processing stages.

[0033] In step 120, the first unit cluster is the brain in the heterogeneous computing architecture of the cerebellum and cerebellum. It undertakes computational tasks such as perception and data parsing. It is the core hardware combination for realizing the interpretation of environmental and human information and can provide effective data support for subsequent control logic.

[0034] The first unit cluster can receive environmental images and limb information in real time, and perform full-dimensional perception processing on the acquired environmental images and limb information. On the one hand, it relies on environmental images to complete the identification and analysis of terrain-related features, and generate terrain semantic parameters. The terrain semantic parameters are a set of parameters obtained by encoding different terrain features. Its core function is to match the corresponding robotic arm control parameters according to the actual state of the terrain, so that the robotic arm's operating characteristics can adapt to various complex terrain environments, and ensure the stability and compliance of the robotic arm in different terrains. On the other hand, it relies on the user's limb motion information and surface electromyography signals to complete the interpretation of movement intention, and obtain the user's movement intention information. The movement intention information can predict the user's subsequent limb movement trajectory and movement tendency, providing a predictive basis for the robotic arm to follow and assist the user's movements.

[0035] After the perception processing is completed, the first unit cluster will store the generated terrain semantic parameters and motion intent information into a distributed memory pool. With the help of the unified storage capability of the distributed memory pool, the data can be centrally managed, making it convenient for the second unit cluster to quickly retrieve and use it.

[0036] In step 230, the second unit cluster undertakes the tasks of instruction generation and motion control, and outputs the robotic arm motion instructions based on the information parsed by the first unit cluster, thereby directly realizing the motion control of the robotic arm.

[0037] As a unified data storage and interaction carrier, the distributed memory pool integrates the local storage resources of each processing unit in the system, realizes efficient data transfer and sharing between different unit clusters, eliminates redundant operations in the data transmission process, and ensures the real-time nature of information interaction.

[0038] The second unit cluster can read stored terrain semantic parameters and motion intent information from the distributed memory pool, and combine the two types of information to calculate and output control commands. During the command generation process, motion intent information can be used to determine the overall motion direction, trajectory, and rhythm of the robotic arm, while terrain semantic parameters can be used to dynamically adjust control parameters such as the robotic arm's operating stiffness, damping, and output torque, so that the robotic arm's motion mode matches both the user's motion requirements and the current terrain environment.

[0039] The control commands output by the second unit cluster can be directly applied to the robot's robotic arm, driving it to complete the corresponding motion actions. At the same time, it can also be combined with real-time feedback information to correct the motion process, ensuring the accuracy and safety of the robotic arm's movement.

[0040] For example, the second unit cluster can be formed by combining a processing unit responsible for motion planning and a processing unit responsible for underlying drive monitoring. The two types of units sequentially complete instruction calculation, signal conversion and device driving, and are assembled in the hardware area of ​​the robot near the robotic arm and the walking chassis, shortening the instruction transmission path.

[0041] In this embodiment, the first unit cluster completes the perception and analysis of environmental and human information and generates corresponding parameters, the distributed memory pool realizes efficient data flow, and the second unit cluster completes instruction calculation and robotic arm control. This enables dual adaptation of environmental terrain and user movement intention in human-machine collaborative scenarios, improves the real-time performance, rationality and environmental adaptability of robotic arm motion control, and ensures the smooth operation of human-machine collaborative work.

[0042] In this embodiment, the human-machine collaborative system comprises a first unit cluster, a second unit cluster, and a distributed memory pool. The first unit cluster acts as the "brain" of the robot control, responsible for high-level perception and understanding tasks. By combining environmental images and limb information, it can make the robot's movement more adapted to the environment. Introducing surface electromyography (EMG) signals allows for the detection of movement intentions before the user's muscles exert force, effectively improving the efficiency and accuracy of robot control. The second unit cluster acts as the "cerebellum" of the robot control, responsible for low-level real-time motion control. Zero-copy data transfer between the "brain" and "cerebellum" is achieved through the distributed memory pool, enabling low-latency data interaction and improving the robot's control efficiency.

[0043] Figure 2 A flowchart illustrating the robot control method, such as... Figure 2 As shown, a robot control method according to one embodiment of this application may include: Step 210: Acquire environmental images and user limb information; wherein, limb information includes the user's limb movement information and surface electromyography signals; Step 220: Based on the graphics processing unit, convert the environmental image into 3D point cloud data and store the 3D point cloud data in a distributed memory pool; Step 230: Based on the neural network processing unit, obtain 3D point cloud data from the distributed memory pool, identify terrain semantic parameters based on the 3D point cloud data, and predict the user's movement intention information based on limb information, and store the terrain semantic parameters and movement intention information in the distributed memory pool. Step 240: Based on the second unit cluster, obtain terrain semantic parameters and motion intentions from the distributed memory pool, and control the robot's robotic arm to move according to the terrain semantic parameters and motion intention information.

[0044] Steps 210 and 240 have been described in detail above and will not be repeated here. Steps 220 and 230 will be described in detail below.

[0045] In step 220, the first unit cluster includes a graphics processing unit and a neural network processing unit; that is, the human-machine collaborative system includes both a graphics processing unit and a neural network processing unit. As a functional unit combination performing perception processing, the first unit cluster is jointly composed of the graphics processing unit and the neural network processing unit. The graphics processing unit mainly undertakes the analysis of environmental images and the construction of 3D data, capable of transforming raw visual image data into structured data that can be used for subsequent intelligent analysis. It is the front-end processing module of the environmental perception link. The neural network processing unit can rely on its intelligent computing capabilities to complete terrain recognition and motion intention prediction, receiving data output from the graphics processing unit and collected limb information to achieve semantic analysis and motion trend prediction. The two work together to complete the overall perception processing flow, providing two types of core information for subsequent control stages.

[0046] The graphics processing unit (GPU) can handle environmental perception tasks by performing the following steps: receiving color and depth images from a depth camera, running a downsampling algorithm to construct 3D point cloud data of the environment, writing the 3D point cloud data into a high-bandwidth region of a distributed memory pool, and labeling it with the current task number. In other words, the GPU receives the acquired environmental images, performs format conversion and feature reconstruction processing on them, and generates 3D point cloud data that represents the spatial geometry. This 3D point cloud data fully preserves the spatial structure information of the environment and can serve as the basis for terrain recognition. After generating the data, the GPU stores the 3D point cloud data in a distributed memory pool, leveraging the unified storage capabilities of the distributed memory pool to support data retrieval by the neural network processing unit.

[0047] For example, the graphics processing unit can be hardware connected to the vision acquisition device and arranged in the area where the robot is equipped with vision sensors. The graphics processing unit and the distributed memory pool can interact with each other through a high-bandwidth transmission link.

[0048] Task numbers can be uniformly assigned and managed by the task scheduler, which can be located in the heterogeneous computing layer of the brain and cerebellum. Each unit in the system can determine the required current task number in the following ways: (1) When the system detects the start of a human-machine collaborative task, the task scheduler assigns a unique current task number according to the type of the current task (such as handling, assembly, guidance, etc.) and writes the number into the global register of the distributed memory pool, which is readable by all processing units; (2) When the graphics processor writes 3D point cloud data, it reads the current task number from the global register and marks the current task number in the metadata header of the 3D point cloud data, so that the source and ownership of all 3D point cloud data have a clear identifier; (3) When the neural network processing unit needs to read 3D point cloud data, it first queries the current task number in the global register, and then quickly locates the required data block (such as 3D point cloud data, semantic labels, sensor data, etc.) in the distributed memory pool with the task number as the index, without having to traverse all data blocks or guess the task number; (4) Similarly, the behavior processing unit queries the task number in the global register and then accurately reads the terrain semantic parameters and motion intention information corresponding to the task number from the memory pool; (5) Similarly, the microcontroller queries the task number and then accurately reads the control instruction sequence corresponding to the task number. The task number is maintained uniformly by the task scheduler. Each unit only needs to read the global register to know which task it is currently processing, thus avoiding the task number negotiation overhead in a distributed system.

[0049] In other words, the first unit cluster includes a graphics processing unit (GPU) and a neural network processing unit (NNF), while the second unit cluster includes a behavior processing unit (HRF) and a microcontroller unit (MCU). The local memory of the GPU, HRF, HRF, and MCU is aggregated into a unified physical address space, supporting direct linear address access across processing units. A global memory management protocol is constructed: when the GPU writes data, it uses task numbers to label data blocks; the NNF directly reads the required data blocks using task numbers; the HRF directly reads semantic mapping results and intent prediction results using task numbers; and the MCU directly reads control instruction sequences using task numbers. The entire data transfer process requires no operating system intervention and no data copying operations; the end-to-end latency from data generation to data reading is controlled within two milliseconds.

[0050] In step 230, the neural network processing unit can read the stored 3D point cloud data from the distributed memory pool, perform terrain feature recognition calculations on the 3D point cloud data, and extract terrain semantic parameters that can match the control requirements of the robotic arm, thereby reflecting the terrain attributes of the current environment and correspondingly adapting the robotic arm's operating parameters.

[0051] The neural network processing unit can also read the collected limb information and combine it with the motion-related data and surface electromyography signals contained in the limb information to predict the user's movement intention information and predict the user's subsequent limb movement state in advance, that is, predict the subsequent limb movement trajectory. In other words, the neural network processor undertakes the tasks of motion intention understanding and terrain semantic recognition. The obtained terrain semantic parameters and user motion intention information can also be stored in a distributed cache, and the terrain semantic parameters and motion intention information can also be labeled with task numbers.

[0052] For example, the graphics processing unit can simultaneously process multiple environmental images to generate multiple sets of 3D point cloud data, and the neural network processing unit can read different batches of 3D point cloud data in time-sharing to perform continuous recognition operations, or perform synchronous operations on a single set of 3D point cloud data and real-time limb information, adapting to the data processing needs under different work scenarios.

[0053] In this embodiment, the graphics processing unit completes the conversion and storage of environmental images into three-dimensional point cloud data, and then the neural network processing unit analyzes the terrain semantic parameters based on the three-dimensional point cloud data and predicts the motion intention information based on limb information. This can complete the environmental perception and human intention analysis in layers, ensuring that the perception processing process is clearly divided and the data flows smoothly, thereby improving the efficiency and accuracy of generating terrain semantic parameters and motion intention information.

[0054] In one implementation, identifying terrain semantic parameters based on 3D point cloud data includes: partitioning the 3D point cloud data to obtain multiple sub-regions in the 3D point cloud data; identifying the terrain category of each sub-region; and determining the terrain semantic parameters based on the terrain category of each sub-region and the coordinate range of each sub-region in the 3D point cloud data.

[0055] Specifically, 3D point cloud data is structured data derived from environmental images, representing the spatial geometry of the environment. It carries the spatial coordinates of various locations within the scene and serves as the fundamental basis for terrain recognition. The neural network processing unit partitions the 3D point cloud data, dividing the overall space into multiple independent analysis units. This reduces the computational burden of a single data volume and enables refined analysis of terrain at different spatial locations. Partitioning can be a method of dividing the complete 3D point cloud data into regions according to spatial distribution rules. After partitioning, multiple independent sub-regions are obtained, each corresponding to a different spatial location within the environment. Each sub-region can be used independently as a terrain recognition object, ensuring that terrain analysis corresponds to a specific spatial range.

[0056] For example, 3D point cloud data can be uniformly partitioned based on spatial adjacency, or non-uniform partitioning can be achieved by combining point cloud density features. The partitioning logic module can be integrated into the neural network processing unit, forming a functional connection with the data reading module. Each sub-region can be labeled as a 3D bounding box. The 3D bounding box is the boundary box of the sub-region, and the location of the 3D bounding box represents the coordinate range of the sub-region.

[0057] The process of identifying terrain semantic parameters from 3D point cloud data involves using partitioned sub-regions as basic units to sequentially determine terrain categories, match spatial locations, and finally integrate them to obtain complete terrain semantic parameters. This process relies on the idea of ​​regional parsing to achieve the fusion of spatial geometric information and terrain attribute information.

[0058] Terrain category identification for each sub-region is the process of determining the terrain type of the corresponding spatial region. Terrain categories distinguish different ground environmental features, intuitively reflecting the environmental attributes of each sub-region and providing attribute basis for subsequent parameter generation. For example, a sparsely activated neural network model can be deployed in the neural network processing unit. This model analyzes the 3D point cloud data generated by the graphics processing unit region by region, identifying the terrain category of each sub-region, such as flat ground, slope, carpet, slippery ground, steps, etc. Combining the terrain category and coordinate range of each sub-region, the complete terrain semantic parameters of the current space are determined. That is, the robotic arm control parameters adapted to the terrain of each sub-region are obtained when the robot moves in the current space.

[0059] In other words, after obtaining the terrain categories of all sub-regions, the distribution of each terrain category in the overall space can be determined by combining the coordinate range of each sub-region. The terrain attributes and spatial coordinates are then linked and integrated to generate complete terrain semantic parameters. The terrain semantic parameters integrate two types of information: terrain type and spatial location. They can accurately reflect the overall terrain status around the robot and provide a complete basis for the dynamic adaptation of the robotic arm control parameters.

[0060] In this embodiment, by partitioning the 3D point cloud data and identifying the terrain category in each region, and then combining the coordinate range of the sub-regions to determine the terrain semantic parameters, it is possible to achieve refined identification and information integration of the environmental terrain, so that the terrain semantic parameters completely correspond to the spatial location and terrain features, thereby improving the comprehensiveness and accuracy of terrain perception.

[0061] In one implementation, determining terrain semantic parameters based on the terrain category of each sub-region and the coordinate range of each sub-region in the 3D point cloud data includes: generating a terrain semantic vector based on the terrain category of each sub-region and the coordinate range of each sub-region in the 3D point cloud data; wherein the terrain semantic vector represents at least one of the terrain category, ground friction coefficient, ground flatness, and ground slope in the environment; generating terrain semantic parameters based on the terrain semantic vector and a preset mapping function; wherein the terrain semantic parameters include at least one of the stiffness coefficient, damping coefficient, maximum output torque, and desired contact force threshold of each joint of the robotic arm.

[0062] Specifically, the terrain semantic vector is a multi-dimensional feature vector generated by fusing the terrain category and spatial coordinate range of a sub-region. It can uniformly encode multiple physical features of the environmental terrain, inherit the recognition results of the 3D point cloud sub-region, and serve as an intermediate data form connecting the terrain perception results and the control parameters of the robotic arm, providing standardized input for subsequent mapping calculations. Ground friction coefficient, ground flatness, and ground slope are all feature quantities describing the physical properties of the terrain. Combined with the terrain category, they are included in the encoding scope of the terrain semantic vector, which can comprehensively depict the actual state of the environmental terrain from different dimensions, making the expression of terrain-related features more comprehensive.

[0063] The neural network processing unit can be responsible for compiling the semantic information of the environmental terrain into robotic arm control parameters, namely terrain semantic parameters, in real time, enabling the robotic arm to adapt to the compliant control requirements under different terrains in advance. Before generating terrain semantic parameters, terrain semantic vectors can be generated first. The process of generating terrain semantic vectors can be a process of transforming discrete terrain category information and spatial coordinate information into continuous feature representations. This process is based on the results of previous sub-region terrain identification, completing the transformation from terrain identification conclusions to standardized feature data, laying the foundation for subsequent parameter mapping.

[0064] For example, the terrain semantic vector generation module can be embedded within the neural network processing unit. Based on the terrain category and coordinate range of each sub-region, it extracts the terrain category, ground friction coefficient, ground flatness, and ground slope of each sub-region from the 3D point cloud data. Combining these four features, it encodes a ten-dimensional terrain semantic vector. Each dimension of this vector takes a continuous value between zero and one. Different terrain categories correspond to different regions in the vector space, and the vector representations of adjacent terrain categories are continuously distributed in space.

[0065] Mapping functions are pre-defined operational rules that convert feature vectors into control parameters. They establish a correspondence between terrain semantic vectors and robotic arm control parameters, ensuring that terrain features can be smoothly transformed into control quantities adapted to the robotic arm's operating state. Terrain semantic parameters are a set of parameters directly used to regulate the robotic arm's operating state, including stiffness coefficients, damping coefficients, maximum output torque, and desired contact force thresholds for each joint. Stiffness and damping coefficients adjust the mechanical compliance characteristics of the robotic arm during movement, maximum output torque limits the power output range of the robotic arm joints, and the desired contact force threshold standardizes the force application during human-robot collaboration or object interaction. These parameters work together to allow the robotic arm's operating characteristics to adapt to the current terrain environment.

[0066] Generating terrain semantic parameters based on terrain semantic vectors combined with preset mapping functions is the core computational process for converting terrain features into robotic arm control parameters. This process continues the output results of the preceding terrain semantic vectors, completes feature mapping according to the established computational logic, and finally obtains parameter data that can be directly applied to the robotic arm control process.

[0067] For example, the computation module corresponding to the mapping function can be deployed in the neural network processing unit, directly interfacing with the terrain semantic vector generation module. The computation results can be directly transmitted to a distributed memory pool for storage. The mapping function can use non-linear computation to achieve the mapping transformation of complex features, or it can use linear computation to complete the corresponding transformation of simple features, thus adapting to parameter generation scenarios with different accuracy requirements.

[0068] For a specific example, the mapping from terrain semantic vectors to terrain semantic parameters can be achieved using a fully connected neural network to realize a differentiable mapping. The network takes a 10-dimensional terrain semantic vector as input and outputs a 24-dimensional vector of robotic arm control parameters, i.e., the terrain semantic parameters. These 24 parameters include: stiffness coefficients (six-dimensional), damping coefficients (six-dimensional), maximum output torque (six-dimensional), and desired contact force thresholds (six-dimensional) for each joint of the robotic arm. The intermediate hidden layers of the neural network have a 64-dimensional dimension, and the activation function uses a smooth-slope learnable modified linear unit to ensure the differentiability of the entire mapping function. The training of this neural network uses a reinforcement learning framework. The specific training process can be as follows: construct various terrain scenarios in a simulation environment, allowing the robot to perform the same grasping task on different terrains, recording the task success rate and control parameter values; using the maximization of task success rate as the optimization objective, update the parameters of the neural network using a policy gradient algorithm; after training, the neural network learns to output the optimal robotic arm control parameter vector under different terrain semantic inputs.

[0069] To ensure the mapping results conform to physical laws, a physical constraint loss can be introduced during training. This loss can include three aspects: kinematic constraints, requiring the generated joint angles to not exceed the physiological limits of the robotic arm and that the velocity is continuous; dynamic constraints, requiring the generated joint torques to be within the rated output range of the motor and satisfy the torque balance equation; and compliance control constraints, requiring the generated stiffness and damping coefficient combination to ensure force control stability during human-machine physical contact. The physical constraint loss is added to the total loss function in the form of the sum of squared residuals, and its weighting coefficients are determined to be appropriate values ​​through cross-validation to ensure that physical rationality accounts for a sufficient proportion of the optimization objective.

[0070] In this embodiment, a terrain semantic vector containing multiple terrain features is first generated by combining the terrain information and coordinate range of the sub-region. Then, the control parameters corresponding to the robotic arm are obtained by converting the terrain semantic vector using a preset mapping function. This enables a continuous and differentiable conversion from terrain features to robotic arm control parameters, ensuring that the control parameters are smoothly adjusted as the terrain changes, and improving the robotic arm's adaptability to different terrain environments.

[0071] In one implementation, a terrain semantic vector is generated based on the terrain category of each sub-region and the coordinate range of each sub-region in the 3D point cloud data. This includes: storing the terrain category of each sub-region and the coordinate range of each sub-region in the 3D point cloud data in a distributed memory pool using a neural network processing unit; obtaining the terrain category of each sub-region and the coordinate range of each sub-region in the 3D point cloud data from the distributed memory pool using a graphics processing unit, and mapping the terrain category of each sub-region and the coordinate range of each sub-region in the 3D point cloud data to the 3D point cloud data to obtain a layered semantic map; wherein the layered semantic map represents 3D point cloud data containing terrain categories and corresponding coordinate ranges; storing the layered semantic map in the distributed memory pool using a graphics processing unit; and obtaining the layered semantic map from the distributed memory pool using a neural network processing unit, and extracting the terrain semantic vector from the layered semantic map.

[0072] Specifically, terrain category and the coordinate range corresponding to the sub-region are two types of basic information obtained by the neural network processing unit. The terrain category identifies the terrain attributes of different spatial regions, and the coordinate range defines the distribution boundary of the corresponding terrain category in three-dimensional space. The combination of these two types of information can completely describe the terrain morphology of a single location and the entire region, serving as the fundamental data source for constructing hierarchical semantic maps and generating terrain semantic vectors. The distributed memory pool acts as a data transfer and persistent storage mechanism between units within the system. It can receive terrain category and coordinate range information output by the neural network processing unit, enabling data sharing between different functional units and ensuring that the graphics processing unit can stably retrieve the corresponding data for subsequent processing. Storing the terrain category and coordinate range of each sub-region into the distributed memory pool is a preliminary step for data temporary storage and cross-unit interaction. This step receives the terrain recognition results and prepares complete input data for the construction of the hierarchical semantic map. The terrain category and corresponding coordinate range can also be labeled with a task number.

[0073] Layered semantic maps are composite data carriers formed by integrating terrain categories and spatial coordinate information on the basis of original 3D point cloud data. They retain the spatial geometric information of the 3D point cloud itself and overlay the terrain semantic information corresponding to each region. They can intuitively realize the unified expression of geometric shape and terrain semantics and can serve as a direct basis for extracting terrain semantic vectors.

[0074] The graphics processing unit (GPU) reads terrain category and coordinate range information from the distributed memory pool and maps these two types of information onto the 3D point cloud data to generate a hierarchical semantic map. This process involves semantic annotation and information fusion of the original point cloud data, establishing a connection between terrain recognition results and 3D spatial data, ensuring that each spatial location matches the corresponding terrain attributes. After completing the construction of the hierarchical semantic map, the GPU stores it back into the distributed memory pool. Leveraging the distributed memory pool's relay capabilities, the fused complete data is then sent back for the neural network processing unit to continue reading and using.

[0075] After retrieving the hierarchical semantic map from the distributed memory pool, the neural network processing unit extracts a terrain semantic vector based on the geometric and terrain semantic information integrated in the hierarchical semantic map. The terrain semantic vector is the result of standardized encoding of the multi-dimensional terrain features in the hierarchical semantic map, inheriting all the effective information from the hierarchical semantic map and providing standardized input data to the subsequent mapping function. The processing logic for extracting terrain semantic vectors from the hierarchical semantic map can perform unified extraction of global map features, or prioritize the extraction of local features of the robot's current spatial region, thereby matching the terrain perception needs of different ranges.

[0076] For example, the graphics processing unit (GPU) undertakes not only environmental perception tasks but also semantic mapping tasks. For semantic mapping, the GPU receives terrain categories output by the neural network processing unit, maps these categories to corresponding coordinates in the 3D point cloud data, and constructs a hierarchical semantic map that integrates geometric and semantic information. The hierarchical semantic map contains both 3D geometric information (such as the XYZ coordinates of the point cloud) and semantic information (the terrain category label corresponding to each point).

[0077] The neural network processing unit receives a hierarchical semantic map as input and extracts four features from the map: terrain category, estimated ground friction coefficient, estimated ground flatness, and estimated ground slope of the area where the robot is currently located. These features are encoded into a ten-dimensional terrain semantic vector. Each dimension of this vector takes a continuous value between zero and one. Different terrain categories correspond to different regions in the vector space, and the vector representations of adjacent terrain categories are continuously distributed in the space.

[0078] In this embodiment, the neural network processing unit and the graphics processing unit rely on a distributed memory pool to complete multiple rounds of data interaction, sequentially realizing information storage, hierarchical semantic map construction, map retrieval, and terrain semantic vector extraction. This enables the orderly fusion processing of three-dimensional spatial geometric information and terrain semantic information, ensuring a coherent terrain feature extraction process and stable data transmission, thereby improving the information integrity and extraction accuracy of terrain semantic vectors. This embodiment no longer discretizes terrain categories into limited labels (such as "flat," "slope," "carpet"), but instead encodes terrain semantic information into a continuous multi-dimensional parameter space. Each point in this space uniquely corresponds to a set of robotic arm control parameters (including stiffness coefficient, damping coefficient, maximum output torque, etc.), and the parameter values ​​smoothly transition with the continuous changes in terrain semantics.

[0079] In one embodiment, a data acquisition electrode and an inertial measurement unit are fixed to the user's limb. The data acquisition electrode is used to acquire surface electromyography (EMG) signals of the user's limb, and the inertial measurement unit is used to acquire motion information of the user's limb. Based on the limb information, the user's motion intention information is predicted, including: acquiring the user's motion image; wherein the motion image represents the position and posture of the user's limb; determining causal coding features based on the limb information and the motion image, and based on a preset joint causal graph; wherein the causal coding features represent the dependencies between the joints of the limb being moved by the user; and predicting the user's motion intention information based on the causal coding features.

[0080] Specifically, the acquisition electrodes are sensors placed on the surface of the user's limbs, primarily used to capture surface electromyography (EMG) signals generated by human muscle activity. These acquired EMG signals can reflect the user's limb movement tendencies in advance, providing early data support for predicting movement intentions. The acquisition electrodes are fixed to the user's limbs, enabling stable and continuous acquisition of signal data, and can work in conjunction with an inertial measurement unit to achieve simultaneous acquisition of multiple types of limb information.

[0081] An inertial measurement unit (IMU) is a motion sensor mounted on different parts of a user's limbs. It can detect motion state data during limb movement, thereby forming motion information of the user's limbs. It provides intuitive feedback on the current posture and movement changes of the limbs and complements the signals collected by the acquisition electrodes to form complete limb information.

[0082] The data acquisition electrodes can be attached to the surface of the user's upper limb muscles using a patch structure, while the inertial measurement unit can be separately arranged at multiple joints of the user's limb. The two can be deployed independently or integrated into the same wearable carrier, and the whole assembly is attached to the user's limb to complete the assembly.

[0083] For example, surface electromyography (SEMG) signals are read from the user's upper limb EMG signals. The user and robot are in the same workspace. The user transmits their movement intentions to the robot via SEMG signals, and the robot uses this information for predictive control, enabling the robotic arm to follow or assist the user's hand movements in a natural and smooth manner. The electrodes for collecting SEMG signals can be placed on the user's upper limbs (such as the forearm flexor muscles, forearm extensor muscles, biceps, triceps, and deltoid muscles), secured using patch electrodes or wearable armbands, eliminating the need for the user to don or remove complex mechanical exoskeletons. During free movement, changes in EMG signals lead actual limb movements (approximately tens to hundreds of milliseconds). This embodiment utilizes this "electromechanical delay" characteristic to detect movement intentions before visible muscle movement occurs, thus providing a predictive lead for robotic arm control.

[0084] The Inertial Measurement Unit (IMU) collects kinematic information about the joints of the user's upper limbs. Specifically, the IMU has six axes, measuring three-axis linear acceleration (via accelerometers) and three-axis angular velocity (via gyroscopes), with a sampling rate of at least 1,000 times per second. The IMU sensors can be fixed at three positions on the operator's upper limb: wrist, forearm, and upper arm, with one six-axis IMU module installed at each position. The IMU data from each position is low-pass filtered to remove high-frequency noise, and then the raw acceleration and angular velocity data are converted into the absolute orientation and angular velocity of each joint using attitude calculation algorithms (such as complementary filtering or Kalman filtering), thus obtaining the motion information of each joint in three-dimensional space. This IMU data, along with surface electromyography (EMG) signals and visual image data, is time-synchronized (with a time deviation of no more than five milliseconds) via a hardware trigger signal. These three data sources together form the multimodal sensor data input, which is then fed into a neural network processing unit for fusion processing. Therefore, the IMU collects the motion state information (position, orientation, and angular velocity) of the user's upper limb joints, used to help predict the user's upper limb movement trajectory over a future period. The acquisition of surface electromyography signals and motion information can be carried out continuously and can also switch the acquisition state according to the start and stop of the task, adapting to different human-machine collaborative work scenarios.

[0085] Predicting a user's movement intent involves a comprehensive process that integrates multi-source sensor data and motion images, combining the movement correlation patterns between joints to deduce movement trends. This process introduces motion images as a supplementary data source on top of limb information, and uses a pre-defined joint causal graph to analyze joint movement logic, gradually completing feature processing and intent prediction. Motion images, acquired by vision devices, visually represent the actual position and posture of the user's limbs in space, supplementing visual dimension information beyond sensor data, enriching the data dimensions used to analyze movement states, and forming a multimodal data combination with surface electromyography signals and movement information.

[0086] The pre-built joint causal graph is a logical model that represents the movement association rules between human limb joints. It reflects the interaction and transmission relationships that exist during the movement of different joints, thus constraining the feature extraction logic to ensure that the extracted features conform to the objective laws of human movement. Causal coding features are feature data obtained by fusing multi-source data and combining it with the joint causal graph analysis. They centrally reflect the movement dependencies between various limb joints and serve as an intermediate carrier connecting multimodal data and movement intention prediction. They can eliminate invalid information, strengthen effective movement association features, and improve the rationality of subsequent prediction processes.

[0087] For example, a pre-constructed graph attention network is used. This network has two layers, each using four attention heads, with a hidden feature dimension of 128. The graph attention network outputs causal encoded features. The information transmission path of the graph attention network strictly follows the directed edges preserved in the joint causal graph; that is, information can only flow from the causal parent node to the child node, and back propagation is not allowed. This design significantly reduces redundant computation, reducing the number of connections by approximately 30% to 50% compared to a fully connected graph structure, and improving computational efficiency by more than 40%. After obtaining the causal encoded features, a pre-constructed deep Transformer network can be used, containing twelve encoding layers and twelve decoding layers, each using eight attention heads, with a hidden feature dimension of 512. The encoder receives the multimodal fusion features, i.e., the causal encoded features, and the decoder uses an autoregressive method to predict future trajectories frame by frame, thus obtaining motion intention information. The deep Transformer network predicts the next five time steps, with each time step spaced at 100 milliseconds, resulting in a total prediction time of 500 milliseconds. The prediction output includes the position coordinates of the upper limb end in three-dimensional space and the rotation angles of each joint. After training, this deep Transformer network achieved performance metrics on a standard dataset, including an average end-position error of less than nine millimeters, an average joint angle error of less than two points and one degree, and an overall prediction accuracy of over 94%.

[0088] Based on the obtained causal coding features, calculations can be performed to accurately predict the user's subsequent limb movements by combining the inherent motion dependencies between joints, ultimately generating corresponding motion intention information. This motion intention information includes the user's subsequent limb trajectory and movement trend, providing a core basis for the coordinated movement of the robotic arm. The conversion of causal coding features into motion intention information can employ a multi-layer network model for deep analysis of complex features, or a lightweight computational model for rapid prediction in conventional scenarios, meeting different real-time and accuracy requirements. For example, the motion intention information prediction module is deployed in the neural network processing unit, serially connected to the causal coding feature generation module. The generated motion intention information can be directly written to a distributed memory pool for subsequent control calls.

[0089] In this embodiment, multiple types of limb information are acquired by collecting electrodes and inertial measurement units. Combined with motion images and relying on joint causal graphs to generate causal coding features, the prediction of motion intention information is then completed based on the causal coding features. This can effectively integrate multimodal data by combining the laws of human joint movement, thereby improving the rationality and accuracy of motion intention prediction results.

[0090] In this embodiment, the electrodes for collecting surface electromyography (EMG) signals are arranged in the main muscle groups of the upper limb, including the forearm flexors, forearm extensors, biceps, triceps, and deltoids, totaling sixteen electrode channels. The sampling rate of each channel is no less than 1,000 times per second. The raw EMG signals are bandpass filtered to remove baseline drift and high-frequency noise, and then smoothed using full-wave rectification and moving average to obtain an envelope signal reflecting the degree of muscle activation. The inertial measurement unit (IMU) has six axes, measuring three-axis linear acceleration and three-axis angular velocity. The sensors are fixed at three positions: wrist, forearm, and upper arm. Data from each position is low-pass filtered and attitude calculated, converting it into the absolute orientation and angular velocity of the joints. The visual sensor uses a depth camera to simultaneously acquire color and depth images, and extracts the position coordinates of each joint in the upper limb in three-dimensional space using a human pose estimation algorithm. The three types of data are time-synchronized via a hardware trigger signal, ensuring that the time deviation of the three modal data acquired at any given time does not exceed five milliseconds.

[0091] In one embodiment, the preset joint causal graph includes multiple joint nodes, and different joint nodes are connected by unidirectional edges. The method further includes: acquiring motion information, surface electromyography (EMG) signals, and motion images within a preset time period; for each joint node, determining the causal parent node of the joint node based on the motion information, EMG signals, and motion images within the preset time period; wherein, the causal parent node represents a node that has an influence on the motion of the joint node; and performing unidirectional edge connections between the joint node and its causal parent node to obtain the preset joint causal graph; wherein, the unidirectional edge connection is from the causal parent node of the joint node to the joint node.

[0092] Specifically, a joint cause-effect graph is pre-constructed, containing multiple nodes, each representing a joint; that is, the nodes in the joint cause-effect graph can be called joint nodes. Edges between different joint nodes are unidirectional. The joint cause-effect graph can be continuously updated according to a preset time period.

[0093] Joint nodes are logical units used to represent the movement states of various joints in the human body. Each joint node corresponds to a limb joint and can carry the movement information, surface electromyography signals, and visual features associated with that joint. They are the basic building blocks for constructing joint causal graphs. Unidirectional edge connections are logical links used to transmit movement relationships between joint nodes. These links have a fixed directionality, used to clarify the transmission direction of movement influence between different joints and ensure that the overall relationship conforms to the objective laws of human limb movement. Figure 3 This is a schematic diagram of a joint cause-effect graph. (Example) Figure 3As shown, the joint causal graph is composed of several joint nodes connected by unidirectional edges. It is used to present the overall motion influence relationships between all limb joints and provides rule constraints for subsequent extraction of causal coding features, ensuring that the feature parsing process conforms to the real-world limb motion logic. The division of joint nodes can be set independently based on the major joints of the human upper limb, or some joints can be combined according to motion linkage characteristics, adapting to different analysis granularity requirements.

[0094] Constructing a pre-defined joint causal graph relies on mining the influence relationships between joint movements using multi-source time-series data. This process visualizes these relationships and solidifies the rules through nodes and directional links. Based on multimodal data within a continuous time period, this process gradually identifies associations, builds links, and ultimately forms a reusable joint causal graph. Motion information, surface electromyography (EMG) signals, and motion images within the pre-defined time period are continuously collected time-series multimodal data. These data comprehensively record the user's muscle activity, joint movement state, and visual posture at different times, serving as the data source for determining the mutual influence relationships between joints. A causal parent node is a related node relative to a single joint node. The joint movement corresponding to the causal parent node can affect the movement of that single joint node, making it the source node for forming inter-joint linkages. Identifying the causal parent node allows us to trace the transmission sequence of limb movements. For each joint node, identifying the corresponding causal parent node using time-series multimodal data is a process of mining movement correlation logic from massive amounts of data. This process distinguishes directly influential joint combinations and eliminates irrelevant interference relationships. Multimodal data includes motion information, EMG signals, and motion images.

[0095] Multimodal time-series data can be uniformly aggregated into a distributed memory pool, where it is read in batches and subjected to correlation analysis by the neural network processing unit. The causal parent node determination module can run in conjunction with the data feature parsing module. Correlation analysis can be performed simultaneously on all joint nodes, or it can be performed on joint nodes one by one according to the sequence of limb movement transmission, adapting to different computation scheduling methods.

[0096] After determining the causal parent node corresponding to each joint node, a unidirectional edge connection is constructed with the causal parent node as the starting point and the corresponding joint node as the ending point. This completes the construction of the joint causal graph. The directional connection method strictly follows the direction of motion influence transmission, ensuring that the correlation represented by the causal graph has physical rationality. The constructed joint causal graph can be used for long-term feature extraction, or it can be dynamically updated by re-performing data collection and correlation analysis at fixed intervals to adapt to changes in user movement patterns.

[0097] For a specific example, the construction of a joint causal graph can employ a time-series causal discovery algorithm to dynamically learn the directed causal dependencies between human joint movements from time-series multimodal data. This algorithm consists of two phases: the first phase is conditional filtering, where candidate causal parent nodes are selected from all possible historical states for each joint node through a pre-defined conditional independence test; the second phase performs a more stringent independence test on the dataset of the candidate causal parent nodes through a pre-defined moment conditional independence test to control the false discovery rate. The final output is a directed causal graph, where each directed edge is accompanied by a causal strength value. Edges with causal strength values ​​greater than a pre-defined strength threshold (e.g., 0.75) can be retained, while weakly correlated or spurious causal connections are discarded. The causal graph is not static; the complete causal discovery process can be re-executed every five seconds to adapt to dynamic changes in user movement patterns.

[0098] The Conditional Independence Test (CIBT) is a fundamental tool in statistics and causal discovery. In this invention, the CIBT specifically means: given a joint node A and another joint node B, and a condition set Z (Z represents the historical states of other joint nodes besides A and B), the CIBT determines whether, given the information in Z, the historical state of A still provides additional predictive information about the current state of B. If the historical state of A no longer provides additional predictive information after Z is known (i.e., A⊥B∣Z), then A has no direct causal relationship with B, and their correlation is entirely explained by Z. Conversely, if the historical state of A still provides additional predictive information after Z is known (A not⊥B∣Z), then A has a direct causal relationship with B, and A is a candidate causal parent node of B. In the first stage of causal graph construction, the CIBT is performed on each pair of joint nodes (e.g., wrist-elbow, elbow-shoulder, etc.) across all possible combinations of historical states to filter out a set of candidate causal parent nodes. Commonly used methods for testing conditional independence include partial correlation tests, kernel-based conditional independence tests, and regression-based tests.

[0099] In the second stage, the conditional independence test of moments is used for a more rigorous test to reduce the false positive rate. The conditional independence test of moments is a statistical hypothesis testing method based on higher-order statistical moments, used to determine whether two variables are independent under a given set of conditions. In this embodiment, the conditional independence test of moments is used as the second stage of the causal discovery process. Specifically, the test objects are each pair of candidate causal parent nodes and their corresponding child nodes selected in the first stage, given a set of conditions Z (other key nodes besides A and B). The conditional independence test of moments does not only compare the first-order moments (mean), but compares all orders of moments (including first-order, second-order, third-order moments, etc.). Independence is determined by checking whether A and B have a dependency on any order of moments under the given Z conditions. If a dependency on any order of moments exists, they are determined to be not independent. Performing the conditional independence test of moments in the second stage of causal discovery provides a more rigorous test of the results of the first stage, effectively controlling the false discovery rate, i.e., reducing the incorrect identification of node pairs with no causal relationship as having one.

[0100] In a directed causal graph of human joint motion, each joint node represents the motion state of a joint (such as the angular velocity of the wrist, the orientation of the elbow, etc.), and the directed edges between nodes represent causal relationships. If there exists a directed edge from node A to node B, then A is the causal parent node of B, and B is the causal child node of A. For example, in human upper limb motion, there is a natural causal chain from shoulder to elbow to wrist. The movement of the shoulder drives the movement of the elbow, and the movement of the elbow drives the movement of the wrist. Therefore, the shoulder is the causal parent node of the elbow, and the elbow is the causal parent node of the wrist. The direction of the causal parent node is strictly unidirectional; information can only flow from the causal parent node to the child node and is not allowed to propagate backward. That is, wrist movement cannot cause shoulder movement, even though the two may occur simultaneously in time, but the wrist is not the causal parent node of the shoulder. In a graph attention network constrained by a causal graph, the message passing path strictly follows the direction of the directed edges, that is, information is passed from the causal parent node to the child node. This design makes the joint motion laws learned by the network conform to the proximal-driven-distal principle in human kinematics, improving the physical rationality of the prediction.

[0101] In this embodiment, the causal parent nodes of each joint node are identified by collecting multimodal time-series data for a specified period, and the edges between nodes are connected according to the orientation rules to generate a joint causal graph. This can objectively sort out the motion linkage relationship of limb joints based on actual motion data, so that the joint causal graph has dynamic adaptability and physical rationality, and provides reliable rule support for the subsequent generation of causal coding features.

[0102] The causal graph-constrained graph attention network is located in the feature extraction stage. Its input is the features of each joint node organized based on the causal graph structure (including electromyographic signal features, IMU pose features, and visual position features), and its output is the causal encoded features of joint motion after causal graph constraints. This causal encoded feature is then fed into the encoder of the deep Transformer network as the input feature for intent prediction. This pre-designed causal constraint allows the subsequent Transformer network to naturally follow the causal dependencies between human joints during prediction, thereby improving the physical plausibility and generalization ability of the prediction.

[0103] In this embodiment, the human-machine collaborative system comprises a first unit cluster, a second unit cluster, and a distributed memory pool. The first unit cluster acts as the "brain" of the robot control, responsible for high-level perception and understanding tasks. By combining environmental images and limb information, it can make the robot's movement more adapted to the environment. Introducing surface electromyography (EMG) signals allows for the detection of movement intentions before the user's muscles exert force, effectively improving the efficiency and accuracy of robot control. The second unit cluster acts as the "cerebellum" of the robot control, responsible for low-level real-time motion control. Zero-copy data transfer between the "brain" and "cerebellum" is achieved through the distributed memory pool, enabling low-latency data interaction and improving the robot's control efficiency.

[0104] Figure 4 A flowchart illustrating the robot control method, such as... Figure 4 As shown, a robot control method according to one embodiment of this application may include: Step 410: Acquire environmental images and user limb information; wherein, limb information includes the user's limb movement information and surface electromyography signals; Step 420: Based on the first unit cluster, perform perception processing on the environmental image and limb information to obtain terrain semantic parameters and user motion intention information, and store the terrain semantic parameters and motion intention information in a distributed memory pool; wherein, the terrain semantic parameters represent the control parameters of the robotic arm adapted to the terrain. Step 430: Based on the behavior processing unit, the terrain semantic parameters and motion intention information are fused to obtain the robotic arm control commands, and the robotic arm control commands are stored in the distributed memory pool. Step 440: Based on the microcontroller unit, obtain the robotic arm control instructions from the distributed memory pool, and control the robot's robotic arm to move according to the robotic arm control instructions.

[0105] Steps 410 and 420 have been described in detail above and will not be repeated here. Steps 430 and 440 will be described in detail below.

[0106] In step 430, the second unit cluster is a combination of functional units that perform motion control of the robotic arm. It consists of a behavior processing unit and a microcontroller unit. The behavior processing unit is responsible for integrating various parameters and generating control commands. It is an intermediate processing module connecting the upper-layer perception data and the lower-layer drive execution. The microcontroller unit is responsible for reading the control commands and directly driving the robotic arm to complete the actions, while also undertaking the related work of monitoring the running status. The two units work together to complete the entire process control of the robotic arm from command generation to action execution. They receive the terrain semantic parameters and motion intention information output by the first unit cluster to ensure that the robotic arm operates stably according to the preset logic. The data output by the behavior processing unit and the microcontroller unit can also be stored in a distributed memory pool and have a unique task code.

[0107] The second unit cluster combines terrain semantic parameters and motion intention information to control the movement of the robotic arm, sequentially completing the complete control process of parameter fusion, instruction generation, instruction distribution and device driving. This process uses a distributed memory pool to realize data interaction between the behavior processing unit and the microcontroller unit, and realizes the hierarchical execution of the control logic.

[0108] The behavior processing unit retrieves terrain semantic parameters and motion intent information from the distributed memory pool, performs fusion processing on the two types of data, and calculates the robotic arm control commands that match the current working conditions by combining the constraints of terrain on the robotic arm's operating characteristics and the guidance of the user's movement trends. The robotic arm control commands are standardized instruction data used to define the action forms and motion states of each component of the robotic arm, serving as the basis for executing robotic arm actions. After generating the robotic arm control commands, the behavior processing unit stores them in the distributed memory pool, leveraging the data transfer capabilities of the distributed memory pool to support the microcontroller unit in reading the commands.

[0109] In other words, the behavior processing unit undertakes the task of motion planning for the robotic arm. Specifically, it can perform the following steps: receive the terrain semantic parameters and user motion intention information output by the neural network processing unit, call the kinematic solver to calculate the angles, angular velocities, and angular accelerations of each joint of the robotic arm; convert the calculated joint spatial trajectories into a sequence of control instructions, write them into the low-latency region of the distributed memory pool, and label the instruction type and effective time.

[0110] For example, the behavior processing unit can be located in the robot's main control area, and interact with the distributed memory pool via a low-latency transmission link. The parameter fusion and instruction generation modules can be connected in series within the behavior processing unit. The parameter fusion processing can globally weight and fuse terrain semantic parameters and motion intent information, and different fusion methods can be selected according to the control accuracy requirements.

[0111] In step 440, the microcontroller unit reads the stored robotic arm control instructions from the distributed memory pool, parses and converts the instructions, and outputs drive signals to control the robot's robotic arm to complete the corresponding motion actions. The microcontroller unit has the ability to respond quickly and drive in real time, ensuring that the robotic arm's actions are executed accurately according to the instructions.

[0112] For example, the microcontroller unit can be positioned close to the drive motor of the robotic arm, shortening the transmission path from the command to the actuator. The microcontroller unit and the distributed memory pool can read commands through a dedicated physical channel, reducing interference during data transmission. The microcontroller unit can read and execute single robotic arm control commands one by one, or it can read multiple sets of commands in batches and execute them sequentially, adapting to different operating scenarios such as continuous actions and single actions.

[0113] The microcontroller unit (MCU) is responsible for chassis motion control and emergency safety monitoring. Specifically, it executes the following steps: It reads the control instruction sequence written to the low-latency region of the distributed memory pool by the behavior processing unit, converts the joint angle control instructions into motor drive phase signals, and outputs them to the motors of each joint of the robotic arm; simultaneously, it reads chassis wheel speed and odometer data and LiDAR data to monitor the chassis motion status; when the LiDAR detects an obstacle entering a preset safe distance range, the MCU generates a hardware interrupt signal. Upon receiving this interrupt signal, the behavior processing unit immediately freezes the robotic arm's movement, and the MCU drives the chassis to perform differential steering obstacle avoidance.

[0114] After the lidar detects an obstacle, the microcontroller executes obstacle avoidance maneuvers on the chassis, not the robotic arm. The microcontroller's safety monitoring function can be divided into two levels: emergency freezing of the robotic arm and safe obstacle avoidance of the chassis. When the lidar detects an obstacle (including people, other objects, walls, etc.) entering the preset safe distance range in the workspace, the microcontroller generates a hardware interrupt signal. Upon receiving the interrupt signal, the behavior processing unit immediately pauses the robotic arm's movement (i.e., freezes the current action), while the microcontroller drives the chassis to perform differential steering, moving the robot away from the obstacle. This combined strategy of robotic arm freezing and chassis obstacle avoidance ensures overall safety; the robotic arm remains stationary to prevent collision damage or injury, while the chassis actively avoids obstacles, increasing the safety margin. The hardware interrupt mechanism of the microcontroller in the cerebellum-cerebellum architecture and the low-latency transmission capability of the dedicated physical channel keep the response delay from lidar detection of an obstacle to complete robotic arm freezing within five milliseconds.

[0115] In this embodiment, the behavior processing unit integrates terrain semantic parameters and motion intention information to generate and store robotic arm control commands. The microcontroller then reads the commands and drives the robotic arm to move. This enables hierarchical processing of the control process, clarifies the functional boundaries of different units, and improves the real-time performance and execution stability of the robotic arm control process.

[0116] In one embodiment, the robotic arm is equipped with a six-dimensional force sensor; the terrain semantic parameters and motion intention information are fused to obtain robotic arm control commands, including: generating expected torque information based on the terrain semantic parameters and motion intention information; wherein the expected torque information represents the desired joint torque of the robotic arm; acquiring the current force information of the robotic arm through the six-dimensional force sensor, and generating corrective torque information based on the current force information and the expected torque information; wherein the corrective torque information is used to compensate for the expected torque information; and generating robotic arm control commands based on the expected torque information and the corrective torque information.

[0117] Specifically, the six-dimensional force sensor is a sensor mounted on the robotic arm that can detect the forces and torques acting on the robotic arm during operation in real time and output the corresponding current force information. The current force information can intuitively reflect the force state between the robotic arm and the surrounding environment and the operator, providing real-time feedback for torque compensation and command correction. It works in conjunction with the behavior processing unit to complete torque calculation and command optimization, thereby improving the dynamic adaptability of the robotic arm's motion control.

[0118] For example, the six-dimensional force sensor can be installed at the end of the robotic arm or integrated into the joint of the robotic arm. The data collected by the sensor can be directly transmitted to the behavior processing unit through hardware circuitry, or it can be stored in a distributed memory pool and then retrieved by the behavior processing unit. The six-dimensional force sensor can continuously collect data and can also be combined with the start and stop functions of the robotic arm's motion status to adapt to different working conditions.

[0119] The process of fusing terrain semantic parameters and motion intent information to obtain robotic arm control commands involves combining a preset torque target with real-time force feedback for torque correction, ultimately generating execution commands. This process relies on dual-channel torque generation and compensation logic to combine open-loop predictive control with closed-loop feedback correction, ensuring the accuracy and safety of the robotic arm's movement. The predicted torque information, calculated by combining terrain semantic parameters and motion intent information, corresponds to the theoretical torque required by the robotic arm based on its motion trend and environmental characteristics, forming the foundation of the robotic arm control commands. The behavior processing unit determines the mechanical output boundaries and compliance characteristics of the robotic arm joints based on the terrain semantic parameters, and determines the robotic arm's motion trajectory and movement rhythm by combining the motion intent information. The predicted torque information is calculated after fusing these two elements, thus clarifying the theoretical output standards for each joint of the robotic arm. A preset dynamic function can be deployed in the behavior processing unit to calculate the predicted torque information; this embodiment does not specifically limit the preset dynamic function.

[0120] For example, the module for generating the expected torque information is integrated within the behavior processing unit. The calculation process can be implemented based on the optimal control framework or completed using conventional kinematics calculation methods. The expected torque information can be generated uniformly for all joints of the robotic arm, or it can be calculated independently for each joint and then aggregated and combined.

[0121] The current force information is real-time force data of the robotic arm collected by a six-dimensional force sensor, which can reflect the deviation between the theoretical torque and the actual force. The correction torque information is a compensation amount generated by comparing the current force information and the expected torque information. It is used to offset the torque error caused by external interference and model deviation, and to dynamically correct the expected torque information so that the actual output torque of the robotic arm conforms to the on-site working conditions.

[0122] After acquiring the current force information, the behavior processing unit compares and analyzes it with the expected torque information, identifies the difference between the two, and generates the corresponding correction torque information. The correction torque information can dynamically adjust the compensation range according to the error magnitude to achieve flexible error correction.

[0123] For example, the module for calculating the correction torque information and the module for generating the expected torque information are arranged serially within the behavior processing unit. Real-time force data can be processed by simple filtering before being used in the calculation. Error analysis and the generation of the correction torque can be achieved using residual learning or traditional feedback control, which can be flexibly selected according to the control accuracy requirements.

[0124] By integrating the expected torque information with the corrected torque information, the final robotic arm control command can be obtained. This command combines theoretical motion requirements with real-time force compensation results, simultaneously meeting multiple requirements such as motion following, terrain adaptation, and force control stability. The integration of torque information can be achieved through weighted superposition, or by switching between primary and secondary output components based on the operational scenario, differentiating the command generation logic between free motion and contact collaboration states. For example, the integrated robotic arm control command is formatted uniformly and stored in a distributed memory pool for easy access and execution by the microcontroller unit.

[0125] In this embodiment, expected torque information is generated based on terrain semantic parameters and motion intention information. Then, the current force information collected by the six-dimensional force sensor is combined to obtain corrected torque information to complete error compensation. Finally, the two types of torque information are integrated to generate control commands for the robotic arm. This can realize the combination of feedforward control and feedback correction, effectively compensate for control deviations caused by external interference, and improve the robustness and force control stability of the robotic arm motion control.

[0126] In other words, the behavior processing unit is responsible for fusing the predicted trajectory corresponding to the motion intention information and the control parameters corresponding to the terrain semantic parameters into the final control command for the robotic arm, and ensuring the accuracy and robustness of the control through a dual-channel residual decoupling mechanism. Dual-channel residual decoupling refers to the residual decoupling of the prediction channel and the correction channel. Specifically, the prediction channel generates feedforward control commands based on the predicted future trajectory corresponding to the motion intention information and the control parameters corresponding to the terrain semantic parameters, providing the main part of the control quantity; the correction channel generates feedback control commands based on the actual sensing feedback at the end of the robotic arm (including readings from the six-dimensional force sensor and the joint encoder), compensating for prediction errors and external disturbances. The outputs of the two channels are decoupled in the residual domain. The output of the prediction channel serves as the reference, and the output of the correction channel serves as the residual correction quantity. The two are added together to obtain the final control command. This dual-channel residual decoupling mechanism allows the prediction channel to focus on learning the user's motion intention pattern, while the correction channel focuses on compensating for dynamic disturbances. The two do not interfere with each other, improving the interpretability and debugging convenience of the control system.

[0127] For the prediction channel, based on motion intention information, the predicted trajectory of the user's upper limb end effector within the next 500 milliseconds is determined, and a linear quadratic regulator optimal control framework is used to generate feedforward control commands. The specific calculation process is as follows: the predicted trajectory is discretized into ten time steps, each with a 50-millisecond interval; the objective function is to minimize the position error of the robotic arm's end effector tracking the predicted trajectory, and the joint torque is used as the control input to solve a finite-time optimal control problem; in the obtained optimal control sequence, the control command for the first time step is immediately sent to the microcontroller for execution, and the control commands for the subsequent nine time steps are cached and re-solved and updated when the next control cycle arrives.

[0128] For the calibration channel, readings from the six-dimensional force sensor at the end of the robotic arm are received, and a feedback controller with a residual learning structure generates calibration control commands. The specific calculation process is as follows: the current force sensor reading, joint encoder reading, and the expected contact force output by the prediction channel are compared to obtain the force tracking error and position tracking error. The error signals are then input into a lightweight residual neural network, which takes the error as input and outputs calibration torque information. The training objective of the residual neural network is to enable the calibration torque information to compensate for the unmodeled dynamic characteristics and external disturbance torques of the prediction channel. After training, the calibration channel can quickly generate calibration torque information when human-machine physical contact occurs, converging the contact force error to an acceptable range.

[0129] The dual-channel fusion process employs learnable gating weights to dynamically fuse the outputs of the prediction and correction channels. Specifically, the fusion process is as follows: First, a current gating weight is calculated, ranging from zero to one. When the gating weight is close to one, the final control command primarily uses the prediction channel output, with the correction channel output serving only as a minor adjustment. This is suitable for the free movement phase before the user has physical contact with the robotic arm. When the gating weight is close to zero, the final control command primarily uses the correction channel output, with the prediction channel output serving only as a motion trend reference. This is suitable for the force-controlled collaborative phase where the user has already made physical contact with the robotic arm. The gating weights themselves are learned online by a two-layer fully connected network. The network's inputs include the amplitude of the six-dimensional force sensor readings, the confidence score of the predicted trajectory, and terrain semantic parameters; the output is the gating weight value.

[0130] Gating weight is a core parameter in dual-channel fusion, used to dynamically determine the mixing ratio of the predictive channel output and the corrective channel output in the final control command. When the gating weight is close to one, the final control command is approximately equal to the output of the predictive channel; when the gating weight is close to zero, the final control command is approximately equal to the output of the corrective channel.

[0131] The gating weights are not fixed preset values, but are calculated online in real time by a two-layer fully connected network. The network's inputs include the amplitude of the current force sensor reading (reflecting whether there is contact), the confidence score of the predicted trajectory (reflecting the reliability of the prediction), and the terrain category (reflecting the environmental context). The output is the gating weight value at the current moment. Therefore, the gating weights are adaptive fusion coefficients, enabling the system to smoothly transition between free movement and force-controlled cooperative modes.

[0132] The specific calculation process for the gating weights can be as follows: In each control cycle, three input features are extracted: the amplitude of the current force sensor reading, the confidence score of the predicted trajectory, and the terrain category. The amplitude of the current force sensor reading is the L2 norm of the six-dimensional force sensor reading, reflecting whether there is physical contact between the robot and the human, measured in Newtons. The confidence score of the predicted trajectory is the average of the attention weights output by the deep Transformer network used to predict the motion intention, reflecting the reliability of the current prediction result, and is between zero and one. The terrain category where the robot is currently located is extracted from the terrain semantic parameters output by the neural network processing unit, representing the environment type. These three features are then fed into a two-layer fully connected network via forward propagation as input vectors: the first layer maps the input to a hidden layer (e.g., 8 or 16 neurons) using a non-linear activation function (e.g., ReLU); the second layer maps the hidden layer output to a scalar, which is then compressed to the (0, 1) interval by a Sigmoid activation function; this scalar is the gating weight.

[0133] In this embodiment, the human-machine collaborative system comprises a first unit cluster, a second unit cluster, and a distributed memory pool. The first unit cluster acts as the "brain" of the robot control, responsible for high-level perception and understanding tasks. By combining environmental images and limb information, it can make the robot's movement more adapted to the environment. Introducing surface electromyography (EMG) signals allows for the detection of movement intentions before the user's muscles exert force, effectively improving the efficiency and accuracy of robot control. The second unit cluster acts as the "cerebellum" of the robot control, responsible for low-level real-time motion control. Zero-copy data transfer between the "brain" and "cerebellum" is achieved through the distributed memory pool, enabling low-latency data interaction and improving the robot's control efficiency.

[0134] Figure 5 A flowchart illustrating the robot control method, such as... Figure 5 As shown, a robot control method according to one embodiment of this application may include: Step 510: In response to the control command to the robot, the graphics processing unit and the neural network processing unit are divided into a first unit cluster, and the behavior processing unit and the microcontroller unit are divided into a second unit cluster. Step 520: Allocate a preset cache pool to the first unit cluster and allocate a preset physical channel from the shared bus to the second unit cluster; wherein, the preset cache pool is used to cache the data required by the graphics processing unit and the neural network processing unit, and the preset physical channel is used for data transmission between the behavior processing unit and the microcontroller unit.

[0135] Steps 510 and 520 are described in detail below.

[0136] In step 510, the graphics processing unit, neural network processing unit, behavior processing unit, and microcontroller unit are the core hardware units in the human-machine collaborative system, undertaking different computational and control functions. These four types of hardware units work together to complete environmental data processing, intelligent reasoning computation, motion command calculation, and low-level drive control, respectively, jointly supporting the operation of the entire robot control method. The first unit cluster is formed by the combination of the graphics processing unit and the neural network processing unit, mainly responsible for environmental perception, feature parsing, and intent prediction related computations. The second unit cluster is formed by the combination of the behavior processing unit and the microcontroller unit, mainly responsible for control command generation, command transmission, and robotic arm motion execution. This unit cluster division allows for the classification of hardware units according to function, facilitating targeted allocation of system resources and optimizing data interaction efficiency.

[0137] The system dynamically reconstructs the hardware interconnection architecture and storage resources based on the task by dividing the unit clusters according to the robot's control instructions and allocating corresponding transmission and storage resources to different unit clusters. This process completes resource allocation based on the task triggering mechanism, distinguishes the resource requirements of perception and computing units and control and execution units, and realizes on-demand allocation of system bus and cache resources.

[0138] For example, the logic for dividing unit clusters can be integrated into the system task scheduling module. The task scheduling module can group units according to the start signal of the job task. The grouped units maintain their original hardware connections and form functional clusters only at the logical level. The division of unit clusters can be completed at the same time as the task starts, or the cluster combination can be dynamically adjusted according to the task operation stage to adapt to the functional requirements of different job stages.

[0139] After the system powers on, it is configured by default in general computing mode, where all processing units communicate via a shared bus. When the system detects the initiation of a human-robot collaborative task, it responds to the robot control command. The task scheduler dynamically allocates interconnection channels according to the current task type, binding the graphics processing unit and neural network processing unit into a "perception enhancement cluster" (the first unit cluster), and binding the behavior processing unit and microcontroller unit into a "real-time control cluster" (the second unit cluster). After the robot control is completed, the system returns to the default general computing mode, and the first and second unit clusters are disbanded.

[0140] In step 520, the preset cache pool is a storage resource specifically allocated to the first unit cluster. It is used to temporarily cache various types of data generated and read during the operation of the graphics processing unit and the neural network processing unit. This improves the throughput of high-volume sensing data, reduces latency during data reading and retrieval, and ensures the smooth flow of data within the first unit cluster. The shared bus is a data transmission channel shared by all hardware units in the system by default, allowing various units to perform regular data interaction. Dividing the shared bus into a preset physical channel is a way to isolate and configure the transmission link. The preset physical channel is a dedicated data transmission link for the second unit cluster, specifically carrying the interactive data between the behavior processing unit and the microcontroller unit. This avoids data contention issues on the shared bus and ensures the real-time performance and stability of control command transmission.

[0141] In other words, after the system powers on, it is configured by default in general computing mode, with all processing units communicating via a shared bus. When the system detects the start of a human-machine collaborative task, the task scheduler dynamically allocates interconnect channels based on the current task type. It binds the graphics processing unit and the neural network processing unit into a "perception enhancement cluster," allocating additional cache to the graphics processing unit to improve the throughput of point cloud data. It binds the behavior processing unit and the microcontroller unit into a "real-time control cluster," allocating dedicated physical channels for control command transmission to avoid bus contention. Once the task is completed, the task scheduler releases the dynamically allocated interconnect channels and cache resources, and the system returns to the default general computing mode. The dynamic reconfiguration of the interconnect channels is completed at the hardware level, with reconfiguration latency controlled within microseconds.

[0142] In the dynamic reconfigurable interconnect channel mechanism, when the system detects the start of a human-machine collaborative task, the task scheduler allocates a dedicated physical channel for the transmission of control commands between the behavior processing unit and the microcontroller unit. This dedicated channel is used exclusively for low-latency control command transmission between the behavior processing unit and the microcontroller unit. Specifically, in the default general computing mode, all processing units communicate through a shared bus, leading to bus contention. After the robot control task starts, the task scheduler reconfigures the interconnect topology at the hardware level, isolating a dedicated physical channel (such as a dedicated on-chip interconnect line) from the shared bus and binding it to the transmission of control commands between the behavior processing unit and the microcontroller unit. This dedicated physical channel is used only for the transmission of control commands (joint angle sequences, motor drive signals, interrupt signals, etc.) to avoid contention with the high-bandwidth point cloud data communication between the graphics processing unit and the neural network processing unit, ensuring the determinism and low latency of control command transmission. Besides the dedicated physical channel for controlling the real-time cluster, the transmission between the graphics processing unit and the neural network processing unit still uses the shared bus, but the cache resources are enhanced.

[0143] Figure 6This is an architecture diagram of a human-machine collaborative system. The heterogeneous computing architecture of the brain and cerebellum is specifically designed for human-machine collaborative scenarios. Zero-copy data transfer between the four layers of algorithms (perception, understanding, decision-making, and control) is achieved through a distributed memory pool, and task-driven on-demand allocation of computing resources is realized through dynamically reconfigurable interconnect channels.

[0144] In this embodiment, by completing the clustering of functional units according to control instructions, and allocating a preset cache pool to the first unit cluster and a dedicated preset physical channel to the second unit cluster, dynamic on-demand configuration of computing and transmission resources can be achieved, reducing mutual interference between different types of data transmission and improving the overall system's data processing and instruction transmission efficiency.

[0145] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0146] It should be noted that in various specific embodiments of this application, when processing is required based on data related to the characteristics of the target object, such as target object attribute information or a set of attribute information, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require obtaining target object attribute information, separate permission or consent from the target object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of the embodiments of this application be obtained.

[0147] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 7 As shown, the electronic device 700 of this embodiment includes: at least one processor 701 ( Figure 7 (Only one is shown in the diagram), memory 702, and computer program 703 stored in said memory 702 and executable on said at least one processor 701, wherein the processor 701 executes said computer program 703 to implement the steps in any of the above robot control method embodiments.

[0148] Electronic device 700 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. This electronic device may include, but is not limited to, a processor 701 and a memory 702. Those skilled in the art will understand that... Figure 7 This is merely an example of electronic device 700 and does not constitute a limitation on electronic device 700. It may include more or fewer components than shown, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0149] The processor 701 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0150] In some embodiments, the memory 702 may be an internal storage unit of the electronic device 700, such as a hard disk or memory of the electronic device 700. In other embodiments, the memory 702 may be an external storage device of the electronic device 700, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 700. Furthermore, the memory 702 may include both internal and external storage units of the electronic device 700. The memory 702 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 702 can also be used to temporarily store data that has been output or will be output.

[0151] This application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0152] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0153] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0154] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0155] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0156] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0157] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0158] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0159] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A robot control method, characterized in that, The method is applied to a human-machine collaborative system, the system comprising a first unit cluster, a second unit cluster, and a distributed memory pool; the method includes: Acquire environmental images and user limb information; wherein, the limb information includes the user's limb movement information and surface electromyography signals; Based on the first unit cluster, the environmental image and the limb information are processed for perception to obtain terrain semantic parameters and the user's motion intention information, and the terrain semantic parameters and the motion intention information are stored in the distributed memory pool; wherein, the terrain semantic parameters represent the robotic arm control parameters adapted to the terrain; Based on the second unit cluster, the terrain semantic parameters and the motion intention information are obtained from the distributed memory pool, and the robot's robotic arm is controlled to move according to the terrain semantic parameters and the motion intention information.

2. The method according to claim 1, characterized in that, The first unit cluster includes a graphics processing unit and a neural network processing unit; The step of performing perceptual processing on the environmental image and the limb information based on the first unit cluster to obtain terrain semantic parameters and the user's motion intention information includes: Based on the graphics processing unit, the environmental image is converted into three-dimensional point cloud data, and the three-dimensional point cloud data is stored in the distributed memory pool; Based on the neural network processing unit, the three-dimensional point cloud data is obtained from the distributed memory pool, terrain semantic parameters are identified based on the three-dimensional point cloud data, and the user's movement intention information is predicted based on the limb information.

3. The method according to claim 2, characterized in that, The step of identifying terrain semantic parameters based on the three-dimensional point cloud data includes: The three-dimensional point cloud data is partitioned to obtain multiple sub-regions in the three-dimensional point cloud data; For each sub-region, identify the terrain category of the sub-region; The terrain semantic parameters are determined based on the terrain category of each sub-region and the coordinate range of each sub-region in the 3D point cloud data.

4. The method according to claim 3, characterized in that, The step of determining the terrain semantic parameters based on the terrain category of each sub-region and the coordinate range of each sub-region in the 3D point cloud data includes: Based on the terrain category of each sub-region and the coordinate range of each sub-region in the 3D point cloud data, a terrain semantic vector is generated; wherein, the terrain semantic vector represents at least one of the terrain category, ground friction coefficient, ground flatness, and ground slope in the environment; Based on the terrain semantic vector, the terrain semantic parameters are generated using a preset mapping function; wherein the terrain semantic parameters include at least one of the stiffness coefficient, damping coefficient, maximum output torque, and desired contact force threshold of each joint of the robotic arm.

5. The method according to claim 4, characterized in that, The step of generating a terrain semantic vector based on the terrain category of each sub-region and the coordinate range of each sub-region in the 3D point cloud data includes: Based on the neural network processing unit, the terrain category of each sub-region and the coordinate range of each sub-region in the three-dimensional point cloud data are stored in the distributed memory pool; Based on the graphics processing unit, the terrain category of each sub-region and the coordinate range of each sub-region in the 3D point cloud data are obtained from the distributed memory pool, and the terrain category of each sub-region and the coordinate range of each sub-region in the 3D point cloud data are mapped to the 3D point cloud data to obtain a hierarchical semantic map; wherein, the hierarchical semantic map represents 3D point cloud data containing terrain categories and corresponding coordinate ranges. Based on the graphics processing unit, the hierarchical semantic map is stored in the distributed memory pool; Based on the neural network processing unit, the hierarchical semantic map is obtained from the distributed memory pool, and the terrain semantic vector is extracted from the hierarchical semantic map.

6. The method according to claim 2, characterized in that, The user's limb is equipped with a data acquisition electrode and an inertial measurement unit. The data acquisition electrode is used to acquire surface electromyographic signals of the user's limb, and the inertial measurement unit is used to acquire motion information of the user's limb. The step of predicting the user's movement intention information based on the limb information includes: Acquire motion images of the user; wherein the motion images represent the position and posture of the user's limbs; Based on the limb information and the motion image, causal coding features are determined according to a preset joint causal graph; wherein, the causal coding features characterize the dependencies between the joints of the limbs moved by the user. Based on the causal coding features, the user's motion intention information is predicted.

7. The method according to claim 6, characterized in that, The preset joint cause-effect graph includes multiple joint nodes, and different joint nodes are connected by unidirectional edges; the method further includes: Acquire motion information, surface electromyography signals, and motion images within a preset time period; For each joint node, the causal parent node of the joint node is determined based on the motion information, surface electromyography signals, and motion images within the preset time period; wherein, the causal parent node represents the node that has an influence on the motion of the joint node; A unidirectional edge connection is made between the joint node and its causal parent node to obtain the preset joint causal graph; wherein the unidirectional edge connection is from the causal parent node of the joint node to the joint node.

8. The method according to claim 1, characterized in that, The second unit cluster includes a behavior processing unit and a microcontroller unit; The step of controlling the robot's robotic arm to move according to the terrain semantic parameters and the motion intent information includes: Based on the behavior processing unit, the terrain semantic parameters and the motion intention information are fused to obtain robotic arm control commands, and the robotic arm control commands are stored in the distributed memory pool. Based on the microcontroller unit, the robot arm control instructions are obtained from the distributed memory pool, and the robot arm is controlled to move according to the robot arm control instructions.

9. The method according to claim 8, characterized in that, The robotic arm is equipped with a six-dimensional force sensor; The process of fusing the terrain semantic parameters and the motion intent information to obtain robotic arm control commands includes: Based on the terrain semantic parameters and the motion intention information, expected torque information is generated; wherein, the expected torque information represents the joint torque of the robotic arm to be obtained. The current force information of the robotic arm is obtained through the six-dimensional force sensor, and a correction torque information is generated based on the current force information and the expected torque information; wherein, the correction torque information is used to compensate for the expected torque information. The robotic arm control commands are generated based on the expected torque information and the corrected torque information.

10. The method according to any one of claims 1-9, characterized in that, The human-machine collaborative system includes a graphics processing unit, a neural network processing unit, a behavior processing unit, and a microcontroller unit; the method further includes: In response to control commands to the robot, the graphics processing unit and the neural network processing unit are divided into a first unit cluster, and the behavior processing unit and the microcontroller unit are divided into a second unit cluster; A preset cache pool is allocated to the first unit cluster, and a preset physical channel is allocated from the shared bus to the second unit cluster; wherein, the preset cache pool is used to cache the data required by the graphics processing unit and the neural network processing unit, and the preset physical channel is used for data transmission between the behavior processing unit and the microcontroller unit.

Images