A heterogeneous layered SoC chip for robot bionic motion control and a computing method thereof

Abstract

A robot bionic motion control heterogeneous layered SoC chip and a computing method thereof, relate to the technical fields of heterogeneous computing chips and robot motion control, the chip integrates a graphic processing unit (GPU) and a spiking neural network (SNN) processor in a single SoC, and the SNN processor comprises a first computing area and a second computing area. The GPU runs a VLM generation task intention latent variable at a first frequency; the first computing area fuses and modulates the task intention latent variable and the proprioceptive signal through a spiking gated loop unit and a gated feature linear modulation circuit at a second frequency higher than the first frequency, to generate a motion latent variable; and the second computing area decodes the motion latent variable into a joint control signal at a third frequency higher than the second frequency as a spinal cord level. The three-level computing units run asynchronously and in parallel at different frequencies, and do not block each other, thereby solving the time domain mismatch problem.

Description

Technical Field

[0001] This application relates to the fields of heterogeneous computing chips and robot motion control technology, and in particular to a heterogeneous layered SoC chip for robot biomimetic motion control and its computing method. Background Technology

[0002] In the field of robot motion control, in order to achieve the unification of high-level semantic understanding and low-level action execution, while avoiding the problems of large system size, high communication latency and many failure points caused by the separate architecture, existing technologies usually adopt a centralized architecture with a single high-performance processor (such as a graphics processing unit GPU or a central processing unit CPU).

[0003] In a centralized architecture using a single high-performance processor, the vision-language-action (VLA) model and motion control program run on the same processor. The VLA model performs semantic reasoning on images and generates task action plans, a process that typically takes 100–500 milliseconds. The motion control loop, to ensure the smoothness and real-time performance of robot motion, typically provides instruction responses in milliseconds. Both are executed in a time-sharing or parallel manner on the same processor. However, the completion time of the semantic reasoning task is much longer than the requirements of the motion control cycle. This leads to a temporal mismatch problem, where the motion control loop cannot obtain timely updates matching its own frequency and can only passively wait for the output of semantic reasoning. This results in excessive motion control latency, motion stuttering, and severely degraded robot performance.

[0004] Therefore, how to effectively improve the temporal mismatch between large model semantic reasoning and real-time motion control while maintaining the advantages of single-chip integration is a key research topic for those skilled in the art. Summary of the Invention

[0005] In a first aspect, this application provides a heterogeneous layered SoC chip for biomimetic motion control of robots. The chip includes: a graphics processing unit (GPU) with a trained visual language model (VLM) deployed on it. The GPU is used to periodically perform inference based on user-inputted task instructions at a first frequency. In each inference cycle, the GPU runs the VLM based on the task instructions and the current visual input image to generate task intent latent variables. The task intent latent variables are semantic feature vectors abstracted by the VLM after understanding scene content and task intent; and a spiking neural network (SNN) processor, including a first computing area and a second computing area. The first computing area is connected to the GPU via an on-chip interconnect bus, and the second computing area is connected to the GPU via a first computing area and a second computing area. The first computing area is connected via an on-chip interconnect bus. The first computing area is equipped with a trained first spiking neural network. This first spiking neural network is used to acquire the task intent latent variable and the current body perception signal from the robot. Based on the task intent latent variable and the current body perception signal, it performs fusion modulation to periodically generate motion latent variables at a second frequency, which is higher than the first frequency. The second computing area is equipped with a trained second spiking neural network. This second spiking neural network is used to receive the motion latent variable and periodically decode it into control signals for driving robot joint movements at a third frequency, which is higher than the second frequency.

[0006] The heterogeneous layered SoC chip method for robot biomimetic motion control provided in this application has the following advantages: First, the chip operates at different frequencies through three levels of computing units (GPU, first computing unit, and second computing unit) without blocking each other. Semantic reasoning no longer blocks the motion control loop, which can avoid problems such as excessive control delay and motion stuttering caused by the passive waiting of motion control for semantic reasoning to be completed in the traditional centralized architecture.

[0007] On the other hand, the first computational region fuses and modulates the task intent latent variables output by the GPU and the current proprioceptive signal at a second frequency to generate motion latent variables, rather than directly mapping the GPU large model output to joint control. Since the output of the GPU large model has randomness and uncertainty, directly mapping it to joint acceleration would severely degrade smoothness. The fusion modulation in the first computational region effectively smooths the randomness in the GPU output, suppresses high-frequency 'intentional tremors,' and improves the smoothness of joint acceleration and jerk indicators. The second computational region decodes the motion latent variables into joint control signals at a third frequency, ensuring that the motion output both follows the high-level task intent and responds in real-time to changes in physical state.

[0008] On the other hand, the chip integrates the GPU and SNN processor into a single SoC, replacing the traditional multi-chip discrete solution, reducing system size, wiring and failure points, and making it suitable for embedded deployment in compact robots; at the same time, on-chip integration can eliminate external bus communication latency and protocol overhead, reducing the overall power consumption and cost of the system.

[0009] In some possible implementations, the first spiking neural network includes a Spiking-GRU (Spiking-Gated Recurrent Unit) and a Gated FiLM (Gated Feature Linear Modulation Circuit). Specifically, the first spiking neural network is used to periodically perform a motion latent variable generation process at the second frequency based on the Spiking-GRU and the Gated FiLM. The motion latent variable generation process in each cycle includes: acquiring the latest task intent latent variable generated by the GPU and acquiring the current body perception signal from the robot's body sensor; recursively encoding the current body perception signal and the hidden state of the previous time step based on the Spiking-GRU to generate the current hidden state h_t, where the initial hidden state is an all-zero vector; and performing the following three-way modulation parameter branch processing on h_t based on the Gated FiLM: Gated signal branch: h_t is input into the first fully connected layer, and a gated signal g_t is output after passing through a Sigmoid activation function. g_t is used to control the contribution ratio of each dimension of semantic information in the task intent latent variable z_sem to the motion latent variable, and g_t satisfies the formula g_t = The first fully connected layer is processed by the following three branches: sigmoid(W_g·h_t), where W_g is the weight matrix obtained by training the first fully connected layer; scaling factor branch: h_t is input into the second fully connected layer, and the output scaling factor γ_t is used to perform element-wise magnitude scaling on the latent variable of the task intent. γ_t satisfies the formula: γ_t = W_γ·h_t, where W_γ is the weight matrix obtained by training the second fully connected layer; offset branch: the state estimate h_t is input into the third fully connected layer, and the output offset β_t is used to adjust the baseline bias of the latent variable of the task intent. β_t satisfies the formula: β_t = W_β·h_t, where W_β is the weight matrix obtained by training the third fully connected layer; after the three-branch processing, the Gated FiLM is also applied based on the formula: z_mod = g_t⊙ (γ_t ⊙ z_sem + The task intent latent variable z_sem, the gate signal g_t, the scaling factor γ_t, and the offset β_t are fused and modulated element-wise to generate the motion latent variable z_mod, where ⊙ represents element-wise multiplication.

[0010] Using this approach, this embodiment recursively encodes the ontology-aware signal using a Spiking-GRU to generate a hidden state h_t. Then, a Gated FiLM circuit processes h_t through three branches—generating a gated signal g_t, a scaling factor γ_t, and an offset β_t. Finally, z_mod = g_t ⊙ (γ_t ⊙ z_sem + β_t) achieves adaptive fusion of the task intent latent variable and the ontology-aware state. Compared to fixed-weight feature fusion, this dynamic modulation mechanism can adaptively adjust the contribution ratio of semantic information to the motion output in each dimension according to the current motion state, enabling the motion output to respond to changes in physical state in real time while adhering to the high-level task intent.

[0011] In some possible implementations, the first spiking neural network in the first computational region is further specifically used to perform no less than two internal recursive modulations to progressively optimize the motion latent variables. In a new iteration, the first spiking neural network updates and generates a hidden state h_t based on the same ontology perception signal and the hidden state h_{t-1} generated in the previous iteration. It also updates the gating signal g_t, scaling factor γ_t, and offset β_t based on the newly generated hidden state h_t. Finally, it applies the new g_t, γ_t, and β_t to the motion latent variable z_mod output in the previous iteration to generate the motion latent variables for the current iteration.

[0012] This approach involves performing at least two internal recursive modulation iterations, using the motion latent variables output from the previous iteration as input for the new iteration, to regenerate the hidden state and three modulation parameters, thus achieving gradual refinement of the motion latent variables. Through this iterative refinement process, bidirectional interaction and iterative coordination are achieved between the high-level semantic intent and the low-level ontology motion state, ensuring that the final output motion latent variables simultaneously satisfy both high-level task constraints and low-level physical constraints. This further suppresses high-frequency jitter in the GPU output and improves the smoothness of the motion trajectory.

[0013] In some possible implementations, the first spiking neural network includes a pulse-gated recurrent unit (Spiking-GRU). The first spiking neural network is also used to perform collision prediction and preventative avoidance functions, which include: within each second frequency cycle, converting a force signal from a sensor into a frequency-modulated pulse sequence via a pulse frequency encoder. The pulse frequency f_pulse of the frequency-modulated pulse sequence is dynamically determined based on the external force amplitude, and f_pulse satisfies the formula: f_pulse = f_min + (f_max - f_min) · σ(F_ext / F_scale). The output pulse frequency f_pulse of the frequency-modulated pulse sequence is also dynamically determined based on the external force amplitude, and f_pulse satisfies the formula: f_pulse = f_min + (f_max - f_min) · σ(F_ext / F_scale). Here, f_min is the baseline frequency without external force, f_max is the maximum external force saturation frequency, F_ext is the external force amplitude, F_scale is the force scaling factor, and σ is the force scaling factor. The sigmoid function is used; the Spiking-GRU, as a forward inward model, receives the frequency modulation pulse sequence and the current joint state signal to predict the predicted motion trajectory q_hat for the next T time steps; the collision risk value collision_risk between q_hat and the obstacle obstacle in the environment is calculated based on the formula: collision_risk = max(0, 1 - min_distance(q_hat, obstacle) / safety_margin), where min_distance(q_hat, obstacle) is the minimum distance min_distance between q_hat and the obstacle, and safety_margin is a preset safety threshold; when the collision risk value is determined to be greater than 0, a bias term opposite to the direction of the external force is applied to the output layer that generates the motion latent variable z_mod to obtain the preventive adjustment motion latent variable, and the preventive adjustment motion latent variable is used as the output of the motion latent variable z_mod; the adjusted motion latent variable satisfies the formula: a_adjusted = a_original - α · collision_risk · F_ext_normalized, where a_original The corresponding component in the pre-adjustment motion latent variable z_mod is denoted as z_mod, the corresponding component in the post-adjustment motion latent variable is denoted as a_adjusted, α is the preset adjustment gain coefficient, and F_ext_normalized is the normalized external force direction vector.

[0014] This approach uses a Spiking-GRU as the forward-inward model, receiving force signal pulse frequency encoding and the current joint state to predict the motion trajectory over the next T time steps. It calculates the collision risk value and applies a bias term opposite to the direction of the external force when the collision risk is greater than 0, thus achieving preventative trajectory adjustment. Compared to purely reactive obstacle avoidance schemes, this predictive mechanism can proactively intervene before a collision occurs, causing the end effector to automatically deflect away from the source of the external force, significantly reducing the collision rate and collision force.

[0015] In some possible implementations, the chip further includes: an on-chip interconnect bus for connecting the GPU, the first computing area, and the second computing area respectively, and a shared on-chip memory coupled to the on-chip interconnect bus for providing unified address space access among the GPU, the first computing area, and the second computing area; the GPU is specifically used to write the task intent latent variable generated in each inference cycle into a first preset address area of ​​the shared on-chip memory; the first computing area is specifically used to read the task intent latent variable from the first preset address area of ​​the shared on-chip memory through a direct memory access channel established by the on-chip interconnect bus; the first computing area is also specifically used to write the motion latent variable finally generated in each cycle into a second preset address area in the shared on-chip memory; the second computing area is specifically used to read the motion latent variable from the second preset address area of ​​the shared on-chip memory through a direct memory access channel established by the on-chip interconnect bus.

[0016] In some possible implementations, the SNN processor further includes a hardwired secure reflection path, which is independent of the first computing area and the second computing area. This hardwired secure reflection path is used to: receive force output signals from preset sensors in an independent and parallel manner; determine whether the force output signal exceeds a preset safety threshold using a pulse frequency threshold comparator; if the force output signal exceeds the preset safety threshold, directly trigger a pre-coded protection action command to be output to the joint actuator, the pre-coded protection action command including a control signal that drives the robot joint to stop moving; and simultaneously generate an asynchronous wake-up interrupt signal, which is sent to the GPU via the on-chip interconnect bus to wake up the GPU for replanning.

[0017] This approach utilizes a direct memory access channel established through the on-chip interconnect bus and a unified address space access shared by on-chip memory. This allows the task intent latent variables generated by the GPU to be read in a zero-copy fashion by the first compute area, and the motion latent variables generated by the first compute area to be read in a zero-copy fashion by the second compute area. Zero-copy reading refers to on-chip direct memory reading without going through an external bus and without generating additional data copies. This mechanism avoids the latency and bandwidth consumption caused by multiple data copies between different processors in traditional solutions, further reducing the overall power consumption of the chip.

[0018] In some possible implementations, the chip is configured with a static hold mode, in which: the GPU enters a reduced-frequency sleep state and stops the inference operation of the VLM; the second spiking neural network in the second computing area is used to continuously output a basis control signal corresponding to the current posture to the joint actuator by utilizing the integral memory characteristics of the neuron membrane potential, so as to maintain the current joint posture of the robot.

[0019] In some possible implementations, the first frequency is a Hz frequency, the second frequency is a 100Hz frequency, and the third frequency is a kHz frequency.

[0020] In some possible implementations, the VLM, the first spiking neural network, and the second spiking neural network are obtained through end-to-end joint training. This end-to-end joint training includes: constructing a training dataset comprising: an operation scene image, task instruction text paired with the operation scene image, a sequence of ontology-aware signals synchronized with the image and instruction time, and expert-demonstrated joint motion trajectories as supervisory signals; training the VLM, the first spiking neural network, and the second spiking neural network based on the training dataset; constructing a loss function based on the error between the joint control signal output by the second spiking neural network and the expert-demonstrated joint motion trajectory during training; and simultaneously updating the parameters of the VLM, the first spiking neural network, and the second spiking neural network through gradient backpropagation, so that the end-to-end trained VLM, the first spiking neural network, and the second spiking neural network can work collaboratively to directly generate joint control signals consistent with the expert-demonstrated joint motion trajectory in response to input task instructions and visual images.

[0021] Secondly, this application also provides a computational method for a heterogeneous layered SoC chip for biomimetic motion control of a robot, characterized by comprising: periodically executing inference at a first frequency based on a graphics processing unit (GPU) and user-input task instructions; within each inference cycle, running a visual language model (VLM) based on the task instructions and the current visual input image to generate task intent latent variables, wherein the task intent latent variables are semantic feature vectors abstracted by the VLM after understanding scene content and task intent; acquiring the task intent latent variables and the current proprioception signal from the robot based on a first computing area in a spiking neural network (SNN) processor; fusing and modulating the task intent latent variables and the current proprioception signal to periodically generate motion latent variables at a second frequency, wherein the motion latent variables are feature vectors generated after fusing and modulating the task intent latent variables and the current proprioception signal, and the second frequency is higher than the first frequency; acquiring the motion latent variables based on a second computing area in the SNN processor; and periodically decoding the motion latent variables into control signals for driving robot joint movement at a third frequency, wherein the third frequency is higher than the second frequency.

[0022] It is understandable that the above-described computational method for the heterogeneous layered SoC chip for biomimetic motion control of robots is applicable. Therefore, the beneficial effects it can achieve can be found in the corresponding methods, and will not be repeated here. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the architecture of a heterogeneous layered SoC chip for robot biomimetic motion control provided in an embodiment of this application; Figure 2 This is a schematic flowchart of a method for generating motion latent variables provided in an embodiment of this application; Figure 3 This is a schematic flowchart of a method for implementing collision prediction and preventive avoidance functions using a Spiking-GRU, as provided in an embodiment of this application. Detailed Implementation

[0024] First, a detailed analysis of existing robot motion control schemes and their shortcomings is conducted to better understand the improvements in this application.

[0025] As mentioned in the background section, there are currently two main architectures for robot motion control systems.

[0026] Option A employs a centralized architecture with a single processor. In this architecture, the Vision-Language-Motion (VLA) model and the motion control program run on the same processor (e.g., GPU or CPU). The VLA model performs semantic reasoning on images and generates task action planning, a process typically taking 100–500 milliseconds. The motion control loop, to ensure the smoothness and real-time performance of robot movement, requires millisecond-level instruction responses. Both are executed in a time-sharing or parallel manner on the same processor. The completion time of the semantic reasoning task is much longer than the requirements of the motion control cycle, leading to a temporal mismatch problem. Specifically, the motion control loop cannot obtain timely updates matching its own frequency and can only passively wait for the output of semantic reasoning, resulting in excessive motion control latency and motion stuttering. Simultaneously, the output of the large model on the GPU has randomness and uncertainty, generating high-frequency "intention tremors," which directly map to joint acceleration, causing a severe decrease in smoothness and a deterioration in acceleration (jerk) metrics. Furthermore, when the robot is in a static position for extended periods (e.g., waiting after grasping an object), the GPU still needs to operate at full power to maintain neural network inference, resulting in unnecessary energy consumption.

[0027] Option B is a discrete heterogeneous architecture. In this architecture, a separate high-performance computer (such as an NVIDIA Jetson) runs the semantic model, while a separate motion controller (such as an MCU or DSP) performs joint control. The two communicate via an external bus (such as CAN, EtherCAT, or PCIe). This architecture introduces additional latency (typically greater than 1ms) between the GPU computer and the motion controller, compromising real-time performance. Furthermore, the discrete architecture results in a large system size, complex wiring, and numerous potential points of failure, making it unsuitable for embedded deployments in compact robots.

[0028] To address the issues of temporal mismatch, communication delay, high-frequency jitter, and power waste in the existing technologies, this application provides a heterogeneous hierarchical SoC chip and computing method for biomimetic motion control of robots. This chip integrates a GPU and an SNN processor within a single system-on-chip (SoC), forming a three-level biomimetic hierarchical architecture: the GPU, acting as the brain level, runs the VLM at a first frequency for semantic understanding, outputting latent variables of task intent; the first SNN computing area, acting as the cerebellum level, fuses and modulates the latent variables of task intent and proprioceptive signals at a second frequency using Spiking-GRU and Gated FiLM to generate motion latent variables; the second SNN computing area, acting as the spinal cord level, decodes the motion latent variables into joint control signals at a third frequency. The three levels of computing units operate asynchronously and in parallel at different frequencies without blocking or waiting for each other, solving the temporal mismatch problem at the architectural level; the three levels of computing units are directly connected via an on-chip interconnect bus, eliminating external bus communication delays; and the first computing area fuses and modulates the latent variables of task intent output by the GPU, effectively suppressing high-frequency jitter.

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described below in conjunction with the accompanying drawings.

[0030] Please see Figure 1 , Figure 1 This is an overall architecture block diagram of a heterogeneous layered SoC chip for robot biomimetic motion control provided in an embodiment of this application.

[0031] like Figure 1 As shown, the chip includes a graphics processing unit (GPU), a spiking neural network (SNN) processor, an on-chip interconnect bus, and shared on-chip memory.

[0032] The following describes the GPU and its configuration: The GPU contains a trained vision language model (VLM).

[0033] For example, VLM can be a BLIP-2 model or a multimodal large language model based on the Transformer architecture, deployed on the CUDA cores or Tensor cores of a GPU.

[0034] In some possible implementations, the GPU also integrates a Q-Former hardware acceleration unit, which is used to accelerate the cross-attention calculation of visual features and text features.

[0035] The GPU is used to perform inference periodically at a first frequency based on task instructions input by the user. In each inference cycle, the GPU runs the VLM based on the task instructions and the current visual input image to generate task intent latent variables. These task intent latent variables are semantic feature vectors abstracted by the VLM after understanding the scene content and task intent.

[0036] In some possible implementations, the first frequency is a Hz-level frequency (less than 100Hz), for example, the first frequency is 1Hz-20Hz (e.g., 5Hz, 10Hz or 15Hz).

[0037] As an example, the process of generating task intent latent variables using a GPU-based VLM is as follows: A VLM-based visual encoder extracts visual features from the visual input image, and a VLM-based language encoder extracts textual features from the task instructions. Next, a Q-Former hardware acceleration unit extracts multi-scale visual features from the intermediate layers of the visual encoder (e.g., layers 8-23). ​​Cross-attention calculation is performed between the multi-scale visual features and a learnable query vector, and then the multi-scale visual features are fused with the textual features to generate fused multimodal features. Finally, the multimodal features are dimensionality-reduced and compressed to output a low-dimensional task intent latent variable z_sem. For example, the dimension of z_sem is 32-256, such as 32, 64, 128, or 256.

[0038] It should be noted that the implementation of the Q-Former hardware acceleration unit is not limited to the above description. In other possible implementations, the Q-Former hardware acceleration unit may also extract features from different layers of the VLM (e.g., layers 6-20), or use different numbers of learnable query vectors (e.g., 16, 32, or 64 query vectors), as long as it can compress high-dimensional visual-linguistic features into low-dimensional intent latent variables.

[0039] The Q-Former hardware acceleration unit performs cross-attention computation between learnable query vectors and multi-scale visual features. This adaptively extracts the most task-relevant semantic information from the intermediate layers of the visual encoder, compressing high-dimensional visual features into low-dimensional intent latent variables while preserving task-related visual semantics to the greatest extent possible. Compared to directly using linear projection layers for feature compression, Q-Former achieves better alignment and fusion of visual and textual features, improving the accuracy of the VLM's understanding of scene content.

[0040] It should be noted that the fusion of visual and textual features can also employ alternative methods besides the Q-Former hardware acceleration unit. For example, a linear projection layer or a shallow MLP (Multilayer Perceptron) can be used to directly project the image features output by the visual encoder into the embedding space of the language model; or non-learning methods such as adaptive average pooling can be used to compress the spatial dimension of the visual feature map. This paper does not impose any restrictions on this approach.

[0041] The following describes the SNN processor and its computational partitions: The SNN processor includes a first computing area and a second computing area. The first computing area is connected to the GPU via an on-chip interconnect bus, and the second computing area is connected to the first computing area via an on-chip interconnect bus.

[0042] The first computation area in the SNN processor is equipped with a trained first spiking neural network. The first spiking neural network is used to acquire task intent latent variables and acquire current proprioceptive signals from the robot. Based on the task intent latent variables and the current proprioceptive signals, the network is fused and modulated to periodically generate motion latent variables at a second frequency.

[0043] In this embodiment, the motion latent variable is a feature vector generated by fusing and modulating the task intent latent variable and the current ontological perception signal.

[0044] In this embodiment, the second frequency is higher than the first frequency. For example, the first frequency is a Hz-level frequency, and the second frequency is a hundred-Hz-level frequency, that is, the second frequency is greater than or equal to 100Hz and less than 1000Hz.

[0045] The second computational region of the SNN processor is equipped with a trained second spiking neural network. This second spiking neural network receives motion latent variables and periodically decodes them into control signals for driving robot joint movements at a third frequency. The third frequency is higher than the second frequency; for example, the first frequency is in the Hz range, the second frequency is in the hundred Hz range, and the third frequency is in the kHz range (such as 1 kHz, 2 kHz, or 5 kHz).

[0046] In this embodiment, the SNN processor is a spiking neural network processor that adopts an event-driven architecture and consumes energy only when neurons generate pulses.

[0047] It should be noted that the first and second spiking neural networks are network structures obtained through offline training. After training, the weight parameters within the network model are permanently deployed in the SNN processor, and only forward impulse calculation is performed during the inference phase. The VLM, the first spiking neural network, and the second spiking neural network can be obtained through end-to-end joint training.

[0048] In some possible implementations, the chip further includes an on-chip interconnect bus for establishing direct memory access channels between the GPU and shared on-chip memory, between the first computing area and shared on-chip memory, and between the second computing area and shared on-chip memory. This allows each computing unit to directly read and write to the shared on-chip memory via the on-chip interconnect bus without the central processing unit (CPU) needing to participate in data transfer. The shared on-chip memory is coupled to the on-chip interconnect bus and provides unified address space access between the GPU, the first computing area, and the second computing area.

[0049] For example, the shared on-chip memory is divided into multiple preset address regions, including a first preset address region and a second preset address region. The GPU writes the task intent latent variables generated in each inference cycle into the first preset address region of the shared on-chip memory. The first compute region reads the task intent latent variables from the first preset address region of the shared on-chip memory through a direct memory access channel established by the on-chip interconnect bus. The first compute region writes the motion latent variables finally generated in each cycle into the second preset address region of the shared on-chip memory. The second compute region reads the motion latent variables from the second preset address region of the shared on-chip memory through a direct memory access channel established by the on-chip interconnect bus.

[0050] It should be noted that the method of dividing the preset address regions is not limited to the above description. In some other possible implementations, the shared on-chip memory can be divided into more address regions, for example, a third preset address region can be used to store the body sensing signal, and a fourth preset address region can be used to store intermediate calculation results, etc. Alternatively, the shared on-chip memory can also adopt a dynamic address allocation method, in which the memory management unit dynamically allocates address regions according to the data size.

[0051] By employing an on-chip interconnect bus to connect the GPU, the first computing area, the second computing area, and the shared on-chip memory, a direct memory access channel between each computing unit and the shared on-chip memory can be established. Data can flow efficiently between each computing unit and the shared on-chip memory without being transferred through the CPU, supporting parallel access from multiple masters and providing a communication foundation for the asynchronous parallel operation of the three levels of computing units at their respective natural frequencies. Simultaneously, unified address space access allows the task intent latent variables generated by the GPU to be read with zero copy by the first computing area, and the motion latent variables generated by the first computing area to be read with zero copy by the second computing area, fundamentally eliminating external bus communication latency and reducing system complexity.

[0052] It should be noted that in other possible implementations, other on-chip interconnect technologies or physical implementations can also be used for data transfer, such as on-chip networks, inter-chip interconnects, etc. This article does not limit these.

[0053] In existing centralized architectures with a single processor, semantic reasoning and motion control programs for large visual language models both run on the same GPU. Semantic reasoning tasks are extremely time-consuming (typically exceeding 100 ms), while the motion control loop must continuously update instructions at millisecond intervals to ensure smooth robot movements. This significant difference in their operating frequencies causes the motion control loop to be unable to receive new task instructions in a timely manner within each control cycle, as the next inference iteration of the large model has not yet been completed. The motion control loop is forced to wait for extended periods, resulting in noticeable stuttering and delays in the robot's movements. In other words, the slow updates of semantic reasoning directly block the high-speed requirements of motion control, creating a severe 'temporal mismatch'.

[0054] The heterogeneous layered SoC chip provided in this application can fundamentally break this mutually blocking relationship. The GPU periodically outputs task intent latent variable vectors at a lower first frequency (e.g., 5-10 Hz). The first computational region of the SNN operates autonomously at a second frequency (e.g., 500 Hz) much higher than the first frequency. During the long time window of hundreds of milliseconds between the GPU outputting the first intent latent variable vector and the output of the next intent latent variable vector, the second computational unit does not fall into any form of waiting or idleness. Instead, it continuously and without blocking merges the latest intent latent variable vector with the real-time received proprioception signal at the second frequency, constantly generating and outputting new motion latent variable vectors, thereby autonomously filling the gaps between high-level semantic instructions, making the robot's motion planning flow dense and continuous.

[0055] Furthermore, the second computational region of the SNN decodes each motion latent variable vector into joint-level control signals in real time at a third frequency (e.g., 2 kHz), significantly higher than the second frequency. This third frequency must be greater than the second frequency, rather than equal to it, because the motion latent variable vector can be understood as describing relatively macroscopic motion modulation information such as the desired end-effector state and compliance parameters, while the joint-level control signals require precise torque or position commands for each joint. There is a natural difference in information granularity and update rate between the two. Only by decoding at a higher frequency can the motion modulation signals be fully unfolded into smooth, jitter-free joint commands, suppressing torque pulsation and meeting the requirements of high-precision motion control.

[0056] Furthermore, based on the design where the third frequency is higher than the second frequency, the third computing unit can also undertake the critical task of the hardware safety reflection path. This hardware safety reflection path needs to complete a full hardware response from perception to execution quickly (e.g., within 20ms) after the sensor detects an abnormal external force. This response process includes pulse frequency threshold comparison, pre-coded protective action triggering, and emergency drive of the joint motor. If the third frequency is equal to the second frequency (e.g., both are 500 Hz), the instruction update cycle of the third computing unit is 2 ms. Within a 20 ms window, only a maximum of 10 instruction updates can be completed, making it difficult to precisely execute a smooth and safe emergency retreat trajectory, let alone simultaneously handle the decoding of normal motion commands and the rapid response to sudden safety events in a very short time. Increasing the third frequency to the kHz level allows for dozens of control command updates within the same 20 ms, ensuring not only the speed and stability of the safety reflection but also enabling normal motion control and safety monitoring to run in parallel without interference within the same computing area.

[0057] On the other hand, the first computational region fuses and modulates the task intent latent variables output by the GPU and the current proprioceptive signal at a second frequency to generate motion latent variables. Compared to directly mapping the GPU's large model output to joint control, the fusion modulation in the first computational region can smooth the randomness and uncertainty in the GPU output, effectively suppress high-frequency intentional jitter, and improve the smoothness of joint acceleration and jerk indicators. Experimental results show that, combined with the Gated FiLM mechanism and K≥2 iterations of refinement, high-frequency jitter can be reduced by more than 75%.

[0058] On the other hand, integrating the GPU and SNN processor into a single SoC replaces the traditional multi-chip discrete solution, reducing system size, wiring and failure points, and lowering system complexity, making it suitable for embedded deployment in compact robots.

[0059] In some possible implementations, the GPU, the first computing area, and the second computing area are directly connected via an on-chip interconnect bus. Data is transmitted within the chip without going through an external bus (such as PCIe, EtherCAT, etc.), avoiding the additional latency (usually greater than 1ms) caused by inter-chip communication in traditional discrete architectures. This can improve the real-time response capability of the system and eliminate external bus communication latency.

[0060] In some possible implementations, the first spiking neural network includes a Spiking-GRU (Pulse-Gated Recurrent Unit) and a Gated FiLM (Gated Feature Linear Modulation Circuit). The Spiking-GRU is a spiking neural network variant of the GRU, implementing selective state updates and resets through a pulse-gating mechanism. The Gated FiLM is a feature linear modulation circuit used to dynamically modulate semantic intent based on the current motion state.

[0061] The first spiking neural network, based on Spiking-GRU and Gated FiLM, periodically executes motion latent variables at a second frequency. (See [link to relevant documentation]). Figure 2 , Figure 2 The process of generating motion latent variables within each cycle during execution includes: S201, acquire the latent variables of the task intent and the current ontology perception signal.

[0062] Specifically, the first computing area obtains the latest task intent latent variable z_sem generated by the GPU, and obtains the current body perception signal from the robot's body sensors.

[0063] It should be noted that, because the second frequency is higher than the first frequency, within one inference cycle of the GPU, the first compute region executes multiple motion latent variable generation cycles. Within each second frequency cycle, the task intent latent variable read from the shared on-chip memory by the first compute region is the same task intent latent variable that the GPU has most recently generated and written in the current inference cycle. This task intent latent variable remains unchanged within the GPU's current inference cycle until the GPU generates and writes a new task intent latent variable in the next inference cycle. Thus, the task intent latent variable is updated at a low frequency (first frequency), while motion modulation in the first compute region is executed at a high frequency (second frequency), achieving decoupling of semantic inference and motion control on the time scale. In some possible implementations, current body sensing signals include joint angles and joint angular velocities acquired by joint encoders, and end-effector force and torque signals acquired by six-dimensional force sensors.

[0064] S202, based on Spiking-GRU, recursively encode the current ontology perception signal to determine the hidden state h_t at the current moment.

[0065] Specifically, the first spiking neural network, based on Spiking-GRU, recursively encodes the current ontology perception signal and the hidden state of the previous time step to generate the hidden state h_t at the current time step.

[0066] For example, the core recursive formula is: h_t = f(h_{t-1}, x_t), where h_{t-1} is the hidden state at the previous time step, and x_t is the input at the current time step (i.e., the current propagation sensing signal). The initial hidden state is a vector of all zeros; that is, the hidden state of the Spiking-GRU is initialized to a vector of all zeros when the chip powers on or at the start of each recursive task. During the recursive encoding in the first cycle (t=0), this vector of all zeros is used as the hidden state h_{-1} from the previous time step in the calculation.

[0067] It should be noted that the initial hidden state can be set in ways not limited to those described above. In other possible implementations, the initial hidden state can also be set to a preset value that is not all zero, such as an empirical value corresponding to the robot's initial pose, to accelerate convergence at the initial moment. This paper does not limit this approach.

[0068] S203 is based on the Gated FiLM for processing the three modulation parameter branches of the hidden state h_t.

[0069] Specifically, the first spiking neural network, based on Gated FiLM, performs the following three-way modulation parameter branching on the hidden state h_t: Branch 1 (Gated Signal Branch): h_t is input into the first fully connected layer, and the gated signal g_t is output after passing through the Sigmoid activation function. g_t satisfies the formula g_t = sigmoid(W_g·h_t), where W_g is the weight matrix obtained by training the first fully connected layer. Each element in the gated signal g_t takes a value in the range [0, 1], and is used to filter the latent variable z_sem of the task intent element by element. The closer g_t is to 1, the greater the contribution of the semantic information of that dimension to the motion latent variable; the closer the value is to 0, the smaller the contribution.

[0070] Branch 2 (Scaling Factor Branch): Input h_t into the second fully connected layer and output a scaling factor γ_t. γ_t satisfies the formula γ_t = W_γ·h_t, where W_γ is the weight matrix obtained by training the second fully connected layer. The scaling factor γ_t is used to scale the latent variable z_sem of the task intent element-wise to adjust the numerical range of z_sem to fit the motion space amplitude range of the current motion state.

[0071] Branch 3 (Offset Branch): Input h_t into the third fully connected layer and output offset β_t. β_t satisfies the formula β_t = W_β·h_t, where W_β is the weight matrix obtained by training the third fully connected layer. The offset β_t is used to provide a baseline bias for the motion latent variables that is independent of the task intent latent variable z_sem.

[0072] It should be noted that the implementation of each fully connected layer in the above three-way branching process is not limited to a single fully connected layer. In other possible implementations, each branch may also adopt a multi-layer fully connected network (e.g., two or three fully connected layers), or a multi-layer perceptron structure with activation functions (e.g., ReLU, tanh, etc.) to enhance nonlinear expressive power. This paper does not limit this approach.

[0073] S204, element-wise fusion modulation generates motion latent variables.

[0074] Specifically, after the three-branch processing, the first spiking neural network uses Gated FiLM to perform element-wise fusion modulation of the task intent latent variable z_sem, the gate signal g_t, the scaling factor γ_t, and the offset β_t based on the formula z_mod = g_t ⊙ (γ_t ⊙ z_sem + β_t) to generate the motion latent variable z_mod. Here, ⊙ represents element-wise multiplication.

[0075] The physical meaning of the above formula is as follows: the scaling factor γ_t first scales the latent variable z_sem of the task intent element-wise, then superimposes the offset β_t to form the affine transformation result, and finally the gating signal g_t controls the proportion of this affine transformation result transmitted to the motion latent variable z_mod element-wise. In scenarios where it is necessary to suppress high-level semantic intent (such as collision risk scenarios), the gating signal g_t approaches 0, thereby blocking the interference of semantic information on motion output and making motion control prioritize the underlying safety requirements.

[0076] It should be noted that the modulation method of Gated FiLM is not limited to the above formula. In other possible implementations, the gate signal g_t can also adopt other gating mechanisms. For example, the gate signal g_t can be directly multiplied element-wise with z_sem before performing an affine transformation: z_mod = γ_t ⊙ (g_t ⊙ z_sem) + β_t, that is, the gate signal directly controls the pass-through ratio of the semantic input. Alternatively, the gate signal g_t can also be used as the gate for the residual connection: z_mod = g_t ⊙ z_sem + (1 - g_t) ⊙ γ_t, realizing the dynamic mixing of semantic input and motion state input.

[0077] S205, internal recursive modulation.

[0078] Specifically, the first spiking neural network in the first computational region is also used to perform no less than two internal recursive modulations to progressively optimize the motion latent variables.

[0079] In the new iteration, the first spiking neural network updates and generates a new hidden state h_t based on the same ontology perception signal and the hidden state h_{t-1} generated in the previous iteration. It then updates the gate signal g_t, scaling factor γ_t, and offset β_t based on the newly generated hidden state h_t. Finally, it applies the new g_t, γ_t, and β_t to the motion latent variable z_mod output from the previous iteration to generate the motion latent variable for the current iteration. It should be noted that during the internal recursive modulation of the first spiking neural network, the ontology perception signal is sampled only once within a second frequency cycle, and the same ontology perception signal is used in each iteration. However, the hidden state of the Spiking-GRU is regenerated in each iteration, recursively transmitting temporal information. Through this method, internal recursive modulation maintains the consistency of the ontology perception signal while utilizing the temporal transmission capability of the recursive hidden state and the round-by-round update of semantic input to achieve iterative refinement of motion latent variables. Furthermore, in the iterations following the first modulation (t=0), the task intent latent variable z_sem or the motion latent variable z_mod (z_sem in the first modulation, z_mod in subsequent iterations) is replaced with the motion latent variable z_mod_prev from the previous iteration as semantic input, thereby enabling bidirectional interaction and iterative coordination between the high-level semantic intent and the low-level ontology motion state.

[0080] In some possible implementations, after generating the motion latent variable z_mod in each iteration, the first spiking neural network determines whether the current iteration count has reached the preset maximum iteration count K (K≥2). If the current iteration count has not reached K, the generated motion latent variable z_mod is used as the new input, and the recursive encoding of Spiking-GRU and the element-wise fusion modulation of GatedFiLM are repeatedly executed for the next iteration. If the current iteration count has reached K, the finally generated motion latent variable z_mod is output to the second computation area.

[0081] It should be noted that the number of iterations K is not limited to a fixed value. In some other possible implementations, the first spiking neural network can also be configured with an early termination condition—when the difference between the latent motion variables output by two adjacent iterations is less than a preset threshold, convergence is determined, the iteration ends early, and the current result is output to save power. Through the above iterative refinement process, the high-frequency jitter problem can be significantly improved.

[0082] In some possible implementations, the second spiking neural network is a deep spiking residual network that employs leaky integrate-and-fire (LIF) neurons.

[0083] For example, the specific process of decoding motion latent variables by the second spiking neural network includes: the second computation area reading the motion latent variable z_mod from a second preset address region of shared on-chip memory. The motion latent variable z_mod is input into a deep spiking residual network. The deep spiking residual network contains multiple layers of soft-reset LIF neurons (e.g., 3, 5, or 7 layers), and the layers transmit spiking signals through residual connections.

[0084] In each layer of soft-reset LIF neurons, the input pulse sequence is spatiotemporally integrated to update the membrane potential. The membrane potential update formula for soft-reset LIF neurons is: u_i(t) = β·u_i(t-1) + - s_i(t)·θ Where u_i(t) is the membrane potential of the i-th neuron at time t, β is the leakage attenuation coefficient (e.g., β=0.95), w_{ij} is the synaptic weight from the j-th neuron to the i-th neuron, s_j(t) is the input pulse of the j-th neuron at time t (takes a value of 0 or 1), the value of j is the range of all presynaptic neurons in the receptive field of the i-th neuron, s_i(t) is the output pulse of the i-th neuron at time t (takes a value of 0 or 1), and θ is the firing threshold.

[0085] When the membrane potential u_i(t) exceeds the firing threshold θ, the neuron fires a pulse (s_i(t)=1) and resets the membrane potential to the resting potential minus the firing threshold (i.e., soft reset: u_i(t) = u_i(t) - θ), maintaining the continuity of the membrane potential and the memory of historical information. When the membrane potential does not exceed the firing threshold, the neuron does not fire a pulse (s_i(t)=0), and the membrane potential gradually decays according to the leakage decay coefficient β.

[0086] The residual connection specifically involves adding the output pulse sequence of each soft-reset LIF neuron layer to its input pulse sequence element-by-element, and then using this sum as the input to the next soft-reset LIF neuron layer. Through the step-by-step pulse transmission across multiple soft-reset LIF neuron layers and the feature multiplexing of the residual connection, pulse-coded motion commands are generated.

[0087] The output layer of the deep impulse residual network uses non-reset integrator neurons. During integration, these neurons do not perform membrane potential reset operations, but continuously accumulate the membrane potential and directly convert it into a joint control signal with continuous amplitude. That is, the output of the non-reset integrator neuron is: y(t) = y(t-1) + Where y(t) is the amplitude of the output control signal at time t, y(t-1) is the amplitude of the output control signal at the previous time, w_i is the weight of the i-th input pulse, and s_i(t) is the value of the i-th input pulse at time t. The non-reset integral neuron continuously accumulates the pulse input from the last layer of the deep pulse residual network, directly mapping the accumulated membrane potential into continuous motion commands, which are then output to the joint actuator to drive the robot's joint movement.

[0088] It should be noted that the number of layers and structure of deep spiking residual networks are not limited to those described above. In other possible implementations, deep spiking residual networks can employ more layers (e.g., 5, 7, or 10 layers) to achieve stronger expressive power, and can also use different forms of residual connections (e.g., cross-layer residual connections, where the output of layer 1 is directly connected to the input of layer 3). In other possible implementations, deep spiking residual networks can also employ other types of spiking neuron models, such as hard-reset LIF neurons, Izhikevich neurons, or SRM neurons.

[0089] In some possible implementations, the first spiking neural network includes a Spiking-GRU, which is also used to perform collision prediction and preventative avoidance functions.

[0090] It should be noted that Spiking-GRU plays a recursive encoding role in the motion latent variable generation process and a forward inward model role in the collision prediction and preventive avoidance functions. Spiking-GRU has already acquired the current ontology perception signal and hidden state during the motion latent variable generation process, and the collision prediction and preventive avoidance functions can be executed synchronously based on this information.

[0091] The collision prediction and preventative avoidance function executes in each second frequency cycle, such as... Figure 3 As shown, the steps include the following.

[0092] S301, force signal pulse frequency encoding.

[0093] Specifically, the Spiking-GRU converts the force signal from the sensor into a frequency-modulated pulse sequence using a pulse frequency encoder.

[0094] Among them, the pulse frequency f_pulse corresponding to the frequency-modulated pulse sequence (the frequency of the force-encoded pulse sequence ultimately fed into the SNN cerebellar module) is dynamically determined based on the external force amplitude: f_pulse = f_min + (f_max - f_min)·σ(F_ext / F_scale) Where, f_min is the baseline frequency without external force, the base pulse frequency received by the SNN when the external force is zero, maintaining a minimum force sensing channel; f_max is the maximum external force saturation frequency, the upper limit of the pulse frequency when the external force is extremely large, preventing the SNN from being overwhelmed by excessively high frequency pulses; F_ext is the external force amplitude, the magnitude of the force signal from the wrist six-dimensional force sensor (scalar, taking the resultant force amplitude); F_scale is the force scaling factor, the reference force value used for normalization, selected near the normal working force range of the robot. Physical meaning: When F_ext = F_scale, σ(1) ≈ 0.73, the pulse frequency is approximately in the middle range; σ is the Sigmoid function.

[0095] Therefore, the larger the external force amplitude F_ext, the higher the output pulse frequency f_pulse, and the stronger the Spiking-GRU's response to the force sensor signal.

[0096] Specifically, the analog voltage signal output by the force sensor is first normalized to the external force amplitude in the range of [0,1] after anti-aliasing filtering and analog-to-digital conversion. Then, the pulse frequency f_pulse is calculated according to the above mapping relationship. The pulse generation circuit in the pulse frequency encoder generates a pulse sequence with this pulse frequency and sends it to the SNN.

[0097] S302, Spiking-GRU forward model predicts trajectory.

[0098] Specifically, Spiking-GRU, as a forward inward model, is used to receive frequency-modulated pulse sequences and current joint state signals (including joint angles and joint angular velocities). Through its recursive pulse network structure, it extrapolates forward step by step to predict the predicted motion trajectory q_hat for the next T time steps.

[0099] In this embodiment, the forward-inward model is implemented using a Spiking-GRU recurrent pulse network to predict the future trajectory of the robotic arm in real time and perform preventative collision avoidance. This model only receives two types of proprioceptive signals as input: current joint state signals, including joint angles and angular velocities of each joint, acquired and provided by a joint encoder at a frequency of at least 500Hz; and external force sensor signals, acquired by a six-dimensional force sensor installed at the end effector of the robotic arm, and converted into a frequency-modulated pulse sequence by a pulse frequency encoder before input.

[0100] For example, the prediction process includes: at the beginning of each prediction cycle, the forward inward model simultaneously feeds the current joint state and the external force signal pulse sequence into the input layer of Spiking-GRU, transforming them into an internal state representation in the pulse domain. The network initiates an internal loop iteration process with a step size of 1 millisecond. In each iteration, Spiking-GRU updates its internal hidden state through a gated recursive mechanism based on the predicted joint state value from the previous time step and the continuously input external force pulse sequence, and outputs the predicted joint angle and angular velocity for that time step. This predicted value will serve as the input state for the next time step, thereby achieving step-by-step deduction in the time dimension. The above iterations are repeated T times (e.g., T=50) to generate a predicted motion trajectory q_hat containing the joint angles and angular velocities for the next 50 milliseconds.

[0101] It should be noted that the prediction window T is not limited to 50ms. In other possible implementations, the prediction window T can be adjusted according to the application scenario. For example, it can be set to 20ms or 30ms for high-speed motion scenarios, and 80ms or 100ms for low-speed, high-precision operation scenarios. Correspondingly, the length of each time step can also be adjusted according to the control frequency, such as 0.5ms or 2ms per step. The number of hidden neurons in Spiking-GRU can be, but is not limited to, 64, and can be set to 32, 128, or 256 depending on the computational resources and prediction accuracy requirements.

[0102] S303, Collision Risk Assessment.

[0103] Specifically, the first calculation area calculates the collision risk value `collision_risk` between the predicted trajectory `q_hat` and the obstacle in the environment based on the formula `collision_risk = max(0, 1 - min_distance(q_hat, obstacle) / safety_margin)`. Here, `min_distance(q_hat, obstacle)` is the minimum distance between the predicted trajectory `q_hat` and the obstacle, `safety_margin` is a preset safety threshold (e.g., 5cm), and `collision_risk` ∈ [0, 1].

[0104] It should be noted that the calculation method for collision risk is not limited to the formula mentioned above. In other possible implementations, collision risk can also be calculated using other forms of functions, such as the exponentially decaying function `collision_risk = exp(-min_distance / safety_margin)`, or a piecewise linear function, as long as the minimum distance between the predicted trajectory and the obstacle can be mapped to a risk value within the interval [0, 1]. The preset safety threshold `safety_margin` can also be adjusted according to the application scenario; for example, it can be set to 10cm for precision operation scenarios and 2cm for large-scale movement scenarios.

[0105] S304, Preventive trajectory adjustment.

[0106] If the collision risk value is determined to be greater than 0, the first computational region applies a bias term opposite to the direction of the external force to the output layer that generates the motion latent variable z_mod, resulting in a preventatively adjusted motion latent variable. This preventatively adjusted motion latent variable is then used as the output of the motion latent variable z_mod. For example, the adjusted motion latent variable satisfies the formula: a_adjusted = a_original - α·collision_risk·F_ext_normalized; Where a_original represents the corresponding component in the motion latent variable z_mod before adjustment, a_adjusted represents the corresponding component in the motion latent variable after adjustment, α is the preset adjustment gain coefficient (e.g., α=0.3), and F_ext_normalized is the normalized external force direction vector (i.e., the direction unit vector obtained after normalizing the external force F_ext detected by the six-dimensional force sensor). The bias term is directed to make the end effector move away from the source of the external force, and its magnitude is controlled by α·collision_risk. The higher the collision risk, the larger the bias magnitude.

[0107] In the embodiments of this application, the bias term is applied just before the collision risk is predicted but before a rigid collision actually occurs, so that the robotic arm can change its trajectory in advance.

[0108] The first computational area outputs the preventatively adjusted motion latent variable as the motion latent variable z_mod to the corresponding preset address area so that the second computational area can read it.

[0109] It should be noted that the value of the gain coefficient α is not limited to 0.3. In some other possible implementations, α can be calibrated according to the response sensitivity of the robot joints; for example, it can be set to 0.1 for high-sensitivity joints and 0.8 for low-sensitivity joints. In other possible implementations, the adjustment formula can also use a nonlinear mapping, for example... This allows for stronger adjustments to be made in high-risk areas.

[0110] In some possible implementations, the SNN processor also includes a hardwired secure reflection path, which is independent of the first and second computational areas and implemented using dedicated hardware logic circuitry. The pulse frequency threshold comparator, the triggering logic for the precoded protection action instruction, and the generation logic for the asynchronous wake-up interrupt signal within this hardwired secure reflection path are all embedded in the dedicated hardware logic circuitry. The hardwired secure reflection path does not depend on the operating state of the first and second spiking neural networks and cannot be disabled or bypassed by software.

[0111] This hardwired security reflection path is used to perform the following steps: 1) Parallel reception of sensor signals.

[0112] The hardwired safety reflection path receives force output signals from preset sensors (such as force sensors or tactile sensors) in an independent and parallel manner. These force output signals are either analog voltage signals or digital pulse signals.

[0113] 2) Pulse frequency threshold comparison.

[0114] The hardwired safety reflection path uses a pulse frequency threshold comparator to detect whether the output signal exceeds a preset safety threshold. The pulse frequency threshold comparator converts the sensor signal into a pulse frequency and compares it with the preset frequency threshold. When the amplitude of the sensor signal exceeds the preset safety threshold, its corresponding pulse frequency also exceeds the preset frequency threshold.

[0115] 3) Trigger the pre-coded protection action command.

[0116] If a sensor signal exceeds a preset safety threshold, the hardwired safety reflection path directly triggers a pre-coded protective action command to be output to the joint actuator. The pre-coded protective action command includes control signals that stop the robot joints (e.g., emergency stop or emergency retreat commands). The end-to-end delay from sensor signal input to protective action command output is less than 20ms.

[0117] 4) Generate asynchronous wake-up interrupt.

[0118] Simultaneously with triggering the precoded protection action instruction, the hardwired security reflection path generates an asynchronous wake-up interrupt signal, which is sent to the GPU via the on-chip interconnect bus to wake up the GPU for replanning.

[0119] It should be noted that the types of precoded protective action commands triggered by the hard-wired safety reflection path are not limited to emergency stop or emergency retreat. In other possible implementations, precoded protective action commands can also include different protective actions such as joint locking, impedance control switching, and force limiting. The specific type of protective action can be triggered in stages based on the type and amplitude of the sensor signal—for example, force limiting is triggered when the threshold is slightly exceeded, and emergency stop is triggered when the threshold is severely exceeded.

[0120] In this embodiment, the aforementioned hardwired safety reflection path and the collision prediction and preventative avoidance functions of the first computing area constitute a two-stage "prediction-response" safety chain. Stage one is the prediction and prevention stage, executed by the first computing area in each second frequency cycle. When the collision risk value (collision_risk) between the predicted trajectory and the obstacle is > 0, a preventative bias adjustment is actively applied to mitigate potential collisions. The end-to-end delay is less than 5ms, used to mitigate approximately 99% of potential collisions. Stage two is the emergency response stage. When the preventative adjustment in stage one fails to prevent an actual collision, the hardwired safety reflection path can respond to the actual contact force signal within less than 20ms and execute an emergency action. Stage one and stage two run in parallel, with stage two independent of stage one and the GPU.

[0121] In some possible implementations, the chip is configured with a static hold mode. The chip enters static hold mode when the robot maintains the same posture for an extended period (e.g., after grasping an object and waiting). In static hold mode: The GPU enters a reduced-frequency sleep state, ceasing VLM inference operations and stopping the generation of new task intent latent variables, thereby significantly reducing power consumption. Additionally, the second spiking neural network in the second computational region utilizes the integral memory characteristics of neuronal membrane potentials to continuously output basis control signals corresponding to the current posture to the joint actuators, maintaining the robot's current joint posture.

[0122] Specifically, in the second spiking neural network, the membrane potential of the soft-reset LIF neurons maintains the pulse firing pattern corresponding to the current posture with a preset decay coefficient (e.g., β=0.95). When there is no external input pulse, the membrane potential gradually decays according to the preset decay coefficient but retains the memory of the historical state. In static hold mode, the second computational region does not need to receive new motion latent variable inputs, but only relies on the integral memory characteristics of the membrane potential to continuously output the basis control signal.

[0123] It should be noted that the chip's operation in static holding mode is not limited to the above description. In other possible implementations, the first spiking neural network in the first computation area can also continuously receive the body perception signal at a monitoring frequency lower than the second frequency (e.g., 10-50Hz). When the body perception signal deviates from the reference value corresponding to the current posture (e.g., external force disturbance causes a joint angle shift), a corrected motion latent variable is generated based on the deviation. The second computation area decodes the corrected motion latent variable into a corrected joint control signal and outputs it to the joint actuator to resist external force disturbance and restore the current joint posture. This method achieves closed-loop posture maintenance with extremely low power consumption.

[0124] For example, in static hold mode, the SNN processor's pulse firing rate is less than 1Hz, and the chip's total power consumption is less than 50mW. It should be noted that the power consumption target is not limited to less than 50mW. In other possible implementations, power consumption can be adjusted according to the chip manufacturing process and voltage regulation; for example, it can be less than 10mW for advanced process chips and less than 100mW for mature process chips.

[0125] In some possible implementations, the aforementioned VLM, first spiking neural network, and second spiking neural network are obtained through end-to-end joint training. End-to-end joint training includes: constructing a training dataset, which includes: operation scene images, task instruction text paired with the operation scene images (e.g., "Place the wine bottle on the cabinet"), a sequence of ontology-aware signals synchronized with the images and instructions in time, and expert-demonstrated joint motion trajectories as supervision signals. The expert-demonstrated joint motion trajectories are acquired through manual remote control operation or teaching. The VLM, first spiking neural network, and second spiking neural network are trained based on the training dataset. During training, a loss function is constructed using the error between the joint control signals output by the second spiking neural network and the expert-demonstrated joint motion trajectories. By simultaneously updating the parameters of VLM, the first spiking neural network, and the second spiking neural network (including the weight matrix W_g of the first fully connected layer, the weight matrix W_γ of the second fully connected layer, the weight matrix W_β of the third fully connected layer, and all synaptic weights of Spiking-GRU and the deep spiking residual network) through gradient backpropagation, the VLM, the first spiking neural network, and the second spiking neural network trained end-to-end can work together to directly generate joint control signals consistent with the joint motion trajectory demonstrated by the expert in response to the input task instructions and visual images.

[0126] Specifically, in one training iteration, the operation scene image and task instruction text are input into the VLM, and the VLM outputs the task intent latent variable. The task intent latent variable and the time-synchronized ontology perception signal sequence are input into the first spiking neural network, and the first spiking neural network outputs the motion latent variable sequence. The motion latent variable sequence is input into the second spiking neural network, and the second spiking neural network outputs the joint control signal sequence. The mean squared error or cross-entropy loss between the joint control signal sequence and the expert-demonstrated joint motion trajectory is calculated, and the gradient is calculated using the backpropagation through time (BPTT) algorithm. The parameters of the VLM, the first spiking neural network, and the second spiking neural network are updated simultaneously.

[0127] It should be noted that model training is not limited to the end-to-end joint training method described above. In other possible implementations, a staged training approach can also be adopted: first, pre-train the VLM, the first spiking neural network, and the second spiking neural network separately, and then perform joint fine-tuning. The loss function mentioned above is not limited to mean squared error or cross-entropy loss; it can also include regularization terms such as joint velocity smoothness loss and acceleration constraint loss to improve the smoothness and safety of the motion trajectory.

[0128] The Spiking-GRU forward inward model described above can also be obtained through imitation learning training in an offline simulation environment. After training, the weights are permanently deployed to the first computation area of ​​the SNN processor, eliminating the need to maintain explicit dynamic model parameters at runtime. The model's input relies solely on joint encoder and force sensor signals, without depending on external vision or depth sensors, ensuring low latency and high reliability of the collision prediction path.

[0129] It should be noted that the training method for the Spiking-GRU forward inward model is not limited to imitation learning in an offline simulation environment. In other possible implementations, the forward inward model can also be continuously updated and adapted on a real robot through online learning.

[0130] After training, the weights of the VLM are deployed in the GPU, while the weights of the first spiking neural network (including the synaptic weights of the Spiking-GRU and the weight matrices W_g, W_γ, and W_β of the Gated FiLM) and the weights of the second spiking neural network (including all the synaptic weights of the deep spiking residual network) are deployed in the SNN processor. After deployment, the chip does not need to perform gradient calculations or parameter updates during the inference phase; it only performs forward impulse calculations and matrix multiplications, ensuring real-time inference and low power consumption.

[0131] Based on the above chip architecture, this application embodiment also provides a heterogeneous layered SoC chip computing method for robot bionic motion control, including: periodically executing inference at a first frequency based on a graphics processing unit (GPU) and user-input task instructions; within each inference cycle, running a visual language model (VLM) based on the task instructions and the current visual input image to generate task intent latent variables, where the task intent latent variables are semantic feature vectors abstracted after understanding scene content and task intent through the VLM; acquiring the task intent latent variables and the current body perception signal from the robot based on a first computing area in a spiking neural network (SNN) processor; fusing and modulating the task intent latent variables and the current body perception signal to periodically generate motion latent variables at a second frequency, where the motion latent variables are feature vectors generated after fusing and modulating the task intent latent variables and the current body perception signal, and the second frequency is higher than the first frequency; acquiring the motion latent variables based on a second computing area in the SNN processor; and periodically decoding the motion latent variables into control signals for driving robot joint movement at a third frequency, where the third frequency is higher than the second frequency.

[0132] In some possible implementations, the first spiking neural network includes a Spiking-GRU (Spiking-Gated Recurrent Unit) and a Gated FiLM (Gated Feature Linear Modulation Circuit). The method further includes: periodically executing a motion latent variable generation process at the second frequency based on the Spiking-GRU and the Gated FiLM, wherein the motion latent variable generation process in each cycle includes: acquiring the latest task intent latent variable generated by the GPU and acquiring the current body perception signal from the robot's body sensor; recursively encoding the current body perception signal and the hidden state of the previous time step based on the Spiking-GRU to generate the current hidden state h_t, where the initial hidden state is an all-zero vector; and performing the following three-way modulation parameter branch processing on h_t based on the Gated FiLM: Gated signal branch: h_t is input into the first fully connected layer, and a gated signal g_t is output after passing through a Sigmoid activation function. g_t is used to control the contribution ratio of each dimension of semantic information in the task intent latent variable z_sem to the motion latent variable, and g_t satisfies the formula g_t = The first fully connected layer is processed by the following three branches: sigmoid(W_g·h_t), where W_g is the weight matrix obtained by training the first fully connected layer; scaling factor branch: h_t is input into the second fully connected layer, and the output scaling factor γ_t is used to perform element-wise magnitude scaling on the latent variable of the task intent. γ_t satisfies the formula: γ_t = W_γ·h_t, where W_γ is the weight matrix obtained by training the second fully connected layer; offset branch: the state estimate h_t is input into the third fully connected layer, and the output offset β_t is used to adjust the baseline bias of the latent variable of the task intent. β_t satisfies the formula: β_t = W_β·h_t, where W_β is the weight matrix obtained by training the third fully connected layer; after the three-branch processing, the Gated FiLM is also applied based on the formula: z_mod = g_t ⊙ (γ_t ⊙ z_sem + ... The task intent latent variable z_sem, the gate signal g_t, the scaling factor γ_t, and the offset β_t are fused and modulated element-wise to generate the motion latent variable z_mod, where ⊙ represents element-wise multiplication.

[0133] It should be noted that the above method may also include collision prediction and preventative avoidance steps, executed by the first computation area in the SNN processor, with specific implementation details described above. The above method may also include a hardwired safe reflection step, executed independently and in parallel with the first and second computation steps by the hardwired safe reflection path, with specific implementation details described above. The above method may also include a static holding step, executed when the chip enters static holding mode, with specific implementation details described above. In the hierarchical computation method, the first, second, and third frequencies increase sequentially, and the three computation steps are executed asynchronously and in parallel, without blocking or waiting for each other, with specific implementation details described above.

[0134] The following provides some application scenario examples.

[0135] Humanoid robot dual-arm operation: The robot uses its two arms to complete the task of "placing a wine bottle on a cabinet". A vision sensor acquires RGB images, the GPU runs a pre-trained VLM (e.g., BLIP-2+LLM), and the Q-Former hardware acceleration unit outputs a 128-dimensional latent intention variable z_sem at a frequency of 5-10Hz. The first computation area reads the latent intention variable z_sem from shared on-chip memory at a frequency of 500Hz, while simultaneously receiving proprioceptive signals from the joint encoder and six-dimensional force sensor, performing K=2 iterations to refine and generate the motion latent variable z_mod. The second computation area reads the motion latent variable z_mod at a frequency of 2kHz, runs a deep impulse residual network (3 layers of soft-reset LIF neurons), and outputs continuous joint commands to the joint actuators. When the robot enters a static hold state after completing the grasping action, the GPU reduces its frequency to sleep, the pulse rate drops to <1Hz, and the total chip power consumption is <50mW. In the event of a collision, the hard-wired safety reflection path outputs a retreat command within <20ms, simultaneously waking up the GPU for replanning.

[0136] The quadruped robot's dynamic walking mechanism utilizes a lightweight navigation model executed only by the GPU, with latent variables compressed to 64 dimensions. The first computational area primarily processes the fused signals from the IMU and joint encoders. The second computational area contains four independent neural nuclei, each controlling the joint movement of one leg, with each leg independently triggering a reflex.

[0137] Collision prediction and preventative avoidance in dynamic obstacle environments: A robotic arm moves a target object within a work area, while another robot (an obstacle) approaches from the side. A Spiking-GRU forward-inward model with 64 hidden neurons and a prediction window of T=50ms is used. Taking the current joint state and six-dimensional force sensor signals as input, it outputs a predicted trajectory for the next 50ms. Preventative trajectory adjustment is triggered when `collision_risk > 0`. Approximately 99% of potential collisions are preventatively mitigated during the prediction phase.

[0138] Low-cost, streamlined configuration: The GPU is replaced with a lightweight neural network processing unit (NPU), and the first and second computing areas are merged into a single SNN processor. The lightweight NPU runs a lightweight VLM model, outputting 32-dimensional latent variables representing the task intent. This configuration is suitable for applications with low computational requirements and cost sensitivity.

[0139] Fault Degradation Operation: When the GPU fails, the first and second computing areas autonomously maintain safe movement by utilizing the membrane potential memory of the spiking neural network, continuously outputting the base control signal corresponding to the current posture, maintaining the robot's current joint posture, and preventing the robot from being in danger due to loss of control signal.

[0140] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A heterogeneous layered SoC chip for biomimetic motion control of robots, characterized in that, The chip includes: A graphics processing unit (GPU) is deployed with a trained visual language model (VLM). The GPU is used to periodically perform inference based on user-input task instructions at a first frequency. In each inference cycle, the VLM is run based on the task instructions and the current visual input image to generate task intent latent variables. The task intent latent variables are semantic feature vectors abstracted by the VLM after understanding the scene content and task intent. A spiking neural network (SNN) processor includes a first computing area and a second computing area. The first computing area is connected to the GPU via an on-chip interconnect bus, and the second computing area is connected to the first computing area via an on-chip interconnect bus. The first computing area is equipped with a trained first spiking neural network. The first spiking neural network is used to acquire the task intent latent variable and the current body perception signal from the robot. Based on the task intent latent variable and the current body perception signal, the network is fused and modulated to periodically generate motion latent variables at a second frequency, which is higher than the first frequency. The second computing area is equipped with a trained second spiking neural network, which receives the motion latent variables and periodically decodes them into control signals for driving robot joint movements at a third frequency, which is higher than the second frequency.

2. The chip as described in claim 1, characterized in that, The first spiking neural network includes a Spiking-GRU (Spiking-Gated Recurrent Unit) and a Gated Feature Linear Modulation (Gated FiLM) circuit. Specifically, the first spiking neural network is used to periodically perform a motion latent variable generation process at a second frequency based on the Spiking-GRU and the Gated FiLM. The motion latent variable generation process within each cycle includes: Obtain the latest task intent latent variables generated by the GPU, and obtain the current body perception signals from the robot's body sensors; Based on the Spiking-GRU, the current ontology perception signal and the hidden state of the previous time step are recursively encoded to generate the hidden state h_t of the current time step, wherein the initial hidden state is an all-zero vector. Based on the Gated FiLM, the h_t is processed using the following three-way modulation parameter branching: Gated signal branch: h_t is input into the first fully connected layer, and a gated signal g_t is output through the Sigmoid activation function. g_t is used to control the contribution ratio of each dimension of semantic information in the latent variable z_sem of the task intent to the latent variable of motion. g_t satisfies the formula g_t = sigmoid(W_g·h_t), where W_g is the weight matrix obtained by training the first fully connected layer. Scaling factor branch: The h_t is input into the second fully connected layer and the scaling factor γ_t is output. The γ_t is used to scale the latent variable of the task intent element by element. The γ_t satisfies the formula: γ_t = W_γ· h_t, where W_γ is the weight matrix obtained by training the second fully connected layer. Offset branch: The state estimate h_t is input into the third fully connected layer and the offset β_t is output. The offset β_t is used to adjust the baseline bias of the latent variable of the task intention. The β_t satisfies the formula: β_t = W_β·h_t, where W_β is the weight matrix obtained by training the third fully connected layer. After the three-way branch processing, the motion latent variable z_mod is generated by element-wise fusing and modulating the task intent latent variable z_sem, the gate signal g_t, the scaling factor γ_t, and the offset β_t based on the formula: z_mod = g_t ⊙ (γ_t ⊙ z_sem + β_t), where ⊙ represents element-wise multiplication.

3. The chip as described in claim 2, characterized in that, The first spiking neural network in the first computing region is also specifically used to perform no less than two internal recursive modulations to progressively optimize the motion latent variables. In a new iteration, the first spiking neural network updates and generates a hidden state h_t based on the same ontology perception signal and the hidden state h_{t-1} generated in the previous iteration. It also updates the gate signal g_t, scaling factor γ_t, and offset β_t based on the newly generated hidden state h_t. Finally, it applies the new g_t, γ_t, and β_t to the motion latent variable z_mod output in the previous iteration to generate the motion latent variables for the current iteration.

4. The chip according to any one of claims 1-3, characterized in that, The first spiking neural network includes a spiking-GRU (Spiking-Gated Recurrent Unit), and is further configured to perform collision prediction and preventative avoidance functions, which include: The force signal from the sensor is converted into a frequency-modulated pulse sequence by a pulse frequency encoder. The pulse frequency f_pulse of the frequency-modulated pulse sequence is dynamically determined based on the external force amplitude. The f_pulse satisfies the formula: f_pulse = f_min + (f_max - f_min) · σ (F_ext / F_scale), where f_min is the baseline frequency without external force, f_max is the maximum external force saturation frequency, F_ext is the external force amplitude, F_scale is the force scaling factor, and σ is the Sigmoid function. The Spiking-GRU, as a forward inward model, is used to receive the frequency modulation pulse sequence and the current joint state signal to predict the predicted motion trajectory q_hat for the next T time steps. The collision risk value (collision_risk) between q_hat and the obstacle obstacle in the environment is calculated based on the formula: collision_risk = max(0, 1 - min_distance(q_hat, obstacle) / safety_margin). Here, min_distance(q_hat, obstacle) is the minimum distance between q_hat and the obstacle obstacle, and safety_margin is a preset safety threshold. If the collision risk value is determined to be greater than 0, a bias term opposite to the direction of the external force is applied to the output layer that generates the motion latent variable z_mod to obtain the preventive adjustment motion latent variable. The preventive adjustment motion latent variable is used as the output of the motion latent variable z_mod. The adjusted motion latent variable satisfies the formula: a_adjusted = a_original - α · collision_risk · F_ext_normalized, where a_original is the corresponding component in the motion latent variable z_mod before adjustment, a_adjusted is the corresponding component in the adjusted motion latent variable, α is the preset adjustment gain coefficient, and F_ext_normalized is the normalized external force direction vector.

5. The chip according to any one of claims 1-3, characterized in that, The chip also includes: An on-chip interconnect bus is used to connect the GPU, the first computing area, and the second computing area, respectively. A shared on-chip memory, coupled to the on-chip interconnect bus, is used to provide unified address space access between the GPU, the first computing area, and the second computing area; Specifically, the GPU is used to write the latent variable of the task intent generated in each inference cycle into the first preset address area of ​​the shared on-chip memory; The first computing area is specifically used to read the task intent latent variable from the first preset address area of ​​the shared on-chip memory through the direct memory access channel established by the on-chip interconnect bus; The first computing area is also specifically used to write the motion latent variables finally generated in each cycle into the second preset address area in the shared on-chip memory; The second computing area is specifically used to read the motion latent variables from the second preset address area of ​​the shared on-chip memory through the direct memory access channel established by the on-chip interconnect bus.

6. The chip as described in claim 5, characterized in that, The SNN processor further includes a hardwired secure reflection path, which is independent of the first computation region and the second computation region. The hardwired secure reflection path is used for: It receives force output signals from preset sensors in an independent and parallel manner; The pulse frequency threshold comparator determines whether the force output signal exceeds a preset safety threshold. If the force output signal is determined to exceed the preset safety threshold, a pre-coded protection action command is directly triggered and output to the joint actuator. The pre-coded protection action command includes a control signal that drives the robot joint to stop moving. Simultaneously, an asynchronous wake-up interrupt signal is generated and sent to the GPU via the on-chip interconnect bus to wake up the GPU for replanning.

7. The chip according to any one of claims 1-3 and 5, characterized in that, The chip is configured with a static hold mode, in which: The GPU enters a reduced-frequency sleep state, stopping VLM inference operations; The second spiking neural network in the second computing area is used to continuously output a basis control signal corresponding to the current posture to the joint actuator by utilizing the integral memory characteristics of the neuron membrane potential, so as to maintain the current joint posture of the robot.

8. The chip according to any one of claims 1-3 and 5, characterized in that, The first frequency is in the Hz range, the second frequency is in the 100Hz range, and the third frequency is in the kHz range.

9. A computational method for a heterogeneous layered SoC chip for biomimetic motion control of robots, characterized in that, include: Based on the graphics processing unit (GPU) and user-input task instructions, inference is performed periodically at a first frequency. In each inference cycle, a visual language model (VLM) is run based on the task instructions and the current visual input image to generate task intent latent variables. The task intent latent variables are semantic feature vectors abstracted by the VLM after understanding the scene content and task intent. The task intent latent variable and the current body perception signal from the robot are obtained based on the first computing area in the spiking neural network (SNN) processor. The task intent latent variable and the current body perception signal are fused and modulated to periodically generate motion latent variables at a second frequency. The motion latent variable is a feature vector generated after fusing and modulating the task intent latent variable and the current body perception signal. The second frequency is higher than the first frequency. The motion latent variables are obtained based on the second computing region in the SNN processor, and the motion latent variables are periodically decoded into control signals for driving robot joint movement at a third frequency, wherein the third frequency is higher than the second frequency.

10. The method as described in claim 9, characterized in that, The first spiking neural network includes a spiking-GRU and a gated feature linear modulation circuit (Gated FiLM), and the method further includes: Based on the Spiking-GRU and the Gated FiLM, a motion latent variable generation process is periodically performed at the second frequency, wherein the motion latent variable generation process in each cycle includes: Obtain the latest task intent latent variables generated by the GPU, and obtain the current body perception signals from the robot's body sensors; Based on the Spiking-GRU, the current ontology perception signal and the hidden state of the previous time step are recursively encoded to generate the hidden state h_t of the current time step, wherein the initial hidden state is an all-zero vector. Based on the Gated FiLM, the h_t is processed using the following three-way modulation parameter branching: Gated signal branch: h_t is input into the first fully connected layer, and a gated signal g_t is output through the Sigmoid activation function. g_t is used to control the contribution ratio of each dimension of semantic information in the latent variable z_sem of the task intent to the latent variable of motion. g_t satisfies the formula g_t = sigmoid(W_g·h_t), where W_g is the weight matrix obtained by training the first fully connected layer. Scaling factor branch: The h_t is input into the second fully connected layer and the scaling factor γ_t is output. The γ_t is used to scale the latent variable of the task intent element by element. The γ_t satisfies the formula: γ_t = W_γ· h_t, where W_γ is the weight matrix obtained by training the second fully connected layer. Offset branch: The state estimate h_t is input into the third fully connected layer and the offset β_t is output. The offset β_t is used to adjust the baseline bias of the latent variable of the task intention. The β_t satisfies the formula: β_t = W_β·h_t, where W_β is the weight matrix obtained by training the third fully connected layer. After the three-way branch processing, the motion latent variable z_mod is generated by element-wise fusing and modulating the task intent latent variable z_sem, the gate signal g_t, the scaling factor γ_t, and the offset β_t based on the formula: z_mod = g_t ⊙ (γ_t ⊙ z_sem + β_t), where ⊙ represents element-wise multiplication.

Images