A robot brain system for embodied intelligence

Abstract

The application relates to a robot brain system for embodied intelligence, which comprises a system on chip, an image acquisition unit and a real-time bus master. The system on chip integrates a neural network processor and runs a real-time operating system, and the processing of environment sensing data, the generation of motion decision and the generation of motion control instructions are completed in the chip; image signals output by an image sensor are transmitted to a serial deserializer through a serial link, are converted into parallel image signals and are input into the system on chip through an image interface; the real-time bus master runs in the system on chip, encapsulates motion control instructions into bus data frames and issues the bus data frames to actuators, and receives state information returned by the actuators to provide the system on chip with motion control instructions for the next cycle. The system can be applied to real-time control of robots with embodied intelligence, shortens a data path and reduces control time delay.

Description

Technical Field

[0001] This application relates to the fields of robotics and artificial intelligence, and in particular to a robotic brain system for embodied intelligence. Background Technology

[0002] Embossed intelligent robots need to simultaneously meet multi-dimensional requirements such as multi-channel real-time visual perception, real-time reasoning and decision-making, high-precision motion control, and multi-sensor fusion. The robot's brain, as the robot's central computing and control system, is responsible for environmental perception, intelligent decision-making, motion control, and multi-sensor fusion, analogous to the role of a biological brain in a living organism.

[0003] Currently, there is a dedicated chip architecture for robot controllers that integrates sensing, computing, and control. For example, Chinese patent application CN120804022A discloses a dedicated chip architecture for robot controllers that integrates sensing, computing, and control. It adopts a multi-core heterogeneous SoC architecture, including a control core, a computing core, and a sensing core. These three types of cores are integrated into a single SoC chip, realizing the integrated fusion of multimodal perception, high-throughput computing, and high real-time control functions.

[0004] While the aforementioned solutions propose an integrated sensing-computing-control chip architecture, they are chip-level designs requiring custom chip design, resulting in long development cycles and high costs. Furthermore, they do not address system-level implementation details such as specific image sensor access schemes, real-time motion control bus protocol implementation, multi-sensor time synchronization, and carrier-level bus resource allocation. Existing technologies also employ the Jetson platform combined with EtherCAT for robot control, or use a discrete hardware architecture to distribute vision processing, general processing, robot control, and EtherCAT communication across multiple independent chips. This introduces additional latency in communication between chips, making it difficult to achieve true sensing and motion control coordination. Summary of the Invention

[0005] To address the technical issues of existing robot brain systems, such as the disconnect between perception and motion control, lack of multi-sensor time synchronization, and insufficient hardware platform interface scalability, this application provides a robot brain system oriented towards embodied intelligence.

[0006] The robot brain system for embodied intelligence provided in this application adopts the following technical solution: A robotic brain system for embodied intelligence, comprising: The system-on-a-chip integrates a neural network processor and runs a real-time operating system. The system-on-a-chip is configured to: process environmental perception data, generate motion decisions, and generate motion control commands on-chip. An image acquisition unit includes an image sensor and a serial deserializer. The image signal output by the image sensor is transmitted to the serial deserializer via a serial link. The serial deserializer is connected to the image interface of the system-on-a-chip and is configured to convert the image signal into a parallel image signal and input it into the system-on-a-chip. A real-time bus master station, which runs within the system-on-a-chip, drives one or more actuators via the communication interface of the system-on-a-chip. The real-time bus master station is configured to encapsulate the motion control commands into bus data frames and send them to the actuators, as well as receive status information returned by the actuators. The system-on-a-chip is also configured to generate motion control commands for the next cycle based on the status information.

[0007] By adopting the above technical solution, the system-on-a-chip integrates a neural network processor, completing the processing of environmental perception data, the generation of motion decisions, and the generation of motion control commands on-chip. This achieves an integrated closed loop of perception, decision-making, and control within the same chip, avoiding communication delays between different chips. The image acquisition unit transmits image signals to a serial deserializer via a serial link, converting them into parallel image signals before inputting them into the system-on-a-chip via an image interface, achieving low-latency image data access. The real-time bus master station runs within the system-on-a-chip, encapsulating motion control commands into bus data frames and sending them to the actuators. It also receives status information returned by the actuators. Based on this status information, the system-on-a-chip generates motion control commands for the next cycle, forming a complete closed loop of perception-decision-control.

[0008] Preferably, the system-on-a-chip also integrates a central processing unit and a graphics processing unit, and the neural network processor includes a deep learning accelerator and a vision accelerator.

[0009] By adopting the above technical solutions, the system-on-a-chip integrates a central processing unit, a graphics processing unit, a deep learning accelerator, and a vision accelerator to form a heterogeneous computing architecture, which can simultaneously meet the needs of various types of computing tasks such as general computing, graphics acceleration, deep learning inference, and vision acceleration.

[0010] Preferably, the system-on-a-chip is a single system-on-a-chip chip.

[0011] By adopting the above technical solution, perception, decision-making and control are all integrated into a single chip, avoiding communication delays between multiple chips and ensuring the real-time response of the system.

[0012] Preferably, the image sensor is a multi-channel image sensor, and the image signals of each image sensor are transmitted to the serial deserializer via their respective GMSL2 serial links.

[0013] By adopting the above technical solution, multiple image sensors transmit image signals through their respective GMSL2 serial links. The GMSL2 serial link has the characteristics of low latency, long-distance transmission and strong anti-interference capability, which is suitable for robot real-time perception and closed-loop control scenarios.

[0014] Preferably, the serial deserializer is a single serial deserializer used to aggregate the various image signals and output the parallel image signal; the image interface includes a CSI controller, the parallel image signal is input to the on-chip system via the CSI controller, the CSI controller supports multiple virtual channels, and each image signal is input in parallel via its respective virtual channel.

[0015] By adopting the above technical solution, a single serial deserializer aggregates multiple image signals and outputs parallel image signals. The CSI controller receives each image signal in parallel through multiple virtual channels, realizing efficient parallel access of multiple image data without occupying USB or PCIe bandwidth.

[0016] Preferably, the real-time bus master is an EtherCAT master, which is a software master running on the real-time operating system.

[0017] By adopting the above technical solution, the EtherCAT master station runs on the real-time operating system of the on-chip system as a software master station, encapsulates motion control commands into EtherCAT data frames and sends them to the actuators, with a control cycle of up to 125μs, thus realizing high-precision real-time motion control.

[0018] Preferably, the system further includes a timing synchronization unit configured on the system-on-a-chip (SoC) for calibrating the SoC's system clock based on an external timing signal and outputting a frame synchronization trigger signal. The frame synchronization trigger signal is transmitted to the image sensor via a serializer / deserializer to enable synchronous exposure of the image sensor. The image interface is used to attach a hardware timestamp to the image data at the time of image acquisition.

[0019] By adopting the above technical solution, the timing synchronization unit calibrates the system clock of the on-chip system based on the external timing signal, outputs a frame synchronization trigger signal to enable multiple image sensors to be exposed synchronously, and the image interface adds a hardware timestamp to the image data at the time of image acquisition, thereby realizing the unified time reference alignment of multi-sensor data with a synchronization accuracy better than 10μs.

[0020] Preferably, the external timing signal is a PPS pulse signal and an NMEA message.

[0021] By adopting the above technical solution, the PPS pulse signal provides a high-precision second-level synchronization reference, and the NMEA message provides time information of year, month, day, hour, minute, and second. The combination of the two can achieve accurate calibration of the system clock.

[0022] Preferably, the system further includes a power management unit and a communication unit; the power management unit includes a microcontroller for controlling the power-on process of the on-chip system according to a preset timing sequence; the communication unit includes a cellular communication module and a wireless local area network communication module for data interaction with external devices.

[0023] By adopting the above technical solutions, the power management unit controls the power-on process of the on-chip system according to a preset timing sequence through the microcontroller, ensuring that the on-chip system is powered on in the standard sequence and improving the reliability of system startup; the communication unit realizes wireless data interaction with the cloud, operation and maintenance platform, and remote control terminal through cellular communication module and wireless local area network communication module.

[0024] Preferably, the system further includes a carrier board on which the system-on-a-chip is disposed; the carrier board is provided with: a first Ethernet physical layer transceiver electrically connected to the RGMII interface of the system-on-a-chip for providing a first Gigabit Ethernet; a PCIe to Ethernet chip electrically connected to the PCIe interface of the system-on-a-chip for providing a second Gigabit Ethernet; a 10 Gigabit Ethernet physical layer transceiver electrically connected to the UPHY interface of the system-on-a-chip for providing 10 Gigabit Ethernet; an isolated CAN transceiver electrically connected to the CAN interface of the system-on-a-chip; and an isolated RS485 transceiver electrically connected to the UART interface of the system-on-a-chip.

[0025] By adopting the above technical solution, the carrier board coordinates the bus resources of the on-chip system, such as the RGMII interface, PCIe interface, UPHY interface, CAN interface, and UART interface, to peripherals such as Gigabit Ethernet, 10 Gigabit Ethernet, isolated CAN bus, and isolated RS485 bus, enabling parallel access of multiple types of high-speed peripherals on a single carrier board without conflict. Ethernet functionality is distributed across three different buses—RGMII, PCIe, and UPHY—to avoid overloading any single bus; the CAN bus and RS485 bus employ isolation design to ensure the communication reliability of the core module in industrial electromagnetic interference environments.

[0026] In summary, this application includes at least one of the following beneficial technical effects: By integrating a neural network processor on a system-on-a-chip and completing environmental perception data processing, motion decision generation, and motion control command generation on-chip, a closed-loop system integrating perception, decision-making, and control within the same chip is achieved, avoiding communication delays between different chips and ensuring the real-time response of the system. The image signal is converted into a parallel image signal through the serial link and serial deserializer of the image acquisition unit and input to the on-chip system through the image interface. Combined with the low latency characteristics of the GMSL2 serial link, low latency visual perception access is realized. The real-time bus master station runs within the system-on-a-chip, encapsulating motion control commands into bus data frames and sending them to the actuators while receiving status feedback. The system-on-a-chip generates the next cycle command based on the status information, thus forming a complete closed-loop control link. By calibrating the system clock based on the external timing signal and outputting the frame synchronization trigger signal through the timing synchronization unit, and by adding a hardware timestamp at the moment of image acquisition through the image interface, a unified time reference alignment of multi-sensor data is achieved. The carrier board can coordinate and allocate various bus resources of the on-chip system to peripherals such as Gigabit Ethernet, 10 Gigabit Ethernet, isolated CAN bus and isolated RS485 bus, so that multiple high-speed peripherals can be accessed in parallel without conflict on a single carrier board. Attached Figure Description

[0027] Figure 1 A system block diagram of the robot brain system provided in the embodiments of this application; Figure 2 This is a schematic diagram of the carrier board interface layout of the robot brain system provided in the embodiments of this application; Figure 3 This is a circuit diagram of the serial deserializer in the image acquisition unit provided in an embodiment of this application; Figure 4 The circuit schematic of the power management unit provided in the embodiments of this application; Figure 5 A schematic diagram of the signal path for the multi-sensor time synchronization mechanism provided in the embodiments of this application.

[0028] Explanation of reference numerals in the attached figures: 110, System-on-a-Chip; 111, Neural Network Processor; 120, Image Acquisition Unit; 121, Image Sensor; 122, Serial Link; 123, Serial Deserializer; 124, Image Interface; 130, Real-Time Bus Master; 131, Communication Interface; 140, Actuator; 150, Time Synchronization Unit; 151, Time Source; 160, Power Management Unit; 170, Communication Unit; 180, Carrier Board. Detailed Implementation

[0029] To facilitate understanding of the technical solution of this application, the following explanations are provided for several definitions and terms involved in this application.

[0030] "System-on-a-Chip" refers to an integrated circuit that integrates computing resources such as central processing unit, graphics processing unit, and neural network processor, as well as interface circuits such as image interface and communication interface, onto the same chip. "Closed loop" means that the processing of environmental perception data, the generation of motion decisions, the generation of motion control commands, and the reception of execution feedback are all completed within the system-on-a-chip 110, and the data of the three links of perception, decision-making, and control flow within the chip; A "real-time operating system" refers to an operating system that can provide a deterministic scheduling environment for periodic control tasks, enabling the tasks to be completed within specified time constraints. "Real-time" means that the system or task can complete the specified processing within the specified time constraints to meet the requirements of periodic control; "Serial link" refers to the transmission link that serializes the parallel image signals output by the image sensor and transmits them to the serial deserializer; A "serial deserializer" is a processing device that converts serial image signals transmitted via a serial link into parallel image signals. "Image interface" refers to the interface circuit located within the system-on-a-chip (SoC) that receives parallel image signals and inputs them into the SoC's internal circuitry.

[0031] "Real-time bus master" refers to a functional unit that runs within the system-on-a-chip and is used to manage the real-time bus and drive actuators based on the real-time bus. It can be a software master. "Bus data frame" refers to a data unit encapsulated in a format specified by the bus protocol, used to carry motion control commands or status information; "Actuator" refers to a controlled execution device driven by a real-time bus master.

[0032] "Frame synchronization trigger signal" refers to a signal that is synchronized with the system clock and is used to trigger the synchronous exposure of each image sensor; "Hardware timestamp" refers to the time information recorded in hardware by the image interface at the moment when image data enters the on-chip system for acquisition; "Time source" refers to the source that provides external time signals to the system, such as the second pulse signal and time message output by the GPS / BeiDou receiver module.

[0033] The following is in conjunction with the appendix Figures 1-5 This application will be described in further detail.

[0034] This application discloses a robot brain system for embodied intelligence. Combined with... Figure 1 The robot's brain system includes a system-on-a-chip (SoC) 110, an image acquisition unit 120, a real-time bus master station 130, and one or more actuators 140. Furthermore, such as Figure 2 As shown, the robot brain system may also include a timing synchronization unit 150, a power management unit 160, a communication unit 170, and a carrier board 180, with each functional unit integrated on the same carrier board 180.

[0035] The system-on-a-chip 110 integrates a neural network processor 111 and runs a real-time operating system. The system-on-a-chip 110 is configured to process environmental perception data, generate motion decisions, and generate motion control commands on-chip. Data from the perception, decision-making, and control stages all flow within the system-on-a-chip 110 without leaving the chip, thus forming an on-chip closed-loop chain.

[0036] Optionally, the system-on-a-chip 110 may also integrate a central processing unit (CPU) and a graphics processing unit (GPU), and the neural network processor 111 may further include a deep learning accelerator and a vision accelerator. The CPU is used to execute real-time operating system and logical operations related to motion decision-making and motion control; the GPU is used for parallel processing of image data; the deep learning accelerator is for accelerating inference of deep learning models, and the vision accelerator is for accelerating visual processing tasks.

[0037] Optionally, the aforementioned system-on-a-chip 110 can be a single system-on-a-chip chip. In this embodiment, the system-on-a-chip 110 adopts the NVIDIA Jetson AGX Orin platform, which integrates a 12-core Arm Cortex-A78AE central processing unit, an Ampere architecture graphics processor, and deep learning and vision accelerators, achieving an AI computing power of up to 275 TOPS (8-bit precision). The real-time operating system is built on the Linux kernel and runs on the central processing unit, providing a deterministic scheduling environment for the generation and issuance of motion control commands.

[0038] The real-time operating system running on the system-on-a-chip (SoC) 110 is responsible for scheduling the computing resources within the SoC 110. The processing of environmental perception data, the generation of motion decisions, and the generation of motion control commands all run as software modules on this SoC. Specifically, the perception processing module runs on the neural network processor 111 and the graphics processor, performing target detection, semantic segmentation, and other processing on the image data input from the image acquisition unit 120, and outputting environmental perception results. The motion decision module generates motion decisions based on the environmental perception results and the task objective. The motion control command generation module calculates the control quantities of each actuator 140 based on the motion decisions and generates motion control commands. All data interaction between these modules is completed within the SoC 110 itself, without passing through an external bus.

[0039] Combination Figure 1 and Figure 3The image acquisition unit 120 includes an image sensor 121, a serial link 122, a serial deserializer 123, and an image interface 124. The image sensor 121 acquires environmental images, and the output image signal is transmitted to the serial deserializer 123 via the serial link 122. The serial deserializer 123 is connected to the image interface 124 of the system-on-chip 110, converting the serial image signal into a parallel image signal, which is then input to the system-on-chip 110 via the image interface 124. After the image signal enters the system-on-chip 110, the computational resources, such as the neural network processor 111, process the environmental perception data.

[0040] Optionally, the image acquisition unit 120 may further include multiple image sensors 121, with the image signals from each image sensor 121 transmitted to the serial deserializer 123 via their respective serial links 122. In this embodiment, the serial link 122 is a Gigabit Multimedia Serial Link (GMSL2), which serializes the parallel image signals output by the image sensors 121 and transmits them via coaxial cable, serving both data transmission and power supply functions. That is, it transmits image data and supplies power to the image sensors 121 simultaneously via the same coaxial cable. The image sensors 121 are four GMSL2 cameras, each with a resolution of 8 megapixels and a frame rate of 30 frames per second. The cameras are connected to the carrier board 180 via coaxial cables, with a transmission distance of up to 15 meters.

[0041] Optionally, the serial deserializer 123 can be a single serial deserializer used to aggregate various image signals and output parallel image signals; the image interface 124 may include a camera serial interface (CSI) controller, through which parallel image signals are input to the on-chip system 110. The CSI controller supports multiple virtual channels, and each image signal is input in parallel through its respective virtual channel. In this embodiment, the serial deserializer 123 uses a single MAX96712 chip, such as... Figure 3 As shown. The MAX96712 has multiple GMSL2 serial input channels and one MIPI CSI-2 parallel output channel. Each image signal is deserialized and aggregated within the serial deserializer 123, and then output to the image interface 124 via the CSI-2 channel. The serial deserializer 123 also has an inter-integrated circuit bus (I2C) configuration interface for receiving configuration commands, such as configuring the enable status and triggering mode of each input channel. The image interface 124 is located within the system-on-a-chip 110. Each image signal is input to the system-on-a-chip 110 in parallel via its respective virtual channel, captured by the hardware capture unit within the image interface 124, and moved to memory via the direct memory access (DMA) unit for processing by the neural network processor 111 and the graphics processor. The power supply, configuration, and signal connection relationships between the serial deserializer 123 and the image interface 124 are as follows. Figure 3 As shown.

[0042] The real-time bus master station 130 runs within the system-on-a-chip 110 and drives one or more actuators 140 via the communication interface 131 of the system-on-a-chip 110. The real-time bus master station 130 encapsulates motion control commands into bus data frames and sends them to the actuators 140, while simultaneously receiving status information returned by the actuators 140. Based on this status information, the system-on-a-chip 110 generates motion control commands for the next cycle, forming a closed loop of perception-decision-control-feedback.

[0043] Optionally, the aforementioned real-time bus master 130 can be an EtherCAT (Ethernet Control Automation Technology) master, and a software master running on a real-time operating system. In this embodiment, the real-time bus master 130 is implemented using the open-source IgH software master and is connected to an external bus network via communication interface 131. Communication interface 131 is a network interface provided by the system-on-chip 110, including a Gigabit Ethernet interface. The real-time bus master 130 accesses the EtherCAT bus network via the Gigabit Ethernet interface. The specific network interface circuit on the carrier board 180 is detailed below.

[0044] Actuator 140 is the controlled object driven by real-time bus master station 130 and is connected to the system via a bus network. Optionally, actuator 140 can be a servo driver. In this embodiment, actuator 140 includes a servo driver and an IO module, and is connected to the bus network as an EtherCAT slave. Real-time bus master station 130 encapsulates motion control commands into EtherCAT data frames and sends them to each actuator 140 according to the control cycle; actuator 140 encapsulates its own status information, such as current position, speed, and fault status, into data frames and sends them back. After receiving them, real-time bus master station 130 hands them over to system-on-chip 110 for processing to generate motion control commands for the next cycle. The control cycle is approximately 125 microseconds. An electrical isolation circuit is provided between communication interface 131 and actuator 140, with an isolation voltage of not less than 3 kV, to ensure the electrical safety of the communication link in the industrial field.

[0045] Combination Figure 5 Optionally, the aforementioned robot brain system may further include a timing synchronization unit 150, configured on the system-on-chip 110. The timing synchronization unit 150 is used to calibrate the system clock of the system-on-chip 110 based on an external timing signal and output a frame synchronization trigger signal; the frame synchronization trigger signal is transmitted to the image sensor 121 via a serializer / deserializer 123 to enable the image sensor 121 to be exposed synchronously; the image interface 124 is used to attach a hardware timestamp to the image data at the time of image acquisition.

[0046] The timing synchronization unit 150 is connected to the timing source 151. Optionally, the external timing signal can be a pulse-per-second (PPS) signal and an NMEA 0183 message. In this embodiment, the timing source 151 is a GPS / BeiDou receiver module that outputs a PPS signal and an NMEA 0183 message. The PPS signal provides a high-precision pulse-per-second, and the NMEA 0183 message provides the corresponding time information. The timing synchronization unit 150 calibrates the system clock of the on-chip system 110 based on both, ensuring that the system clock is synchronized with the timing source 151. Optionally, the timing synchronization unit 150 can send a configuration to the serial deserializer 123 via the I2C interface, causing the serial deserializer 123 to forward the frame synchronization trigger signal to each image sensor 121. The frame synchronization trigger signal is synchronized with the system clock, and each image sensor 121 exposes simultaneously according to the frame synchronization trigger signal, thereby ensuring that the multiple image sensors 121 are time-aligned in their acquisition.

[0047] The hardware timestamp is recorded by the image interface 124 at the instant the image data enters the on-chip system 110, rather than being added retroactively by the software layer, thus avoiding time uncertainty caused by software scheduling delays. In this embodiment, after the image data is marked with a hardware timestamp, the time synchronization accuracy is better than approximately 10 microseconds. Optionally, the aforementioned hardware timestamp can also be applied to the data acquired by other sensors such as the inertial measurement unit (IMU) and lidar, so that the data from multiple types of sensors are under the same time reference.

[0048] The robot's brain system may also include a power management unit 160 for supplying power to the system-on-a-chip 110 and other functional units, and controlling the power-on process. In this embodiment, the power management unit 160 includes a microcontroller, which controls the establishment sequence of each power supply rail according to a preset timing sequence when the system is powered on. Specifically, after the system is connected to an external DC power supply, the microcontroller detects and confirms that the power input is normal, and then enables each power supply rail in sequence, including: first establishing the HV power supply rail (15.6 volts), then establishing the +3.3 volt and +1.8 volt power supply rails, and controlling the power-on of the system-on-a-chip 110 module through the CARRIER_PWR_ON signal. The power management unit of the system-on-a-chip 110 takes over the subsequent power supply management. The establishment path of each power supply rail and the control connection relationship of the microcontroller for the power-on process are as follows: Figure 4 As shown.

[0049] The robot brain system may also include a communication unit 170 for data interaction between the robot brain system and external devices. In this embodiment, the communication unit 170 includes a cellular communication module and a wireless local area network (WLAN) communication module. The cellular communication module is a 5G full-network compatible module, connected to the system-on-a-chip (SoC) 110 via a universal serial bus, and is used for remote data interaction with remote devices, such as data interaction with a cloud server. The WLAN communication module is a WiFi / Bluetooth module, connected to the SoC 110 via an M.2 interface, and is used for short-range wireless data interaction, such as data interaction with debugging terminals and peripheral devices.

[0050] Combination Figure 2 The robot's brain system may also include a carrier board 180 for housing various functional units. In this embodiment, the system-on-chip 110, the serializer 123, the network interface circuit corresponding to the communication interface 131, the power management unit 160, and the communication unit 170 are all integrated on the same carrier board 180. The system-on-chip 110 is mounted on the carrier board 180 in a modular form; the serializer 123 is mounted on the carrier board 180, and its output parallel image signal is transmitted to the image interface 124 of the system-on-chip 110 via the on-board traces of the carrier board 180. The image signal is transmitted within the board without passing through external cables, resulting in a short and definite transmission path.

[0051] The carrier board 180 is also equipped with various interface circuits, which are electrically connected to the corresponding interfaces of the system-on-a-chip 110: The first Ethernet physical layer transceiver is electrically connected to the Reduced Gigabit Media Independent Interface (RGMII) of the System-on-Chip 110 to provide a first Gigabit Ethernet port. In this embodiment, the first Ethernet physical layer transceiver uses a Marvell 88E1512 chip, and the RGMII interface is the native Gigabit interface of the System-on-Chip 110.

[0052] The PCIe-to-Ethernet chip is electrically connected to the Fast Peripheral Interconnect (PCIe) interface of the System-on-Chip 110 to provide a second Gigabit Ethernet port. In this embodiment, the PCIe-to-Ethernet chip is an Intel I210 chip.

[0053] A 10 Gigabit Ethernet physical layer transceiver, electrically connected to the high-speed serial interface of the System-on-Chip 110, is used to provide 10 Gigabit Ethernet.

[0054] An isolated Controller Area Network (CAN) transceiver is electrically connected to the CAN interface of the System-on-Chip 110. In this embodiment, the isolated CAN transceiver uses a CA-IS2062A CAN controller with built-in isolation, providing two isolated CAN interfaces.

[0055] An isolated RS485 transceiver is electrically connected to the Universal Asynchronous Receiver / Transmitter (UART) interface of the System-on-Chip 110. In this embodiment, the isolated RS485 transceiver uses a CA-IS3092W isolation device, providing four isolated RS485 interfaces with an isolation voltage of not less than 3 kV.

[0056] The real-time bus master station 130 is connected to the bus network via the interface circuit corresponding to the first or second gigabit Ethernet, and drives the actuator 140.

[0057] In addition, the carrier board 180 may also be equipped with USB expansion circuitry, storage interface circuitry, audio interface circuitry, etc. In this embodiment, the USB expansion circuitry includes a USB 2.0 hub and a USB 3.0 hub to expand the number of USB ports; the storage interface circuitry is an M.2 interface circuitry for connecting to non-volatile memory as memory storage for the robot's brain; the audio interface circuitry includes an audio codec for audio input and output.

[0058] The working principle of the robot brain system in this application embodiment is as follows: Power-on phase: After the system is connected to an external DC power supply, the microcontroller of the power management unit 160 detects that the power input is normal, establishes each power rail sequentially according to the preset timing, and powers on the on-chip system 110 via the CARRIER_PWR_ON signal. The on-chip system 110 starts the real-time operating system, and the real-time bus master station 130 starts running accordingly.

[0059] Timing phase: The timing synchronization unit 150 calibrates the system clock of the on-chip system 110 based on the PPS pulse signal of the timing source 151 and the NMEA 0183 message, and outputs a frame synchronization trigger signal synchronized with the system clock; the frame synchronization trigger signal is forwarded to each image sensor 121 via the serializer 123.

[0060] Acquisition Phase: Each image sensor 121 synchronously exposes according to the frame synchronization trigger signal to acquire environmental images. Each image signal is serialized and transmitted to the serial deserializer 123 via its respective serial link 122. The serial deserializer 123 converts the image signal into a parallel image signal and inputs it to the on-chip system 110 via the image interface 124. The image interface 124 adds a hardware timestamp at the moment the image data enters.

[0061] Perception and decision-making stage: The neural network processor 111 and the graphics processor in the system-on-chip 110 process the image data to complete the environmental perception; the motion decision module generates motion decisions based on the environmental perception results; the motion control command generation module generates motion control commands for each actuator 140 based on the motion decisions.

[0062] Control feedback phase: The real-time bus master station 130 encapsulates the motion control command into a bus data frame and sends it to each actuator 140 via the communication interface 131 according to the control cycle; the status information returned by the actuator 140 is returned to the on-chip system 110 via the real-time bus master station 130 and used to generate the motion control command for the next cycle.

[0063] This completes a full closed loop, and the cycle repeats accordingly.

[0064] The above embodiments are exemplary descriptions of this application, and the scope of protection of this application is not limited to the specific embodiments described above. While maintaining the core concept of an in-chip closed loop for perception, decision-making, and control, those skilled in the art can make substitutions or modifications to the following aspects: In addition to the NVIDIA Jetson AGX Orin platform, the System-on-a-Chip 110 can also be replaced with other heterogeneous computing platforms that integrate a central processing unit, a graphics processing unit, and a neural network processor.

[0065] In addition to the GMSL2 serial link, the serial link 122 can also be replaced with other link solutions that support long-distance serial transmission.

[0066] In addition to a single MAX96712, the serial deserializer 123 can be replaced with multiple deserializers connected in parallel or cascaded to support more video signals; it can also be replaced with other models of serial deserializers.

[0067] In addition to the EtherCAT master, the real-time bus master 130 can also be replaced with other real-time industrial Ethernet masters or real-time fieldbus masters.

[0068] In addition to servo drivers and I / O modules, actuator 140 may also include other types of controlled actuators such as stepper drivers and pneumatic actuators.

[0069] In addition to GPS / BeiDou PPS and NMEA, the timing source 151 can also be replaced with PTP / gPTP network timing mode, which obtains timing signals via Ethernet network as a redundant backup for GPS / BeiDou timing; other timing modes can also be used.

[0070] In addition to the 5G cellular communication module and WiFi / Bluetooth module, the communication unit 170 can also be replaced with other communication methods such as 4G module and Ethernet interface.

[0071] The number of channels, resolution, and frame rate of the image sensor 121 can be adjusted according to the application scenario. For example, if a two-channel, six-channel, or eight-channel image sensor is used, the resolution and frame rate can also be adjusted.

[0072] In addition to the microcontroller controlling the power-on sequence, the power management unit 160 can also be replaced with a dedicated power management chip (PMIC) or a discrete power-on sequence control circuit.

[0073] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A robot brain system for embodied intelligence, characterized in that, include: The system-on-a-chip (110) integrates a neural network processor (111) and runs a real-time operating system. The system-on-a-chip (110) is configured to: process environmental perception data, generate motion decisions and generate motion control commands on-chip. The image acquisition unit (120) includes an image sensor (121) and a serial deserializer (123). The image signal output by the image sensor (121) is transmitted to the serial deserializer (123) via a serial link (122). The serial deserializer (123) is connected to the image interface (124) of the system on-chip (110) and is configured to convert the image signal into a parallel image signal and input it into the system on-chip (110). A real-time bus master (130) runs within the system-on-a-chip (110). The real-time bus master (130) drives one or more actuators (140) via the communication interface (131) of the system-on-a-chip (110). It is configured to encapsulate the motion control instructions into bus data frames and send them to the actuators (140), and to receive status information returned by the actuators (140). The system-on-a-chip (110) is also configured to generate motion control instructions for the next cycle based on the status information.

2. The robot brain system for embodied intelligence according to claim 1, characterized in that: The system-on-a-chip (110) also integrates a central processing unit and a graphics processing unit, and the neural network processor (111) includes a deep learning accelerator and a vision accelerator.

3. The robot brain system for embodied intelligence according to claim 1, characterized in that: The system-on-a-chip (110) is a single system-on-a-chip chip.

4. The robot brain system for embodied intelligence according to claim 1, characterized in that: The image sensor (121) is a multi-channel image sensor, and the image signals of each image sensor (121) are transmitted to the serial deserializer (123) via their respective GMSL2 serial links (122).

5. A robot brain system for embodied intelligence according to claim 4, characterized in that: The serial deserializer (123) is a single serial deserializer used to aggregate the image signals and output the parallel image signals; the image interface (124) includes a CSI controller, and the parallel image signals are input to the on-chip system (110) through the CSI controller. The CSI controller supports multiple virtual channels, and each image signal is input in parallel through its own virtual channel.

6. A robot brain system for embodied intelligence according to claim 1, characterized in that: The real-time bus master (130) is an EtherCAT master, which is a software master running on the real-time operating system.

7. A robot brain system for embodied intelligence according to claim 1, characterized in that: It also includes a timing synchronization unit (150), which is configured on the system-on-chip (110) for calibrating the system clock of the system-on-chip (110) based on an external timing signal and outputting a frame synchronization trigger signal. The frame synchronization trigger signal is transmitted to the image sensor (121) via the serializer (123) so that the image sensor (121) is exposed synchronously. The image interface (124) is used to attach a hardware timestamp to the image data at the time of image acquisition.

8. A robot brain system for embodied intelligence according to claim 7, characterized in that: The external timing signals are PPS pulse signals and NMEA messages.

9. A robot brain system for embodied intelligence according to claim 1, characterized in that: It also includes a power management unit (160) and a communication unit (170). The power management unit (160) includes a microcontroller for controlling the power-on process of the system-on-chip (110) according to a preset timing sequence; The communication unit (170) includes a cellular communication module and a wireless local area network communication module, used for data interaction with external devices.

10. A robot brain system for embodied intelligence according to claim 1, characterized in that: It also includes a carrier board (180), on which both the system-on-a-chip (110) and the serializer (123) are mounted. The parallel image signal output by the serializer (123) is transmitted to the image interface (124) of the system-on-a-chip (110) via the on-board traces of the carrier board (180). The carrier board (180) is also provided with: The first Ethernet physical layer transceiver is electrically connected to the RGMII interface of the system-on-chip (110) to provide a first gigabit Ethernet; The PCIe to Ethernet chip is electrically connected to the PCIe interface of the system-on-chip (110) to provide a second Gigabit Ethernet. A 10 Gigabit Ethernet physical layer transceiver is electrically connected to the high-speed serial interface of the system-on-chip (110) for providing 10 Gigabit Ethernet; An isolated CAN transceiver is electrically connected to the CAN interface of the system-on-chip (110); and An isolated RS485 transceiver is electrically connected to the UART interface of the system-on-chip (110).

Images