End side intelligent model-based intelligent controller architecture with body and operation method

Abstract

The invention discloses a body intelligent controller architecture based on an end side intelligent model and an operation method, and belongs to the technical field of industrial body intelligence. According to the architecture, a CPU, a GPU and an FPGA are integrated by adopting a heterogeneous multi-core processor, and high-speed interconnection is realized through a star-type + NoC hybrid topology; a multi-level task scheduling mechanism is constructed based on a real-time operating system, and it is ensured that control tasks are executed preferentially. End-side visual model lightweight (YOLOv5s model parameters are reduced to 3.5 M) and FPGA hardware acceleration (control instruction time delay is 10ns) are innovatively realized on an algorithm framework layer, and a transmission bottleneck is eliminated through a zero-copy data path. The system supports model hot switching less than or equal to 10ms and FPGA bit stream online updating less than or equal to 50ms, and has MD5 check and an automatic rollback mechanism. Compared with a traditional separation type scheme, the processing time delay is reduced by 25 times, the hardware cost is reduced, the size is reduced, and the real-time performance, reliability and deployment flexibility of the industrial robot and other intelligent agents with the body are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of industrial embodied intelligent technology, and specifically to an embodied intelligent controller architecture and operation method based on a terminal-side intelligent model. Background Art

[0002] In the application field of industrial embodied intelligence, various embodied intelligent entities will rely on on-device visual recognition and perception, as well as real-time computing and processing by the on-device brain. In the current industrial robot system architecture, dedicated controllers are used for robot control, while independent industrial computers are required to run visual algorithm models for visual perception.

[0003] The pain points of existing technologies are: 1. Architectural separation leads to inefficiency: In traditional embodied intelligent systems, visual perception computing (such as object detection and semantic segmentation) relies on high-performance industrial computers, while robot control (such as motion planning and trajectory tracking) is implemented by dedicated controllers. The two interact through communication protocols, resulting in high latency, synchronization difficulties, and complex wiring.

[0004] 2. Waste of resources and high cost: The dual-device solution requires two sets of hardware and power supply systems, which are bulky and energy-intensive, making it unsuitable for miniaturized or mobile robot applications.

[0005] 3. High deployment complexity: Industrial computers and controllers need to be debugged and maintained separately, software compatibility is poor, and system integration is difficult.

[0006] Therefore, an embodied intelligent controller architecture and operation method based on an end-side intelligent model is proposed, which is suitable for the control and perception scenarios of embodied intelligent bodies such as industrial robots, miniaturized or mobile robots. Summary of the Invention

[0007] The present invention aims to solve the problems of low efficiency, resource waste, high cost, and complex deployment caused by the separation of visual perception computing and robotic control in existing industrial embodied intelligent systems. It provides an embodied intelligent controller architecture and operation method based on an end-side intelligent model to achieve the integration of visual perception computing and robotic control, optimize resource allocation, and simplify deployment and maintenance.

[0008] To achieve the above objectives, the present invention provides the following technical solutions: an embodied intelligent controller architecture based on an end-side intelligent model, comprising a hardware layer, an operating system layer, an algorithm framework layer, and an application interface layer.

[0009] Preferably, the hardware layer uses a heterogeneous multi-core processor as the core, integrating CPU, GPU and FPGA.

[0010] The heterogeneous multi-core processor's on-chip interconnect and data movement mechanism utilizes a hybrid "star + NoC" interconnect topology. The CPU (a quad-core ARM Cortex-A55) connects to the GPU (a Mali-G52) via four AXI-HP buses (each with 1 GB / s bandwidth) and to the FPGA (an Artix-7 series) via two AXI-Stream buses (each with 2 GB / s bandwidth). The on-chip NoC (network-on-chip) supports coherent access to shared memory between the CPU, GPU, and FPGA, with a total bandwidth of ≥10 Gbps. Six DMA channels are configured: GPU→CPU (two for vision data), CPU→FPGA (two for control parameters), FPGA→CPU (one for status feedback), and CPU→GPU (one for model parameters). Each DMA channel supports a maximum transfer block size of 4 MB and supports both interrupt-triggered and polling modes.

[0011] Functional Units: The CPU is responsible for overall system scheduling and complex logic processing; the GPU, with its powerful parallel computing capabilities, is used to run edge-side intelligent models for visual perception calculations; the FPGA is used for real-time operations related to robot control, ensuring low control latency. The hardware layer also includes an image acquisition interface (USB 3.0, supporting 1080P @ 60fps), a motion control interface (EtherCAT, 1ms cycle time), a storage unit (16GB eMMC + 32MB SRAM), and a power management unit (supporting independent voltage domain control for the CPU, GPU, and FPGA).

[0012] Preferably, the operating system layer is built based on a real-time operating system (RTOS, RT-Thread) and has a task priority scheduling mechanism.

[0013] RTOS priority scheduling and preemption strategies for GPUs and FPGAs: Utilizes a hybrid scheduling algorithm combining multi-level ready queues and EDF (earliest deadline first). Queues are divided into levels 0-7 (with level 0 being the highest). FPGA control tasks (such as collision detection) are fixed at level 0, GPU vision tasks are at level 3, and CPU complex logic tasks are at levels 4-7.

[0014] Isolation mechanism: GPU core occupancy is limited to a 5ms / 20ms window (using timer-enforced preemption) to prevent prolonged occupancy from blocking high-priority FPGA tasks. When an FPGA task is triggered, the CPU temporarily raises the priority of associated tasks through priority inheritance to ensure smooth data flow. Formal analysis demonstrates a worst-case scheduling delay (WCD) of ≤200μs (99.9% confidence level).

[0015] Preferably, the algorithm framework layer includes a terminal-side intelligent model module and a robot control algorithm module.

[0016] 1. Lightweight Strategy and Deployment Process for the Device-Side Visual Model: 1. Lightweight strategy: YOLOv5s is pruned (removing redundant convolution kernels and retaining 60% of the channels), INT8 quantized (using the TensorRT toolchain, with a quantization error of ≤2%), and BatchNorm fusion (reducing the number of inference steps by 30%). After processing, the model parameters are reduced from 20M to 3.5M, the video memory usage is reduced from 80MB to 15MB, and the GPU inference frame rate is ≥50fps (1080P input).

[0017] 2. Deployment Process: When loading a model, it is bound to a fixed area via GPU memory address mapping (starting at address 0x80000000, a continuous 15MB space). The kernel is compiled using TensorRT (options: --fp16 --useDLACore=0) to generate an optimized inference engine. Multi-model hot switching utilizes double-buffered loading: the foreground buffer runs the current model, while the backend buffer asynchronously loads the new model. After loading, the new model is switched atomically via a pointer (taking ≤10ms). If inference fails on the new model, the model automatically falls back to the foreground model.

[0018] 2. Hardware pipeline architecture of FPGA control algorithm: 1. Kinematic forward and inverse solutions, PID control, and other algorithms are implemented on the FPGA using a three-stage pipeline: the first stage is input preprocessing (data alignment and range check), the second stage is core operations (32-bit multiplier, CORDIC module accuracy 1e-6), and the third stage is output filtering (sliding average).

[0019] 2. Fixed-point bit width is 32 bits (16-bit integer + 16-bit fractional), with resource utilization of 45% for LUTs, 30% for FFs, 20% for BRAMs, and 50% for DSPs (Artix-735T device). Control instruction generation has a single-cycle latency of 10ns, and supports a maximum control frequency of 100kHz.

[0020] 3. Zero copy and ring buffer design in high-speed data paths: 1. Zero-copy implementation: The GPU directly accesses CPU shared memory through CUDA-IPC (inter-process communication). The FPGA implements zero-copy communication with the CPU user-mode ring buffer (4MB, 8 blocks) through the AXI-Stream interface, avoiding secondary data transfer.

[0021] 2. Buffer mechanism: A double-ended ring queue is used. The visual frame buffer sets high watermark (75%) and low watermark (25%) thresholds. When the production rate exceeds the consumption rate (for example, the GPU frame rate exceeds the CPU processing rate), dynamic window sampling is triggered (one frame is sampled every other frame). The control instruction buffer uses a static watermark (50%). When the watermark exceeds the limit, the oldest instruction is discarded and an overflow log is recorded to prevent accumulation of delays.

[0022] 4. Online model update and FPGA bitstream hot replacement process: 1. Model update: After the new model undergoes MD5 verification (segmented verification, 64KB per segment) and RSA digital signature verification (2048-bit key), it is downloaded in segments (1MB per segment, with resumable downloads) through the communication module. After the download is complete, it is written to the backup storage area (eMMC partition 2), loaded into the GPU memory in the background, and verified and inferred 10 times. Upon success, an atomic switch is performed (timing: triggered at time t0 → completed at time t0 + 30ms).

[0023] 2. FPGA bitstream hot replacement: The bitstream file (.bit) is transmitted using segmented encryption, decrypted upon receipt, and written to the FPGA configuration flash memory (SPI Flash). It is loaded in the background via the ICAP interface. During loading, the FPGA maintains the current control state (latched via registers). After loading is complete (t0+50ms), a soft reset is triggered. If the loading fails, the original bitstream (stored in the backup area) is automatically loaded to ensure uninterrupted control.

[0024] Preferably, the application interface layer provides a unified software development interface (API), including a visual perception interface (C++ / Python) and a control interface (based on Protobuf). The interface supports zero-copy mapping in high-speed data paths, allowing developers to directly access the ring buffer address without data copying.

[0025] A method for operating an embodied intelligent controller architecture based on an end-side intelligent model includes the following steps: S1, hardware self-test and driver initialization: After the system starts, the hardware layer completes self-test (taking ≤ 1s), the operating system layer loads and initializes each hardware driver (GPU driver initialization ≤ 500ms, FPGA configuration ≤ 200ms), the algorithm framework layer loads the lightweight YOLOv5s model (INT8 quantized version) and FPGA control algorithm bitstream, and the application interface layer initializes the interface service (registers the callback function); S2, data acquisition and processing: The image acquisition interface acquires 1080P image data via USB3.0 and writes it directly to the GPU memory via DMA (zero copy), taking ≤10ms; The end-side intelligent model module on the GPU processes the image data to obtain visual perception results (target location and category), which are then mapped to the CPU shared memory (address 0x80000000) via CUDA-IPC. The inference time is ≤ 20ms. S3, resource scheduling and task allocation: The CPU detects the visual perception results (shared memory is not empty) through the RTOS scheduling mechanism, promotes it from level 3 to level 4 (priority inheritance), parses the results and extracts control parameters (such as target coordinates); Based on the system load (CPU utilization <70%), the CPU sends control parameters to the FPGA via the AXI-Stream bus (transmission time ≤50μs). The FPGA control algorithm module generates control instructions through a three-stage pipeline operation (single cycle 10ns). S4, control instruction execution: The motion control interface (EtherCAT) sends the control instructions generated by the FPGA to the execution component (robotic arm / motor), with a cycle of 1ms and instruction transmission delay ≤50μs; S5, state feedback and adjustment: The sensor (6-axis IMU) collects robot state information and feeds it back to the CPU via the FPGA→CPU DMA channel (1 channel) (taking ≤ 100 μs); Based on the status information and visual perception results, the CPU adjusts the PID parameters in the FPGA through the algorithm framework layer (for example, the proportional coefficient changes from 5.0 to 5.5) to achieve closed-loop control; Multi-level power management and DVFS strategy: The power management unit monitors the temperature of each core in real time (sampling rate 10Hz). When the GPU temperature exceeds 85°C, DVFS is triggered (reducing the frequency to 800MHz, reducing power consumption by 30%). When the robot is stationary, the CPU enters sleep mode (power consumption decreases from 2W to 0.5W), the GPU shuts down some cores, and the FPGA remains in low-power mode (100mA). Typical power consumption is 3W at rest, 8W at low speed, and 15W during high-load crawling.

[0026] Beneficial effects The present invention provides an embodied intelligent controller architecture and operation method based on a device-side intelligent model, which has the following beneficial effects: 1. Extremely improved efficiency: Heterogeneous multi-core interconnection bandwidth ≥10Gbps, data loopback latency ≤2ms (acquisition → processing → execution → feedback), a 25-fold improvement over the traditional separate solution (≥50ms). The total latency for visual perception and control command generation is ≤50ms, meeting the needs of high-real-time scenarios.

[0027] 2. Resource utilization optimization: After YOLOv5s is lightweight, video memory usage is reduced by 81%, FPGA control algorithm resource utilization is ≤50% (Artix-735T), and multi-tasking is supported. The zero-copy mechanism reduces data transfer time by 90% and reduces CPU load by 40%.

[0028] 3. Cost and size advantages: A single hardware platform replaces the dual devices of "industrial computer + controller", reducing hardware costs by 40% and volume by 65% ​​(150mm×100mm×50mm), making it suitable for small mobile robots.

[0029] 4. Simplified deployment and maintenance: A unified API supports "one-time development, multi-platform adaptation." Online model updates and hot replacement of FPGA bitstreams require no downtime (switching time ≤ 50ms), improving maintenance efficiency by 80%.

[0030] 5. Reliability and security: Multi-level power management extends battery life by 50%, and priority scheduling and isolation mechanisms ensure that high-priority tasks on the FPGA are not blocked. The signature verification and rollback mechanism for model updates reduces the update failure rate to less than 0.1%. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a block diagram of the embodied intelligent controller architecture based on the device-side intelligent model of the present invention; Figure 2 This is a diagram of the on-chip interconnect architecture of the heterogeneous multi-core processor of the present invention, showing the connection relationship between the CPU, GPU, and FPGA through AXI-HP, AXI-Stream, and NoC, the DMA channel data flow (visual data, control parameters, etc.) and the shared memory mapping address; Figure 3 is a flow chart of the operating method of the present invention; Figure 4 This is the FPGA control algorithm pipeline architecture diagram of the present invention, showing the hardware implementation of the three-stage pipeline (input preprocessing → core operation → output filtering), and annotating the resource usage (LUT / FF / DSP) and latency of each stage. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0033] Example 1, please refer to Figure 1-4 The present invention provides a technical solution: application in a small mobile grasping robot.

[0034] This embodied intelligent controller architecture and operation method based on the end-side intelligent model is applied to a small mobile grasping robot to realize the recognition, positioning and grasping of target objects in the surrounding environment.

[0035] 1. Hardware layer configuration: Heterogeneous multi-core processor: CPU (quad-core Cortex-A55 @ 2.0 GHz), GPU (Mali-G52 @ 800 MHz), FPGA (Artix-735T).

[0036] Image acquisition interface: USB3.0 connects to a 1080P@30fps camera; motion control interface: EtherCAT connects to four DC motors (mobile) and a 6-axis robotic arm.

[0037] Storage unit: 16GB eMMC (system area, model area, backup area), 32MB SRAM (shared memory).

[0038] 2. Operating system layer configuration: RT-Thread 4.0, enable multi-level scheduling queues (levels 0-7), set FPGA control tasks (collision detection) to level 0, and GPU vision tasks to level 3.

[0039] 3. Algorithm framework layer configuration: On-device intelligent model: Lightweight YOLOv5s (3.5M parameters, INT8 quantization), GPU inference frame rate of 50 fps, and recognition accuracy of 92% (industrial parts dataset).

[0040] FPGA control algorithm: kinematics forward and inverse solution (3-stage pipeline), PID control (32-bit fixed point), control frequency 100kHz, single-step instruction generation delay 10ns.

[0041] 4. Operation process: Initialization: The system completes self-test within 1 second after startup, loads the YOLOv5s model (15MB) and FPGA bitstream, and initializes the API interface.

[0042] Data acquisition: The camera captures one frame every 33ms and writes it to the GPU memory via DMA (zero copy), which takes 8ms.

[0043] Visual processing: The GPU runs the YOLOv5s model, outputting part positions (e.g., (x=320, y=240, category=bolt)) within 18 ms, mapped to CPU shared memory via CUDA-IPC.

[0044] Resource scheduling: The CPU detects shared memory data, parses and generates control parameters (robotic arm target coordinates) within 1ms, and sends them to the FPGA via AXI-Stream (taking 40μs).

[0045] Control execution: The FPGA generates control instructions through a three-stage pipeline operation (30ns). The EtherCAT interface sends the instructions to the robotic arm within 1ms, driving it to move to the target position (positioning error ≤ 0.5mm).

[0046] State feedback: The 6-axis IMU provides feedback on the robotic arm's position every 1 ms. The CPU compares the visual results and adjusts the PID parameters in the FPGA (the proportional coefficient changes from 5.0 to 5.2) to achieve closed-loop control.

[0047] Model update: Upload a new model (optimized bolt recognition) through the API. The system completes background loading and switching (taking 45ms) when the robot is paused (idle state). During the switching, the robotic arm remains stationary and there is no jitter.

[0048] Experiments show that in small mobile grasping robots, the visual perception processing delay of this controller is stable at 18±2ms, the control instruction generation delay is 8±1ms, and the overall response speed is 80% higher than the traditional separate solution (industrial computer + controller); it can run continuously for 24 hours without any fault, and the part grasping success rate is over 95%; the power consumption is 15W under grasping conditions and 3W when stationary, and the battery life is improved by 50% compared with the traditional solution; the hardware cost is reduced by 40%, and the volume is reduced by 65% ​​(150mm×100mm×50mm), fully meeting the application requirements of small mobile robots.

[0049] Embodiment 2 includes embodiment 1, and based on embodiment 1, the present invention provides a technical solution: using a 4nm process GPU (such as Mali-G710) to improve the energy efficiency of visual processing and support higher resolution (4K) image input.

[0050] Expand FPGA to the Zynq UltraScale+ series and add AI acceleration engines (such as DPU) to achieve deep integration of edge control and lightweight AI.

[0051] Improve the safety certification mechanism (such as adding SE safety chip), support ISO 26262 functional safety standard, and adapt to autonomous driving robot scenarios.

[0052] Develop a cloud-edge collaborative platform to implement cloud-based training and batch deployment of edge models through the 5G interface, thereby improving the system's adaptability.

[0053] The drawings of the embodiments disclosed in the present invention only involve structures related to the embodiments disclosed in the present invention. Other structures may refer to conventional designs. In the absence of conflicts, the same embodiment and different embodiments of the present invention may be combined with each other. Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An embodied intelligent controller architecture based on an end-side intelligent model, comprising a hardware layer, an operating system layer, an algorithm framework layer, and an application interface layer, characterized by: The hardware layer uses a heterogeneous multi-core processor as its core, integrating the CPU, GPU, and FPGA. It configures a "star + NoC" hybrid topology interconnect structure and multiple DMA channels. The on-chip NoC supports consistent access to shared memory by the CPU, GPU, and FPGA. The hardware layer also includes an image acquisition interface, motion control interface, storage unit, and power management unit. The operating system layer is built on a real-time operating system and features a task priority scheduling mechanism. It uses a hybrid scheduling algorithm called "multi-level ready queue + EDF" and implements a hardware resource isolation mechanism. When an FPGA task is triggered, the CPU temporarily increases the priority of the associated task through a priority inheritance mechanism to ensure smooth data paths. The algorithm framework layer includes the device-side intelligent model module and the robot control algorithm module. The device-side intelligent model module uses a lightweight YOLOv5s model. By pruning to retain 60% of the channels, INT8 quantization, and BatchNorm integration, the model parameters are reduced to 3.5M. The robot control algorithm module implements three-stage pipeline hardware acceleration on the FPGA, supports a 100kHz control frequency, and uses the Q16.16 fixed-point format. The application interface layer provides a unified API software development interface, including a visual perception interface and a control interface. The interface supports zero-copy mapping in high-speed data paths, allowing developers to directly access the ring buffer address without data copying.

2. The embodied intelligent controller architecture based on a device-side intelligent model according to claim 1, characterized in that: The hardware layer includes: The CPU is connected to the GPU via a 4-way AXI-HP bus and to the FPGA via a 2-way AXI-Stream bus; There are 6 DMA channels, and the 6 DMA channels are configured as follows: 2 channels from GPU to CPU, 2 channels from CPU to FPGA, 1 channel from FPGA to CPU, and 1 channel from CPU to GPU. Each DMA channel supports a maximum transfer block size of 4MB and supports both interrupt triggering and polling modes. The image acquisition interface supports USB3.0 1080P@60fps input, the motion control interface adopts EtherCAT protocol, and the power management unit supports independent voltage domain control of CPU, GPU, and FPGA.

3. The embodied intelligent controller architecture based on a device-side intelligent model according to claim 1, characterized in that: The operating system layer includes: Task priorities are divided into levels 0-7, FPGA control tasks are fixed at level 0, GPU vision tasks are ≤ level 3, and CPU complex logic tasks are level 4-7; The GPU core occupancy time is limited to a 5ms / 20ms window to prevent long-term occupancy and blocking of high-priority tasks on the FPGA. Formal verification ensures that the worst-case scheduling delay is ≤ 200μs.

4. The embodied intelligent controller architecture based on a device-side intelligent model according to claim 1, characterized in that: The deployment process of the terminal-side intelligent model module adopts a double-buffered loading mechanism. When the model is loaded, it is bound to a fixed area through GPU memory address mapping, and TensorRT is used for kernel compilation to generate an optimized inference engine. Model hot switching is achieved through atomic pointer exchange, and the switching time is ≤10ms. When the new model reasoning fails, it automatically falls back to the original model.

5. The embodied intelligent controller architecture based on a device-side intelligent model according to claim 1, characterized in that: The three-stage pipeline of the robot control algorithm module includes: The input preprocessing stage performs data alignment and range checking; The core operation stage integrates a 32-bit multiplier and a CORDIC module; The output filtering stage uses sliding average.

6. The embodied intelligent controller architecture based on a device-side intelligent model according to claim 1, characterized in that: The algorithm framework layer also includes: Zero-copy and buffering: The GPU directly accesses CPU shared memory via CUDA-IPC, while the FPGA implements zero-copy across components through the AXI-Stream interface and the CPU user-mode ring buffer. A double-ended ring queue is used, and high and low watermark thresholds are set for the visual frame buffer. When the production rate exceeds the consumption rate, dynamic window sampling is triggered. The control instruction buffer uses a static watermark. When the watermark is exceeded, the oldest instruction is discarded and an overflow log is recorded to prevent accumulation of delays. Online update unit, supporting model MD5 segment checksum and FPGA bitstream ICAP hot replacement.

7. A method for operating an embodied intelligent controller architecture based on a device-side intelligent model according to any one of claims 1 to 6, characterized in that: The following steps are involved: S1, hardware self-test and driver initialization: After the system starts, the hardware layer completes self-test, which takes ≤1s. The operating system layer loads and initializes each hardware driver. The algorithm framework layer loads the lightweight YOLOv5s model and FPGA control algorithm bitstream. The application interface layer initializes the interface service. S2, data acquisition and processing: The image acquisition interface acquires 1080P image data via USB3.0 and directly writes it into the GPU memory via DMA, taking ≤10ms; The end-side intelligent model module on the GPU processes image data to obtain visual perception results, which are mapped to CPU shared memory via CUDA-IPC. The inference time is ≤ 20ms. S3, resource scheduling and task allocation: The CPU detects the visual perception results through the RTOS scheduling mechanism, upgrades it from level 3 to level 4, analyzes the results and extracts control parameters; Based on the system load, when the CPU utilization is less than 70%, the CPU sends the control parameters to the FPGA via the AXI-Stream bus. The FPGA control algorithm module generates control instructions through a three-stage pipeline operation. S4, control instruction execution: The motion control interface EtherCAT sends the control instructions generated by the FPGA to the execution component, with a cycle of 1ms; S5, state feedback and adjustment: The sensor collects robot state information and feeds it back to the CPU via the FPGA→CPUDMA channel; Based on the status information and visual perception results, the CPU adjusts the PID parameters in the FPGA through the algorithm framework layer to achieve closed-loop control; Multi-level power management and DVFS strategy: The power management unit monitors the temperature of each core in real time. When the GPU temperature exceeds 85°C, DVFS is triggered. When the robot is stationary, the CPU enters sleep mode, the GPU shuts down some cores, and the FPGA remains in low-power mode. Typical power consumption is 3W at rest, 8W at low speed, and 15W at high load.

Images