Robotic control system and method

Abstract

This application relates to the field of robot control technology and provides a robot control system and method, including a heterogeneous fusion hardware layer and a real-time operation layer. The heterogeneous fusion hardware layer is built on a heterogeneous system-on-a-chip (SoC), which includes a general-purpose processor core and a programmable logic unit (PLU). The PLU is configured with a vision-motion passthrough engine, which is configured to directly receive the structured perception results output by the vision recognition accelerator to calculate the end effector path point and send control commands to the motor driver through an industrial real-time bus interface. The real-time operation layer includes an event-driven vision-control scheduler, which is configured to generate a visual event carrying structured data after the vision recognition accelerator completes one frame of inference, so as to directly index the pre-stored motion primitive lookup table and trigger the execution of the corresponding motion primitive, thus solving the problems of high latency and poor stability in robot motion control.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a robot control system and method. Background Technology

[0002] With the development of embodied intelligence, robots need to independently complete the perception, decision-making, and execution loop in dynamic, unstructured environments. Visual recognition, as a core perception method, places extremely high demands on the computing power, real-time performance, and development efficiency of the main control platform. Embodied intelligence requires robots to complete the perception, decision-making, and action loop through real-time interaction with the physical environment. Vision is the robot's primary means of environmental perception. However, the visual processing in existing robot main control solutions is usually completed on a host computer (such as an industrial PC), sending results to the motion controller via Ethernet or USB. This results in significant communication latency and jitter, making it difficult to achieve sub-millimeter-level visual servoing. Existing systems mostly passively receive images from fixed cameras; when the target is occluded or the lighting is poor, they cannot actively adjust their own observation posture to improve recognition accuracy, which does not conform to the core concept of embodied intelligence—perception assisted by action. Simultaneously running deep learning visual models and high-frequency motion control (above 1kHz) on miniaturized embedded main controllers is challenging, resulting in either slow visual recognition or motion control malfunctions. The heterogeneous development environments and communication protocols for visual recognition, task planning, and motion control lead to complex overall system integration and difficult debugging. Summary of the Invention

[0003] This invention provides a robot control system and method to solve the problems of high delay and poor stability in robot motion control.

[0004] In a first aspect, embodiments of this application provide a robot control system, including a heterogeneous fusion hardware layer and a real-time operation layer; The heterogeneous fusion hardware layer is built on a heterogeneous system-on-a-chip, which includes a general-purpose processor core and a programmable logic unit. The programmable logic unit is equipped with a vision-motion pass-through engine, which is configured to directly receive the structured perception results output by the vision recognition accelerator to calculate the end effector path point and send control commands to the motor driver through the industrial real-time bus interface. The real-time execution layer includes an event-driven vision-control scheduler, which is configured to generate visual events carrying structured data after the vision recognition accelerator completes one frame of inference, so as to directly index a pre-stored motion primitive lookup table and trigger the execution of the corresponding motion primitive.

[0005] Secondly, the present invention provides a robot control method applied to a heterogeneous system-on-a-chip (SoC). The heterogeneous SoC includes a general-purpose processor core and a programmable logic unit (PLU). The PLU is configured with a reconfigurable visual recognition accelerator group, a vision-motion passthrough engine, and a shared memory and interrupt routing unit. The reconfigurable visual recognition accelerator group includes multiple visual recognition acceleration cores, including: The target information of the current task is obtained, and a visual recognition acceleration core and a motion primitive lookup table corresponding to the current task are configured in the programmable logic unit according to the target information; wherein, the motion primitive lookup table stores multiple motion primitives and their corresponding parameter mapping rules; Image data is acquired through the robot's vision module, and the image data is inferred by the visual recognition acceleration to generate a visual event package containing structured perception results. The visual event packets are directly transmitted to the visual-motion passthrough engine via the shared memory and interrupt routing unit. The hardware association matcher within the vision-motion passthrough engine matches the target motion primitive and its corresponding parameter mapping rule from the motion primitive lookup table based on the feature vector in the vision event package. The motion primitive generator within the vision-motion direct pass engine generates motion control commands in real time based on the target motion primitive and the parameter mapping rules, and outputs them to the actuator via a real-time bus.

[0006] In one solution provided by the aforementioned robot control system and method, a hardware direct path from visual recognition to motion control is constructed within the programmable logic unit. This allows visual event packets to be directly transmitted to the vision-motion direct path engine without passing through a general-purpose processor core, significantly shortening the end-to-end latency from visual perception to motion command generation and improving the real-time performance and determinism of robot control. Simultaneously, this solution offloads visual inference and motion primitive matching to hardware execution, reducing the load on the general-purpose processor core and avoiding latency jitter introduced by software scheduling. Furthermore, the reconfigurable visual recognition acceleration core and motion primitive lookup table can be dynamically configured according to the task, enabling the system to possess high performance, low latency, and flexible multi-tasking adaptability. Attached Figure Description

[0007] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0008] Figure 1This is a schematic diagram of the basic structure of a robot control system in one embodiment of the present invention; Figure 2 This is a schematic diagram of the operating logic of a robot control system in one embodiment of the present invention; Figure 3 This is a schematic diagram of the collaborative development tool layer of the robot control system in one embodiment of the present invention; Figure 4 This is a schematic diagram of the structure and operation logic of a reconfigurable visual recognition accelerator group in a robot control system according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the internal operating logic of the programmable logic unit of the robot control system in one embodiment of the present invention; Figure 6 This is a schematic diagram of the internal operation logic of the real-time operation layer of the robot control system in one embodiment of the present invention; Figure 7 This is a schematic diagram of the internal structure of the collaborative development tool layer of the robot control system in one embodiment of the present invention; Figure 8 This is a schematic diagram of the circuit structure of a robot control system in one embodiment of the present invention; Figure 9 This is a schematic diagram of the steps of a robot control method in one embodiment of the present invention. Detailed Implementation

[0009] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0010] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. It should also be understood that, as used in this specification and the appended claims, the term "and / or" refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0011] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0012] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0013] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0014] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0015] In one embodiment, such as Figures 1-9 As shown, a robot control system is provided, including a heterogeneous fusion hardware layer and a real-time operation layer; The heterogeneous fusion hardware layer is built on a heterogeneous system-on-a-chip, which includes a general-purpose processor core and a programmable logic unit. The programmable logic unit is equipped with a vision-motion pass-through engine, which is configured to directly receive the structured perception results output by the vision recognition accelerator to calculate the end effector path point and send control commands to the motor driver through the industrial real-time bus interface. The real-time execution layer includes an event-driven vision-control scheduler, which is configured to generate visual events carrying structured data after the vision recognition accelerator completes one frame of inference, so as to directly index a pre-stored motion primitive lookup table and trigger the execution of the corresponding motion primitive.

[0016] In this embodiment, the robot control system adopts a layered architecture, including at least a heterogeneous fusion hardware layer and a real-time operation layer. The heterogeneous fusion hardware layer constitutes the physical execution foundation of the system, and the real-time operation layer constitutes the scheduling and control logic foundation of the system. The two work together to achieve a low-latency, deterministic closed loop from visual perception to motion control.

[0017] In one embodiment, the robot control system can be used to control a robot with a multi-joint robotic arm. The robot's end effector is equipped with end effectors such as grippers, suction cups, and welding torches. The visual recognition accelerator can perform target detection, pose estimation, or scene segmentation on images captured by the camera and output structured perception results. The structured perception results are not the original image data, but rather data processed by the accelerator, such as target category, bounding box, confidence score, 6D pose, segmentation mask, and key point coordinates, which are convenient for subsequent control.

[0018] The heterogeneous converged hardware layer is built on a heterogeneous system-on-a-chip (SoC), which refers to a system that integrates at least two different computing architectures of processing resources within the same chip or package. For example, the heterogeneous SoC may be a Zynq series chip containing an ARM processor core and an FPGA programmable logic unit, an Intel SoC FPGA containing an x86 processor core and an FPGA, or other heterogeneous chips containing a RISC-V core and a programmable logic unit.

[0019] The heterogeneous system-on-a-chip includes a general-purpose processor core and a programmable logic unit. The general-purpose processor core can be configured to run a standard operating system, such as Linux, RT-Linux, or other general-purpose operating systems. The general-purpose processor core can perform functions such as upper-layer task planning, non-real-time human-computer interaction, logging, and host-side proxy for development tools. Since these tasks typically have low real-time requirements, executing them with a general-purpose processor core can fully utilize its versatility and ecosystem advantages.

[0020] The programmable logic unit (PLU) can be an FPGA, CPLD, programmable array, or other reconfigurable logic resource. The PLU features strong parallel processing capabilities, high timing determinism, and low latency, making it suitable for hard real-time tasks. The PLU is equipped with a vision-motion passthrough engine, which is a hardware logic module or hardware state machine located within the PLU. The vision-motion passthrough engine is configured to directly receive the structured perception results output by the vision recognition accelerator. Here, "directly receiving" means that the structured perception results output by the vision recognition accelerator do not need to pass through a general-purpose processor core, an operating system, or traditional robot middleware. Instead, they are directly transmitted to the vision-motion passthrough engine through the PLU's internal data path, streaming interface, shared memory, or direct memory access channel. This avoids the latency and jitter caused by CPU interrupt handling, process scheduling, and data copying.

[0021] In one embodiment, a visual recognition accelerator is also configured in the programmable logic unit. After the visual recognition accelerator completes the inference of a frame of image, it writes the structured perception result into the shared memory or streaming FIFO in the programmable logic unit. The vision-motion passthrough engine directly reads the structured perception result from the shared memory or streaming FIFO, thereby forming a hardware-level passthrough path.

[0022] After receiving the structured perception results, the vision-motion direct-access engine is configured to calculate the path points of the end effector. The path points may include the position, orientation, velocity, and acceleration of the end effector in Cartesian space, and may also include the angle and angular velocity in joint space. The methods for calculating the path points may include: performing coordinate transformation based on the target pose and the robot's current pose; generating straight lines, arcs, or spline paths based on preset geometric relationships; or mapping the Cartesian space target to joint space commands through a pre-stored inverse kinematics lookup table.

[0023] The vision-motion direct-access engine also sends control commands to the motor driver via an industrial real-time bus interface. The industrial real-time bus interface can be a fieldbus interface suitable for industrial real-time control, such as EtherCAT, CANopen, Powerlink, Profinet IRT, or Sercos. In one embodiment, the industrial real-time bus interface is an EtherCAT interface. The control commands can include position commands, speed commands, torque commands, enable signals, synchronization signals, etc. After receiving the control commands, the motor driver drives the corresponding motor to move, thereby causing the end effector to move according to the calculated path point.

[0024] Because the vision-motion direct-access engine is located in the programmable logic unit, its process of calculating and sending control commands has hardware determinism and low latency characteristics. The link from vision results to motor commands is greatly shortened, thereby improving the real-time performance of the robot's vision closed loop.

[0025] The real-time operation layer includes an event-driven vision-control scheduler, which can be implemented as hardware logic, software program, firmware, or a combination thereof. In one embodiment, the vision-control scheduler runs on a microprocessor within the programmable logic unit, or on a coprocessor that works in conjunction with the programmable logic unit.

[0026] The vision-control scheduler is configured to generate a visual event carrying structured data after the vision recognition accelerator completes one frame of inference. The visual event can be triggered by the vision recognition accelerator through an interrupt, a flag, a message queue, or a hardware signal after inference is completed. The structured data carried in the visual event may include: frame identifier, timestamp, number of targets, target category, target pose, target velocity, confidence level, segmentation result index, etc.

[0027] After receiving a visual event, the vision-control scheduler is configured to directly index a pre-stored motion primitive lookup table. The motion primitive lookup table can be stored in block memory, distributed memory, or external non-volatile memory within a programmable logic unit. The motion primitive lookup table stores the mapping relationship between visual result features and motion primitives. For example, the mapping relationship can be: when the target category is workpiece A and the pose is within the grasping area, the index is directed to the grasping motion primitive; when the target category is an obstacle and the distance is less than a threshold, the index is directed to the avoidance motion primitive; when the target pose is an assembly hole, the index is directed to the insertion motion primitive.

[0028] The vision-control scheduler directly indexes the motion primitive lookup table and triggers the execution of the corresponding motion primitive. The motion primitive can be a predefined action primitive, such as approach, grasp, lift, move, place, rotate, track, etc. Each motion primitive can correspond to a set of preset trajectory parameters, velocity parameters, acceleration parameters, force control parameters, or joint space command sequences. Triggering the execution of the corresponding motion primitive can be done by loading the parameters corresponding to the motion primitive into the vision-motion passthrough engine, or by sending the motion primitive identifier to the deterministic motion control middleware to start the execution of the motion primitive.

[0029] Because the vision-control scheduler adopts an event-driven approach and directly indexes motion primitives through lookup tables, there is no need for complex task planning or trajectory optimization at runtime. Once a vision event is generated, the corresponding motion primitive can be quickly identified and triggered, thereby further reducing the vision-to-control response time.

[0030] See Figure 8 , Figure 8 This is a schematic diagram of the internal structure of the collaborative development tool layer of the robot control system in one embodiment of the present invention. The circuit connection relationship shows the various electronic components, including heterogeneous on-chip systems, to realize the actual electrical connection layout of the robot control system.

[0031] See Figure 3 In one embodiment, a collaborative development tools layer is also included; The collaborative development tool layer includes a vision-control flow graph editor and a heterogeneous code automatic compilation and deployment tool. The vision-control flow graph editor is configured to construct a data flow graph consisting of a vision processing module and a control primitive module by dragging and dropping. The data flow graph is used to define the mapping relationship from vision output fields to motion primitive parameters. The heterogeneous code automatic compilation and deployment tool is configured to generate a deployable unified firmware image based on the data flow graph.

[0032] The vision-control flow graph editor is configured to build data flow graphs using a drag-and-drop method. The data flow graph consists of a vision processing module and a control primitive module.

[0033] In one embodiment, the vision-control flow graph editor provides a graphical user interface, which includes a module library, a canvas, a connection tool, and an attribute panel. The module library contains multiple vision processing modules and multiple control primitive modules. Developers can drag and drop the required modules from the module library onto the canvas and connect the input and output ports of each module using the connection tool to construct the data flow graph.

[0034] The vision processing module may include, but is not limited to, an image acquisition module, an image preprocessing module, an object detection module, a 6D pose estimation module, a scene segmentation module, a feature extraction module, a coordinate transformation module, a filtering module, and a tracking module. Each vision processing module may have one or more input ports and output ports, and the output ports output vision output fields, such as object category, confidence score, bounding box coordinates, object pose, segmentation mask, key point coordinates, etc.

[0035] The control element modules may include, but are not limited to: proximity motion elements, grasping motion elements, lifting motion elements, moving motion elements, placing motion elements, rotating motion elements, tracking motion elements, avoiding motion elements, force control elements, etc. Each control element module may have one or more parameter input ports and execution output ports. The parameter input ports are used to receive motion element parameters, such as target position, target attitude, velocity, acceleration, force threshold, grasping width, etc., and the execution output ports are used to output motion element trigger signals or motion element execution commands.

[0036] The data flow graph is used to define the mapping relationship from visual output fields to motion primitive parameters. In one embodiment, developers can connect the output port of a vision processing module to the parameter input port of a control primitive module by wiring, thereby establishing the mapping relationship from visual output fields to motion primitive parameters. For example, the target pose.x field output by the target detection module can be connected to the target position.x parameter port of the grasping motion primitive module, the target pose.y field can be connected to the target position.y parameter port, and the target pose.angle field can be connected to the target attitude.yaw angle parameter port.

[0037] In another embodiment, the mapping relationship may also include conditional mapping, expression mapping, lookup table mapping, or scaling mapping. For example, a condition judgment module can be inserted into the data flow diagram. When the target category is workpiece A, the visual output field is mapped to the grasping motion primitive parameter; when the target category is obstacle, the visual output field is mapped to the avoidance motion primitive parameter. For another example, a mapping formula can be defined through an expression edit box, such as motion primitive speed = visual confidence level × maximum speed. For yet another example, the visual output field can be used as an index and mapped to motion primitive parameters through a pre-stored lookup table.

[0038] The vision-control flow graph editor can also be configured to perform syntax checks, type checks, and connection integrity checks on the data flow graph. For example, it can check whether the data type of the vision output field matches the data type of the motion primitive parameter, check whether there are any unconnected required input ports, check whether there are any circular dependencies, etc. The editor can also provide a simulation preview function to simulate the execution result of the data flow graph without deploying hardware.

[0039] The heterogeneous code automatic compilation and deployment tool is configured to generate a deployable unified firmware image based on the data flow graph. In one embodiment, the heterogeneous code automatic compilation and deployment tool receives a data flow graph description file output by the visual-control flow graph editor. The data flow graph description file may be XML, JSON, YAML or other structured formats, including information such as module type, module parameters, connection relationships, and mapping relationships.

[0040] The heterogeneous code automatic compilation and deployment tool parses and divides the data flow graph, mapping the vision processing module and control primitive module in the data flow graph to different computing resources of the heterogeneous on-chip system. For example, it maps the computationally intensive, low-latency vision processing module to the vision recognition accelerator in the programmable logic unit; it maps the event-driven scheduling logic and motion primitive lookup table to the vision-motion passthrough engine or real-time execution layer in the programmable logic unit; and it maps the non-real-time human-computer interaction, log recording, and task management state machine to the general-purpose processor core.

[0041] In one embodiment, the heterogeneous code automatic compilation and deployment tool is further configured to: compile the connection relationships of the vision modules in the data flow graph into a streaming interface configuration and direct memory access transfer descriptor between accelerators within the programmable logic unit. For example, if the output of the object detection module in the data flow graph is connected to the input of the 6D pose estimation module, the compilation tool generates a corresponding streaming interface configuration, so that the output of the object detection accelerator is transferred to the input of the 6D pose estimation accelerator via first-in-first-out memory or shared memory; at the same time, a direct memory access transfer descriptor is generated to describe the length, address, transmission direction, triggering conditions, etc. of the data block.

[0042] Based on the mapping relationship from visual results to control primitives, the hardware lookup table initialization data of the vision-motion passthrough engine and the task management state machine code on the general-purpose processor core are generated. For example, if the data flow graph defines the mapping relationship of target category = workpiece A → grasping motion primitive, the compilation tool generates the initialization data of the motion primitive lookup table and writes the mapping relationship into the lookup table; at the same time, the task management state machine code on the general-purpose processor core is generated to manage the task queue, state switching and exception handling.

[0043] The heterogeneous code automatic compilation and deployment tool can also generate configuration files for the programmable logic units, such as bitstream files; generate software code for the general-purpose processor core, such as C / C++ code, device trees, drivers, and applications; generate configuration parameters for the real-time operation layer, such as motion primitive queue depth, safety monitoring thresholds, and scheduling cycles; and generate communication configurations for the industrial real-time bus interface, such as EtherCAT slave configurations and process data object mappings.

[0044] The heterogeneous code automatic compilation and deployment tool links the aforementioned configuration files, software code, initialization data, and configuration parameters into a unified firmware image. This unified firmware image can be a single binary file, a single image file, or a single deployment package. The unified firmware image can be deployed to the heterogeneous system-on-a-chip (SoC) via a debug interface, Ethernet interface, USB interface, or industrial bus interface. After deployment, the heterogeneous SoC operates according to the vision-control logic defined in the data flow diagram.

[0045] In one embodiment, the heterogeneous code automatic compilation and deployment tool also supports incremental compilation. When developers modify some modules or mapping relationships in the data flow diagram, the compilation tool only recompiles the affected modules and links, without having to recompile the entire system, thereby improving development iteration efficiency.

[0046] In another embodiment, the collaborative development tool layer may also include version management interfaces and remote deployment interfaces to support team collaboration and on-site updates.

[0047] See Figure 4 In one embodiment, the programmable logic unit is further configured with a reconfigurable visual recognition accelerator group, which includes multiple dynamically loadable hardware acceleration cores and is configured to switch the hardware acceleration cores without interrupting the main control operation through partial reconfiguration technology to adapt to different visual recognition tasks; the visual recognition tasks include target detection, 6D pose estimation, and scene segmentation.

[0048] In this embodiment, the reconfigurable visual recognition accelerator group is configured to switch the hardware acceleration core without interrupting the main control operation through partial reconfiguration technology. The main control may include the general-purpose processor core, the real-time operation layer, or other continuously running logic in the programmable logic unit. Specifically, the programmable logic unit can be divided into a partially reconfigurable region. The hardware acceleration core is stored in external memory or configuration memory in the form of a partial bitstream. When a new visual recognition task needs to be adapted, the real-time operation layer or the vision-control scheduler can trigger a partial reconfiguration process to load a partial bitstream of the target hardware acceleration core into the partially reconfigurable region, thereby dynamically replacing the current hardware acceleration core. During the switching process, the remaining logic in the programmable logic unit, the general-purpose processor core, and the real-time operation layer continue to run without requiring a complete chip reset or restart.

[0049] The visual recognition task includes target detection, 6D pose estimation, and scene segmentation. When the robot needs to switch between different processes, such as switching from target grasping to precision assembly, the reconfigurable visual recognition accelerator group can switch from the target detection acceleration core to the 6D pose estimation acceleration core accordingly. When environmental understanding or obstacle avoidance is required, it can switch to the scene segmentation acceleration core. The hardware acceleration core after switching outputs structured perception results, which can be directly provided to the vision-motion passthrough engine to participate in the end effector path point calculation and motion primitive triggering.

[0050] The reconfigurable visual recognition accelerator group can also cooperate with shared memory and interrupt routing units to enable different hardware acceleration cores to reuse the same data path, cache resources and interrupt mechanism before and after switching, thereby reducing switching overhead. Compared with related technologies, this embodiment realizes dynamic switching of hardware acceleration cores through partial reconfigurable technology, so that the same programmable logic unit can adapt to multiple visual recognition tasks without interrupting the main control operation, taking into account the real-time performance, resource utilization and task flexibility of hardware acceleration.

[0051] See Figure 5In one embodiment, the programmable logic unit is further configured with a shared memory and interrupt routing unit, which is configured to implement data exchange between the visual recognition accelerator within the programmable logic unit, the general-purpose processor core, and external sensor data streams.

[0052] The shared memory and interrupt routing unit includes a shared storage area, a direct memory access controller, and interrupt routing logic. The shared storage area can be divided into multiple buffers or circular queues to cache external sensor data, input data from the visual recognition accelerator, and structured perception results, respectively.

[0053] The shared memory and interrupt routing unit is configured to enable data exchange between the visual recognition accelerator within the programmable logic unit, the general-purpose processor core, and external sensor data streams. Specifically, the external sensor data streams are written to the shared memory area via the sensor interface and the direct memory access controller. The visual recognition accelerator reads data from the shared memory area and writes the inferred structured perception results back to the shared memory area. The general-purpose processor core accesses the shared memory area via the system bus to perform configuration, monitoring, or non-real-time processing. The vision-motion passthrough engine can directly read the structured perception results from the shared memory area to calculate the end effector path points.

[0054] The interrupt routing logic receives interrupt or event signals generated by external sensor interfaces, direct memory access controllers, and vision recognition accelerators, and routes them to the general-purpose processor core, the vision-control scheduler, or the real-time runtime layer according to pre-configured routing relationships. The interrupt routing logic can also perform priority arbitration, interrupt merging, masking, or clearing. For example, after the vision recognition accelerator completes one frame of inference, it generates an inference completion interrupt. This interrupt can be routed to the vision-control scheduler via the interrupt routing logic to trigger a vision event carrying structured data.

[0055] Compared with related technologies, this embodiment establishes a low-latency data path and event synchronization mechanism between the visual recognition accelerator, the general-purpose processor core and the external sensor data stream through shared memory and interrupt routing unit, thereby reducing data copying and intervention of the general-purpose processor core, and improving data exchange efficiency and real-time determinism.

[0056] In one embodiment, the general-purpose processor core is configured to run a standard operating system and perform upper-layer task planning, non-real-time human-computer interaction, and host-side proxy functions for development tools.

[0057] The general-purpose processor core is configured to run a standard operating system. The standard operating system can be Linux, embedded Linux, etc., providing general functions such as process scheduling, memory management, file system, network protocol stack, and device drivers. The general-purpose processor core is isolated from the hard real-time logic in the programmable logic unit and does not participate in the hard real-time closed loop of the vision-motion passthrough engine.

[0058] The general-purpose processor core executes upper-layer task planning, such as receiving production tasks or user instructions, performing task decomposition, process scheduling and resource allocation, generating motion primitive sequences or updating motion primitive lookup tables, and sending the corresponding configurations to the real-time execution layer. The above task planning is a non-real-time or soft real-time task.

[0059] The general-purpose processor core performs non-real-time human-machine interaction, such as providing a graphical user interface, teach pendant interface, status monitoring, parameter configuration, alarm display, log recording, and remote communication.

[0060] The general-purpose processor core also performs host-side proxy functions for development tools. For example, the general-purpose processor core acts as a proxy between the host development environment and the heterogeneous on-chip system, communicating with the visual-control flow graph editor and heterogeneous code automatic compilation and deployment tools on the host, receiving data flow graph descriptions, unified firmware images or debugging instructions, and performing firmware deployment, online debugging, log feedback, version management and configuration updates.

[0061] Compared with related technologies, this embodiment uses a general-purpose processor core to handle non-real-time tasks, allowing the programmable logic unit to focus on hard real-time vision-motion control, thus balancing system flexibility, scalability, and real-time determinism.

[0062] See Figure 6 In one embodiment, the real-time runtime layer further includes a visual recognition acceleration library and a deterministic motion control middleware; the visual recognition acceleration library is configured to provide a unified application programming interface for the hardware accelerators in the programmable logic unit to support loading, running, and destroying different visual model acceleration cores; the deterministic motion control middleware is configured to run on a coprocessor or a microprocessor within the programmable logic unit to manage a queue of motion primitives and to perform security monitoring of the trigger signals of the vision-motion passthrough engine.

[0063] The visual recognition acceleration library is configured to provide a unified application programming interface (API) for the hardware accelerators in the programmable logic unit (PLU). This API shields the upper layer from the differences in register mapping, direct memory access (DMI) configuration, interrupt numbers, and configuration timing among different hardware acceleration cores. Through this interface, the upper layer can perform loading, running, and destruction operations on different visual model acceleration cores: loading is used to write the configuration data or a portion of the bitstream of the target visual model acceleration core into the PLU; running is used to start inference and pass input / output buffers; and destruction is used to stop the acceleration core and release DMI channels, shared memory, and interrupt resources. This supports the dynamic management and switching of different visual model acceleration cores.

[0064] The deterministic motion control middleware is configured to run on a coprocessor or a microprocessor within the programmable logic unit. The coprocessor can be a real-time coprocessor that works with a heterogeneous on-chip system. The microprocessor within the programmable logic unit can be a soft-core processor such as MicroBlaze or RISC-V. The deterministic motion control middleware is used to manage the motion primitive queue, for example, receiving motion primitives triggered by the vision-control scheduler, enqueuing, dequeuing, and scheduling execution according to priority, timing, and dependencies, and providing motion primitive parameters to the vision-motion passthrough engine.

[0065] Meanwhile, the deterministic motion control middleware performs safety monitoring on the trigger signals of the vision-motion direct pass engine. For example, it monitors whether the trigger signals are valid, whether they time out, whether they exceed the limit, whether they violate the execution order, whether the motion primitive queue overflows, and whether the speed or acceleration exceeds the limit. When an abnormality is detected, the deterministic motion control middleware can block the trigger signals, clear the motion primitive queue, output an alarm, or put the robot into a safe state.

[0066] See Figure 7 In one embodiment, the heterogeneous code automatic compilation and deployment tool is configured as follows: The connection relationships of the vision modules in the data flow graph are compiled into a streaming interface configuration and direct memory access transfer descriptor between accelerators within the programmable logic unit. Based on the mapping relationship from visual results to control primitives, the hardware lookup table initialization data of the vision-motion passthrough engine and the task management state machine code on the general-purpose processor core are generated.

[0067] The connection relationships of the vision modules in the data flow graph are compiled into a streaming interface configuration and direct memory access transfer descriptor between accelerators within the programmable logic unit. Specifically, the compilation tool parses the connection relationships between vision processing modules and generates a streaming interface configuration to define the data flow direction, data bit width, handshake protocol, first-in-first-out memory depth, and backpressure mechanism between adjacent hardware acceleration cores. At the same time, it generates a direct memory access transfer descriptor to define the source address, destination address, transmission length, transmission triggering condition, and interrupt response mode, thereby realizing the efficient flow of data between vision modules within the programmable logic unit.

[0068] Based on the mapping relationship from visual results to control primitives, the hardware lookup table initialization data of the vision-motion passthrough engine and the task management state machine code on the general-purpose processor core are generated. Specifically, the compilation tool converts the mapping relationship between visual output fields defined in the data flow graph and motion primitive parameters or motion primitive identifiers into lookup table initialization data, which is directly indexed by the vision-motion passthrough engine at runtime. At the same time, the task management state machine code on the general-purpose processor core is generated to perform task state switching, motion primitive queue management, exception handling, and non-real-time scheduling.

[0069] Compared with related technologies, this embodiment automatically maps the graphical data flow graph into a streaming interface, direct memory access descriptor, hardware lookup table initialization data, and general processor core code within the programmable logic unit, reducing manual coding and improving the consistency and deployment efficiency of heterogeneous hardware and software configurations.

[0070] In one embodiment, the collaborative development tool layer further includes a hardware-in-the-loop simulation and performance profiler, which is configured to: During the development phase, the timing of the vision-motion direct path is simulated, and sensor data is injected for algorithm verification. During the operation phase, the timestamps from the triggering of visual events to the output of motor commands are captured in real time, and the time consumed in each stage is displayed in a waterfall chart.

[0071] During the development phase, the hardware-in-the-loop simulation and performance profiler is configured to simulate the timing of the vision-motion passthrough path and inject sensor data for algorithm verification. Specifically, it can construct a timing model that includes a vision recognition accelerator, a vision-motion passthrough engine, an industrial real-time bus interface, and a motor driver. Synthesized or recorded sensor data is injected into the data flow graph to verify whether the mapping relationship between the vision processing module, the vision output field and the motion primitive parameters, the motion primitive triggering logic, and the safety monitoring strategy meet the expected timing and functional requirements. Thus, algorithm and logic verification can be completed without deploying a complete robot system.

[0072] During the runtime phase, the hardware-in-the-loop simulation and performance profiler is configured to capture the timestamps from the triggering of visual events to the output of motor commands in real time, and display the time consumption of each stage in a waterfall chart. Specifically, it can set timestamp capture points at nodes such as visual event generation, motion primitive lookup table indexing, end effector path point calculation, control command sending, industrial real-time bus transmission, and motor command output, record the corresponding timestamps, calculate the time consumption of each stage, and then generate a waterfall chart arranged by the time axis. The waterfall chart can be used to display the time consumption, jitter, and worst execution time of each stage, which is convenient for locating latency bottlenecks and optimizing real-time performance.

[0073] Compared with related technologies, this embodiment supports timing simulation and algorithm verification during the development phase and end-to-end timestamp capture and waterfall plot analysis during the runtime phase through hardware-in-the-loop simulation and performance profiler, thereby improving the observability, verifiability and real-time performance optimization efficiency of the vision-motion direct path.

[0074] In one embodiment, the industrial real-time bus interface includes an EtherCAT interface.

[0075] Secondly, the present invention provides a robot control method, such as... Figure 9 As shown, this is applied to a heterogeneous system-on-a-chip (SoC). The heterogeneous SoC includes a general-purpose processor core and a programmable logic unit (PLU). The PLU is configured with a reconfigurable visual recognition accelerator group, a vision-motion passthrough engine, and a shared memory and interrupt routing unit. The reconfigurable visual recognition accelerator group includes multiple visual recognition acceleration cores, including: S10: Obtain the target information of the current task, and configure the visual recognition acceleration core and motion primitive lookup table corresponding to the current task in the programmable logic unit according to the target information; wherein, the motion primitive lookup table stores multiple motion primitives and their corresponding parameter mapping rules; S20: Collect image data through the robot's vision module, and use the visual recognition to accelerate the inference of the image data to generate a visual event package containing structured perception results; S30: The visual event packet is transmitted directly to the visual-motion pass-through engine through the shared memory and interrupt routing unit without passing through the general-purpose processor core; S40: The hardware association matcher in the vision-motion direct pass-through engine matches the target motion primitive and the corresponding parameter mapping rule from the motion primitive lookup table based on the feature vector in the vision event package; S50: The motion primitive generator in the vision-motion direct pass engine generates motion control commands in real time according to the target motion primitive and the parameter mapping rules, and outputs them to the actuator through the real-time bus.

[0076] In this embodiment, the heterogeneous system-on-a-chip (SoC) includes a general-purpose processor core (e.g., ARM Cortex-A series) and a programmable logic unit (e.g., FPGA fabric). The programmable logic unit is divided into multiple dynamically loadable regions through partial reconfiguration technology, including a reconfigurable visual recognition accelerator group, a vision-motion passthrough engine, and a shared memory and interrupt routing unit. The reconfigurable visual recognition accelerator group consists of multiple independently configurable visual recognition acceleration cores, each corresponding to a visual recognition task, such as object detection, semantic segmentation, grasp point localization, obstacle recognition, etc.

[0077] In step S10, the system first obtains the target information of the current task. The target information can come from the upper-level task planner, operator instructions, or a preset task sequence. Its content includes the task type (e.g., "grab workpiece A"), the required visual recognition task identifier, the visual feature description of the target object, motion constraints, etc. After the general-purpose processor core parses the target information, it generates configuration instructions and writes them to the configuration register of the programmable logic unit through the configuration bus (e.g., AXI-Lite or PCAP interface). The configuration instructions include at least: the identifier or bitstream address of the visual recognition acceleration core to be loaded; the initialization data of the motion primitive lookup table, which stores multiple motion primitives and their corresponding parameter mapping rules; and the data path parameters between the visual recognition acceleration core and the vision-motion passthrough engine, such as shared memory address mapping and interrupt routing table.

[0078] Taking partially reconfigurable technology as an example, when switching the visual recognition acceleration core is required, the reconfiguration controller (such as the ICAP primitive) inside the programmable logic unit loads the new acceleration core bitstream into the designated reconfigurable region without interrupting the operation of the general-purpose processor core. Simultaneously, the motion primitive lookup table can be written by the general-purpose processor core to the block RAM (BRAM) or distributed RAM within the programmable logic unit via DMA, or loaded from external flash memory by the hardware initialization state machine. Each entry in the motion primitive lookup table includes: a motion primitive identifier (such as "straight-line approach," "circular arc grasp," or "obstacle avoidance"), a feature vector template or index value, and parameter mapping rules (e.g., mapping the object position in the visual features to the target pose, velocity, acceleration, and other parameters of the end effector).

[0079] Through this configuration step, the programmable logic unit is customized into dedicated vision-motion processing hardware that matches the current task, avoiding unnecessary general computing overhead.

[0080] The robot's vision module can be an industrial camera, depth camera, binocular camera, or event camera. The image data acquired by the vision module is transmitted to the programmable logic unit (PLU) through a high-speed interface (such as MIPI CSI-2, LVDS, USB3 Vision, or GigE Vision). Inside the PLU, the image data first goes through a preprocessing module (such as ISP, noise reduction, image scaling, and format conversion) and then enters the configured vision recognition acceleration core.

[0081] Visual recognition acceleration cores are hardware accelerators optimized for specific visual tasks. Their internal implementation can include convolutional neural network (CNN) inference engines, feature extraction pipelines, template matching units, etc. Acceleration cores are designed based on hardware description languages ​​(such as Verilog / VHDL) or high-level synthesis (HLS), support fixed-point or floating-point operations, and utilize the parallelism of FPGAs to achieve high-throughput inference. For example, for object detection tasks, acceleration cores can perform convolution, pooling, activation and other operations on input images, and output results such as detection boxes, class confidence, key point coordinates, etc.

[0082] After inference, the visual recognition acceleration kernel generates structured perception results. These structured perception results are not raw images or unorganized tensors, but rather semantic information that has been parsed by the hardware and can be directly used for motion control. For example, a visual event package may contain the following fields: event type identifier (e.g., "object detected," "grab point located," "obstacle appears on the left"); timestamp, recording the time the event occurred; feature vectors, such as the target object's position coordinates, pose quaternions, size, velocity estimate, confidence level, etc.; associated image frame number or region identifier; and optional priority markers for subsequent arbitration.

[0083] The data structure of the visual event packet is defined during the configuration phase and stored in a specific area of ​​shared memory. The visual recognition acceleration core writes the event packet into a pre-allocated buffer in shared memory through the DMA engine and sends a trigger signal to the vision-motion passthrough engine through the interrupt routing unit. If multiple visual recognition acceleration cores work in parallel, multiple visual event packets may be generated at the same time. These event packets are written to different addresses in the same format or managed uniformly by the hardware arbitrator.

[0084] In traditional solutions, visual results need to be read, parsed, and then transmitted to the motion controller by a general-purpose processor core, involving multiple memory copies, system calls, and software scheduling. This invention constructs a hardware-level pass-through data path through shared memory and an interrupt routing unit.

[0085] Specifically, the shared memory is implemented by a reserved area of ​​BRAM within the programmable logic unit or external DDR, which can be accessed simultaneously by the vision recognition acceleration core and the vision-motion passthrough engine. The interrupt routing unit is a hardware module that can directly route interrupt signals from the vision recognition acceleration core to the trigger port of the vision-motion passthrough engine according to the configuration, without going through the interrupt controller and interrupt service routine of the general-purpose processor core. The interrupt routing unit can support multiple inputs, priority encoding, masking and enable control.

[0086] Once the visual recognition acceleration core completes inference and writes the visual event packet to shared memory, its internal completion flag or status register triggers an interrupt request. The interrupt routing unit captures this interrupt and sends a trigger signal to the corresponding input channel of the vision-motion passthrough engine according to a preset routing table. Upon receiving the trigger signal, the vision-motion passthrough engine directly reads the visual event packet from shared memory. Throughout this process, the general-purpose processor core does not participate in data transfer or event transmission; it is only interrupted when monitoring or exception handling is required.

[0087] This hardware pass-through mechanism eliminates the latency and uncertainty of the software protocol stack, enabling the interval from the completion of visual reasoning to the generation of motion control commands to be shortened to the microsecond level.

[0088] The hardware associative matcher within the vision-motion passthrough engine is responsible for establishing the mapping relationship between visual perception results and motion primitives. The hardware associative matcher can be designed as a content-addressable memory (CAM) or a matching network based on a parallel comparator.

[0089] In practice, the hardware associative matcher extracts feature vectors from visual event packets. These feature vectors can be high-dimensional, such as containing the 3D position, pose, and category encoding of the target object. The motion primitive lookup table stores multiple motion primitive templates. Each template contains a feature vector reference value or feature range, as well as the corresponding motion primitive identifier and parameter mapping rules. The matching process can adopt one of the following methods: nearest neighbor matching: calculate the Euclidean distance or cosine similarity between the input feature vector and the feature vectors of each template in the lookup table, and select the entry with the smallest distance or the highest similarity as the matching result. To accelerate the calculation, parallel distance calculation units and sorting networks can be used in the hardware; index matching: if some fields in the feature vector (such as the object category ID) can be directly used as the lookup table address, the hardware directly reads the corresponding entry through address decoding to achieve zero-latency matching; fuzzy matching: for continuous feature values, multiple intervals or fuzzy membership functions can be set. The hardware comparator determines which interval the input value falls into, thereby selecting the corresponding entry.

[0090] After a successful match, the hardware association matcher outputs the identifier of the target motion primitive and the parameter mapping rules. The parameter mapping rules define how to map the specific values ​​in the visual event packet to the control parameters of the motion primitive. For example, the parameter mapping rules can be a set of affine transformation equations, lookup tables, or polynomial coefficients to convert the object coordinates into the target pose of the robot end effector, or to map the object's motion velocity to the velocity command of the robot joint.

[0091] The motion primitive generator is the execution unit in the vision-motion passthrough engine. It is responsible for instantiating selected motion primitives into specific motion control command sequences. The motion primitive generator can be implemented by a hardware state machine, a digital signal processing (DSP) unit, or a dedicated motion control IP core. The motion primitive generator has pre-built algorithm descriptions for various basic motion modes, such as: point-to-point motion: generating trapezoidal or S-shaped velocity curves based on the target pose and calculating the position and velocity commands of each joint; linear trajectory motion: performing linear interpolation in Cartesian space and solving for joint angles through inverse kinematics; circular trajectory motion: performing circular interpolation; force-controlled or impedance-controlled motion: adjusting position commands based on force feedback.

[0092] The parameter mapping rules provide the input parameters required for the above motion modes, such as target pose, motion time, speed limit, acceleration limit, force threshold, etc. The motion element generator calculates the desired joint position, speed or torque for each control cycle in real time based on these parameters, and packages the results into instruction frames that conform to the real-time bus protocol. The real-time bus can be EtherCAT, PROFINET, CANopen, Sercos III or a dedicated pulse direction interface, etc.

[0093] Motion control commands are output to actuators (such as servo motor drivers, hydraulic valves, pneumatic valves, etc.) via a real-time bus. Since the entire process is completed in hardware, the control cycle can be extremely short (e.g., less than 100 microseconds) and has strict determinism.

[0094] In one embodiment, when multiple visual recognition acceleration cores generate multiple visual event packets in parallel, the hardware arbitrator in the visual-motion passthrough engine fuses the multiple visual events according to preset priorities and timing constraints to determine the final triggered target motion primitive and parameter mapping rules.

[0095] In one embodiment, multiple visual events are fused according to preset priorities and temporal constraints, specifically including the following steps: S01: Obtain the timestamp of each visual event package, match the event type of each visual event package according to the preset priority, obtain the corresponding priority, and generate channel weights for each visual event package according to the timestamp and the priority; S02: Input the weights of each channel into a pulse-side inhibition competition network composed of mutually inhibiting nodes to calculate the net activation value of each node; S03: When the net activation value of only one node exceeds the preset threshold, select the motion primitive and parameter mapping rule corresponding to that node; S04: When the net activation values ​​of multiple nodes all exceed the preset threshold and the mutual inhibition coefficient between nodes is less than the preset value, the net activation values ​​of each node are used as weights to perform a weighted summation of the motion primitives and parameter mapping rules corresponding to each visual event package, so as to obtain the fused target motion primitives and parameter mapping rules.

[0096] In one embodiment, each node of the pulse-side suppression competition network corresponds to a visual event packet, and each node is laterally suppressed by a preset suppression coefficient. The competition intensity is output after subtracting the sum of suppression from other nodes from the channel weight of the node and passing it through a nonlinear function to generate the net activation value of each node.

[0097] In one embodiment, before vector fusion, if the angle between any two parameter vectors exceeds a preset angle threshold, fusion is abandoned, and only the parameter vector corresponding to the node with the largest net activation value is retained.

[0098] In this embodiment, the data structure of the visual event package includes a timestamp field, which records the moment when the visual recognition acceleration core completes inference and generates the event package. The timestamp can be generated by a global clock counter to ensure the accuracy of the timing relationship between each event package. The hardware arbitrator reads each visual event package to be fused from the shared memory and extracts its timestamp and event type identifier.

[0099] The preset priority table is stored in a lookup table (LUT) or register inside the hardware arbitrator. This table defines the priority values ​​corresponding to different event types. For example, event types can be divided into "emergency obstacle avoidance", "target capture", "routine monitoring", etc., and assigned priorities of 10, 5 and 1 respectively. The hardware arbitrator matches the event type identifier of each visual event packet with the preset priority table to obtain the corresponding priority value.

[0100] Based on timestamps and priorities, the hardware arbitrator generates a channel weight for each visual event packet, with higher priority and newer event packets receiving larger channel weights.

[0101] In another implementation, priority and timestamp can be mapped to different weight components, and then a weighted sum can be performed. Regardless of the calculation method used, the generated channel weights reflect the relative importance of the event packet in the competition and are used as input to the subsequent competing network.

[0102] The pulse lateral inhibition competitive network is a hardware computing structure that simulates the lateral inhibition mechanism between biological neurons. In this embodiment, the network consists of multiple nodes, each corresponding to a visual event packet. The nodes inhibit each other laterally through a preset inhibition coefficient, that is, the activation of each node will be inhibited by the activation of other nodes.

[0103] In actual hardware implementation, this iterative process can be completed by combinational logic or pipelined arithmetic units without software intervention. Since N is usually small (e.g., 2 to 4), hardware resource overhead is controllable, and iterative convergence can be completed within a fixed clock cycle, ensuring deterministic processing latency. After several iterations or until the network stabilizes, each node outputs a net activation value, which reflects the degree to which the event packet wins in the competition after considering the inhibitory effect of other event packets.

[0104] After obtaining the net activation value of each node, the hardware arbiter compares each net activation value with a preset threshold. The preset threshold can be the value used by the nonlinear function in step S02, or it can be a separately set decision threshold.

[0105] If only one node's net activation value exceeds the preset threshold, it indicates that the event packet corresponding to that node has a significant advantage in the competition, and the activation of other event packets is effectively suppressed or insufficient. In this case, the hardware arbitrator selects the motion primitives and parameter mapping rules of the visual event packet corresponding to the winning node as the final trigger result. Specifically, the hardware arbitrator extracts the target motion primitive identifier and parameter mapping rules from the visual event packet (or the motion primitives and parameters matched from the motion primitive lookup table based on the feature vector of the event packet), and then passes them to the motion primitive generator.

[0106] When the net activation values ​​of multiple nodes all exceed the preset threshold, it indicates that multiple event packets have strong activation, and the competition between them has not led to a single node completely winning. At this time, if the mutual inhibition coefficient between nodes is less than the preset value, it indicates that the inhibition between these event packets is weak. They may represent compatible or complementary control intentions and are suitable for fusion.

[0107] In this case, the hardware arbitrator uses a weighted summation method to fuse the motion primitives and parameter mapping rules corresponding to multiple visual event packets. For motion primitive identifiers, if the motion primitive types corresponding to each event packet are the same, the target motion primitive after fusion is of that type; if the motion primitive types are different, the motion primitive corresponding to the node with the largest net activation value can be selected as the target motion primitive, or a selection can be made according to preset rules (e.g., selecting the motion primitive type with the highest priority). The fused parameter vector is used as the parameter of the target motion primitive.

[0108] Before weighted summation, the hardware arbitrator performs a vector fusion feasibility check. Specifically, if the angle between any two parameter vectors exceeds a preset angle threshold, fusion is abandoned, and only the parameter vector corresponding to the node with the largest net activation value is retained. The reason for this mechanism is that when the differences in the directions of the parameter vectors are too large, it indicates that there is a fundamental conflict between the expected motion directions or control objectives of multiple event packets. A simple weighted average may lead to unreasonable or even dangerous intermediate states. The angle threshold can be set according to the robot's kinematic constraints and task requirements, for example, set to 30 degrees, 45 degrees, or 90 degrees. This check ensures the physical rationality of the fusion result. Finally, the hardware arbitrator outputs the determined target motion primitives and parameter mapping rules (i.e., the fused parameter vectors or the single winning parameter vector), which are then handed over to the motion primitive generator to generate real-time motion control commands.

[0109] In one embodiment, when the visual recognition acceleration core and motion primitive lookup table configured in the programmable logic unit need to be replaced, the original visual recognition acceleration core and motion primitive lookup table and the latest visual recognition acceleration core and motion primitive lookup table are constructed into a dynamic sequence, and the image data collected by the robot's vision module is monitored by the programmable logic unit so as to dynamically switch the visual recognition acceleration core and motion primitive lookup table of the dynamic sequence at a specified time.

[0110] In this embodiment, when the system receives a task switching instruction or determines that the visual recognition acceleration core and motion primitive lookup table need to be replaced according to the preset task flow, the general-purpose processor core or the upper-level task manager will prepare the latest configuration data. The latest configuration data includes the bitstream file of the new visual recognition acceleration core and the initialization data of the new motion primitive lookup table. The original visual recognition acceleration core and motion primitive lookup table are still retained in the corresponding reconfigurable areas and storage units of the programmable logic unit, maintaining their current working state.

[0111] A dynamic sequence is a sequence of configuration states maintained in hardware or software to manage multiple configurations in chronological order. In this embodiment, the dynamic sequence contains at least two elements: the current configuration in use (denoted as configuration A) and the latest configuration to be switched to (denoted as configuration B). The dynamic sequence can be stored in the configuration management register set in the programmable logic unit, the descriptor table in BRAM, or maintained in memory by the general-purpose processor core and read by hardware via DMA.

[0112] Each configuration item in the dynamic sequence contains the following information: the identifier or bitstream address of the visual recognition acceleration core, the storage address and size of the motion primitive lookup table, the time window or triggering condition for the configuration to take effect, the switching priority, etc. The construction process of the dynamic sequence can be completed on a general-purpose processor core, and then the descriptor is written into the configuration management unit in the programmable logic unit and managed by the hardware state machine.

[0113] In a preferred embodiment, the dynamic sequence can be extended to multiple configurations to form a configuration queue. For example, when the robot needs to perform multiple stage tasks in sequence, multiple configurations can be arranged in order at once, and the hardware switches sequentially according to the supervision signal without having to go through the general-purpose processor core each time.

[0114] Information supervision refers to the process by which a programmable logic unit (PLC) analyzes the image data collected by the robot's vision module in real time without interfering with the normal operation of the current vision recognition acceleration core, and detects whether the switching conditions are met. Supervision can be carried out at multiple levels and is flexible.

[0115] One implementation involves a lightweight image supervision module within the programmable logic unit. This module can be a simple feature detector, frame difference analyzer, or hardware-based classifier. The supervision module runs in parallel with the currently operating visual recognition acceleration core, sharing the same image data stream. For example, after preprocessing, the image data is simultaneously fed into the visual recognition acceleration core and the supervision module. The supervision module extracts specific features from the image, such as color histograms, motion regions, and specific markers, and compares them with preset switching trigger conditions.

[0116] Another approach is to directly use the output of the current visual recognition acceleration core as the basis for supervision. For example, when the visual recognition acceleration core detects that the confidence of the target object is consistently below the threshold, or when it detects the appearance of a specific type of object, the feature vector or event type in its output visual event package can be captured by the supervision module as a switching trigger signal. This approach does not require additional hardware resources, but it may rely on the inference results of the current configuration and may not be able to detect the scene that needs to be switched in advance.

[0117] The triggering conditions for the supervision module can be time-related or event-related. Time-related conditions include "switching to configuration B after 500 milliseconds from the effective date of configuration A"; event-related conditions include "a red circular marker was detected in the image" or "the target object in the image left the field of view". The triggering conditions can be written into the hardware register as part of the descriptor when constructing the dynamic sequence.

[0118] The result of information supervision is a Boolean switching enable signal or trigger flag. When the supervision module determines that the switching conditions are met, it will send a trigger request to the dynamic sequence management state machine, which may also include a timestamp or supervision confidence information.

[0119] The specified time can be the moment when the monitoring module detects the trigger condition, or it can be a preset time point. In order to ensure the smoothness of the switching, seamless switching needs to be implemented at the hardware level. This invention adopts partially reconfigurable technology and a double buffering mechanism to achieve dynamic switching.

[0120] Specifically, the programmable logic unit allocates two reconfigurable regions (or one reconfigurable region in conjunction with a shadow configuration) to the visual recognition acceleration core. The current configuration A runs in one reconfigurable region, while the bit stream of configuration B has been pre-loaded into the cache of the configuration controller (such as ICAP), or has been written to another reconfigurable region but is in a reset state. The motion primitive lookup table also adopts double buffering, and the lookup table data of configuration B has been written to the spare BRAM or the spare area of ​​external memory.

[0121] When the monitoring module triggers the switching signal, the dynamic sequence management state machine performs the following operations: To prevent mixed data from being generated during the switching process, the state machine first suspends the event transmission from the visual recognition acceleration core to the visual-motion direct pass-through engine via a gating signal, ensuring that the currently processed event packet is completed or discarded; it loads the bit stream of configuration B into the target reconfigurable region via a partially reconfigurable control interface (such as ICAP), or directly switches to a pre-configured backup region via a multiplexer. If a backup region is used, the switching delay can be shortened to a few clock cycles; the base address of the motion primitive lookup table is switched from the storage area of ​​configuration A to the storage area of ​​configuration B via address mapping switching. If a double-buffered BRAM is used, only the selection signal of the read port needs to be changed; the visual recognition acceleration core of configuration B is reset and released, allowing it to start receiving image data and performing inference; at the same time, the interrupt routing table and shared memory mapping are updated to ensure that new visual event packets are correctly transmitted to the visual-motion direct pass-through engine; the transmission path from the visual recognition acceleration core to the visual-motion direct pass-through engine is enabled, and the system enters the configuration B working phase.

[0122] The entire switching process is controlled by a hardware state machine, without the need for intervention from a general-purpose processor core. The switching latency can be controlled at the microsecond level or even lower, avoiding control gaps. At the same time, due to the use of double buffering or preloading, the switching will not lose image frames or generate erroneous visual events.

[0123] When there are only configurations A and B in the dynamic sequence, after the switch is completed, configuration A becomes a historical configuration and can be released or used as a backup area for the next switch. If the dynamic sequence contains multiple configurations, the state machine switches sequentially. After the switch is completed, the general-purpose processor core can be notified of the switch completion event through an interrupt and can prepare subsequent configurations, update the dynamic sequence descriptor, and realize a continuous task flow.

[0124] In one embodiment, the method further includes: providing a graphical development environment to receive a data flow diagram constructed by a user through drag-and-drop, and parsing the data flow diagram to generate configuration data configured in the programmable logic unit, wherein the configuration data is used to configure the visual recognition acceleration core and motion primitive lookup table of the programmable logic unit.

[0125] In one embodiment, after generating motion control commands, the motion control commands are verified in real time according to preset safety constraint parameters, the preset safety constraint parameters including at least one of maximum speed, maximum acceleration, and joint angle limit; When the motion control command exceeds the preset safety constraint parameters, the motion control command is limited or replaced with a preset safe motion primitive, and an interrupt notification is sent to the general-purpose processor core through the shared memory and interrupt routing unit.

[0126] In this embodiment, the graphical development environment can run on a general-purpose computer or workstation, providing a graphical user interface (GUI). The data flow graph consists of nodes and connections. Nodes represent vision processing modules or control primitive modules, and connections define the data flow direction. Its core semantics is the direct mapping relationship from vision output fields to motion primitive parameters.

[0127] Specifically, the output port of the vision processing module corresponds to the structured fields in the vision event package, such as "target position X", "target position Y", "target size", "confidence level", and "category ID". The input port of the control primitive module corresponds to the parameter slots in the motion primitive lookup table, such as "target pose X", "target pose Y", "approach velocity", and "grasping force threshold". When the user connects from the vision output port to the control primitive input port, the value of that vision field is defined as the source of the corresponding motion primitive parameter.

[0128] After the user completes the data flow graph construction, a configuration generation operation is triggered. The development environment parses the data flow graph and performs the following steps: checks for errors such as loops, dangling ports, and type mismatches in the graph, and arranges each module in topological order; for each vision processing module, selects the corresponding bitstream or HLS-generated configuration description from the pre-compiled acceleration core library, and generates the initialization configuration of the acceleration core (such as model weight address, input feature map size, quantization parameters, etc.) according to the module parameters. If multiple vision processing modules are cascaded, a streaming interface configuration and DMA transfer descriptor between acceleration cores are generated; for each control primitive module, a motion primitive identifier is assigned, and a lookup table entry is generated according to the mapping function of the connection and the parameter source. Each entry contains a feature vector template (or index), a motion primitive identifier, and a parameter mapping rule. The parameter mapping rules are derived from the mapping functions on the connection lines; based on the connection relationships, the input port binding information and matching logic configuration of the hardware associative matcher are generated, as well as the interrupt routing table configuration, to ensure that the corresponding fields in the visual event packet can be correctly routed to the input of the matcher; the task management state machine code on the general-purpose processor core is generated, including task startup, monitoring, exception handling, etc., as well as optional dynamic sequence descriptors.

[0129] All the above configuration data, bitstreams, and software code are packaged into a unified firmware image, which is then burned into the target hardware's storage medium by the deployment tool. Through this process, developers do not need to write underlying hardware configuration code; they can complete the configuration of the vision-motion passthrough path simply by dragging and dropping graphically, which significantly reduces development difficulty and the probability of errors.

[0130] This invention provides a mechanism for real-time security verification after generating motion control commands. This mechanism is implemented in hardware within a programmable logic unit and does not compromise the low-latency characteristics of the vision-motion direct path.

[0131] During task configuration (e.g., in step S10), the general-purpose processor core writes preset safety constraint parameters into the safety verification module within the programmable logic unit. These preset safety constraint parameters include, but are not limited to: maximum speed: the maximum angular velocity of each joint or the maximum linear velocity and angular velocity of the end effector; maximum acceleration: the maximum angular acceleration of each joint or the maximum linear acceleration and angular acceleration of the end effector; joint angle limits: the soft and hard limit values ​​of each joint; end effector workspace limits: the boundary of the area that the end effector is allowed to reach in Cartesian space; force / torque limits: the maximum allowable force or torque of the joint or end effector. The safety constraint parameters are stored in registers or BRAM within the safety verification module and can be dynamically updated by the general-purpose processor core during operation.

[0132] The safety verification module is located between the motion primitive generator and the real-time bus interface within the vision-motion passthrough engine. It performs cycle-by-cycle verification of the motion control commands output by the motion primitive generator. Its internal structure includes: a command input buffer: receiving the motion control commands (such as joint position, velocity, and acceleration commands) of the current control cycle output by the motion primitive generator; a constraint comparator array: composed of multiple parallel comparators, comparing each component in the command with the corresponding safety constraint parameters; a limiting correction unit: limiting the components that exceed the constraints based on the comparison results; a safe motion primitive storage unit: storing at least one preset safe motion primitive and its parameters (such as emergency stop, deceleration to zero, etc.); and an interrupt generation and routing interface: used to send interrupt notifications to the general-purpose processor core.

[0133] The hardware verification process is as follows: Instruction Acquisition: The safety verification module acquires the motion control instructions generated by the motion primitive generator for the current control cycle. These instructions contain the desired position, velocity, and acceleration values ​​for each joint or end effector. Constraint Comparison: The constraint comparator array compares each component in the instruction with its corresponding safety constraint parameters in parallel to determine if it exceeds the upper or lower limit. Normal Pass Processing: If all components are within the safety constraint range, the instruction passes directly through the safety verification module without modification and is output to the actuator via the real-time bus interface. Exceedance Processing: If at least one component exceeds the safety constraint range...

[0134] The safety verification module performs the following operations: Limit correction: truncates the component exceeding the constraint to the corresponding upper or lower limit value, keeps other components unchanged, and generates a corrected instruction. Limit correction is suitable for situations where the degree of exceeding the limit is small and reasonable motion can still be maintained after correction. If the degree of exceeding the limit is large or exceeds the limit in multiple consecutive control cycles, the limit correction can no longer guarantee safety. In this case, the safety verification module reads the preset safety motion primitive and its parameters from the safety motion primitive storage unit and generates a replacement instruction.

[0135] Regardless of whether amplitude limiting correction is used or a safe motion primitive is replaced, the safety verification module sends an interrupt notification to the general-purpose processor core through shared memory and the interrupt routing unit. The interrupt notification includes information such as the type of over-limit, the value of over-limit, the timestamp, and the corrective measures taken, so that the general-purpose processor core can perform subsequent diagnosis, task degradation, or human-machine interaction. The motion control instructions after verification, correction, or replacement are output to the actuator through the real-time bus interface.

[0136] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0137] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0138] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A robot control system, characterized in that, It includes a heterogeneous hardware layer, a real-time operation layer, and a collaborative development tool layer; The heterogeneous fusion hardware layer is built on a heterogeneous system-on-a-chip, which includes a general-purpose processor core and a programmable logic unit. The programmable logic unit is equipped with a vision-motion pass-through engine, which is configured to directly receive the structured perception results output by the vision recognition accelerator to calculate the end effector path point and send control commands to the motor driver through the industrial real-time bus interface. The real-time execution layer includes an event-driven vision-control scheduler, which is configured to generate a visual event carrying structured data after the vision recognition accelerator completes one frame of inference, so as to directly index the pre-stored motion primitive lookup table and trigger the execution of the corresponding motion primitive. The collaborative development tool layer includes a vision-control flow graph editor and a heterogeneous code automatic compilation and deployment tool. The vision-control flow graph editor is configured to construct a data flow graph consisting of a vision processing module and a control primitive module by dragging and dropping. The data flow graph is used to define the mapping relationship from vision output fields to motion primitive parameters. The heterogeneous code automatic compilation and deployment tool is configured to generate a deployable unified firmware image based on the data flow graph. The programmable logic unit is also configured with a reconfigurable visual recognition accelerator group, which includes multiple dynamically loadable hardware acceleration cores and is configured to switch the hardware acceleration cores without interrupting the main control operation through partial reconfiguration technology to adapt to different visual recognition tasks. The visual recognition task includes object detection, 6D pose estimation, and scene segmentation.

2. The robot control system according to claim 1, characterized in that, The programmable logic unit is also configured with a shared memory and interrupt routing unit, which is configured to realize data exchange between the visual recognition accelerator within the programmable logic unit, the general-purpose processor core, and external sensor data streams.

3. The robot control system according to claim 1, characterized in that, The general-purpose processor core is configured to run a standard operating system and perform upper-layer task planning, non-real-time human-computer interaction, and host-side proxy functions for development tools.

4. The robot control system according to claim 1, characterized in that, The real-time runtime layer also includes a visual recognition acceleration library and a deterministic motion control middleware; the visual recognition acceleration library is configured to provide a unified application programming interface for the hardware accelerators in the programmable logic unit to support loading, running and destroying different visual model acceleration cores; The deterministic motion control middleware is configured to run on a coprocessor or a microprocessor within the programmable logic unit to manage the motion primitive queue and to perform security monitoring of the trigger signals of the vision-motion passthrough engine.

5. The robot control system according to claim 1, characterized in that, The heterogeneous code automatic compilation and deployment tool is configured as follows: The connection relationships of the vision modules in the data flow graph are compiled into a streaming interface configuration and direct memory access transfer descriptor between accelerators within the programmable logic unit; Based on the mapping relationship from visual results to control primitives, the hardware lookup table initialization data of the vision-motion passthrough engine and the task management state machine code on the general-purpose processor core are generated.

6. The robot control system according to claim 1, characterized in that, The collaborative development tool layer also includes a hardware-in-the-loop simulation and performance profiler, which is configured as follows: During the development phase, the timing of the vision-motion direct path is simulated, and sensor data is injected for algorithm verification. During the operation phase, the timestamps from the triggering of visual events to the output of motor commands are captured in real time, and the time consumed in each stage is displayed in a waterfall chart.

7. The robot control system according to claim 1, characterized in that, The industrial real-time bus interface includes an EtherCAT interface.

8. A robot control method, characterized in that, This is applied to heterogeneous on-chip systems, which include a general-purpose processor core and a programmable logic unit. The programmable logic unit is configured with a reconfigurable visual recognition accelerator group, a vision-motion passthrough engine, and a shared memory and interrupt routing unit. The reconfigurable visual recognition accelerator group includes multiple visual recognition acceleration cores, including: The target information of the current task is obtained, and a visual recognition acceleration core and a motion primitive lookup table corresponding to the current task are configured in the programmable logic unit according to the target information; wherein, the motion primitive lookup table stores multiple motion primitives and their corresponding parameter mapping rules; Image data is acquired through the robot's vision module, and the image data is inferred through the visual recognition to generate a visual event package containing structured perception results. The visual event packets are directly transmitted to the visual-motion passthrough engine via the shared memory and interrupt routing unit. The hardware association matcher within the vision-motion passthrough engine matches the target motion primitive and its corresponding parameter mapping rule from the motion primitive lookup table based on the feature vector in the vision event package. The motion primitive generator within the vision-motion direct pass engine generates motion control commands in real time based on the target motion primitive and the parameter mapping rules, and outputs them to the actuator via a real-time bus.

Images