Hand-eye cooperative robot control system and method based on dynamic operator arrangement

Abstract

The invention discloses a hand-eye cooperative robot control system and method based on dynamic operator arrangement, an upper computer planning layer operates a master control computer to periodically trigger task scheduling, a joint controller of a lower computer execution layer receives an instruction through a redundant bus to perform servo control, hand-eye camera data triggers a visual assembly line through an interrupt event, and a visual assembly line is controlled through a control interface. According to the method, pressure is calculated through visual processing, motion planning and joint control in a load sharing mode, a double-buffering mechanism is adopted to enable current frame visual processing and previous frame motion control to be executed in an overlapping mode, real-time scheduling and parallel computing of tasks are completed, hot data are cached in a memory database, cold data are archived to an HDFS and migrated through an LRU strategy, and the real-time scheduling and parallel computing of the tasks are completed. Data type conversion is automatically derived and performed based on a feature type registry, and conditional branch execution is triggered according to real-time sensor data. According to the control system, cross-hardware plug and play, algorithm flow configurability and data flow real-time sharing are achieved, and the requirement for improving the efficiency of complex operation tasks is met.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of image recognition, and in particular to a hand-eye collaborative robot control system and method based on dynamic operator arrangement. BACKGROUND

[0002] In the existing field of hand-eye collaborative robot control, the traditional hand-eye system (such as ROSMoveIt) has many problems, the algorithm is fixed, the code needs to be rewritten for each task, and it cannot be flexibly adapted to new tasks; the hardware compatibility is poor, different brands of cameras (such as OBI Zhongguang and Intel RealSense) and mechanical arms (such as UR and KUKA) need to be customized to develop a driver interface; the data processing is inefficient, the visual data and pose information are stored in a scattered manner, and repeated calculation causes operation delay (actual measurement > 200 ms).

[0003] In view of the need to realize cross-hardware plug and play, algorithm process configurability and data flow real-time sharing to improve the efficiency of complex operation tasks, a hand-eye collaborative robot control method based on dynamic operator arrangement and unified feature database is proposed by using technologies such as dynamic operator arrangement and unified feature database. SUMMARY

[0004] The purpose of the application is to provide a hand-eye collaborative robot control system and method based on dynamic operator arrangement to solve the problems in the background art.

[0005] To achieve the above purpose, the application provides the following technical scheme: a hand-eye collaborative robot control method based on dynamic operator arrangement, the control method comprising the following steps: The bottom drivers of different brands of devices are packaged into a unified interface, the corresponding plug-in is dynamically loaded through the device ID at runtime, the visual processing, pose calculation and force control adjustment are packaged into standardized operators, the topological relationship between the standardized operators is defined through a configuration file, a directed acyclic graph execution pipeline is constructed, the operator dynamic library is dynamically loaded by means of dlopen(), the preset operator library is combined, and the task flow dynamic arrangement is completed; The host computer planning layer triggers task scheduling periodically, the joint controller of the lower computer execution layer receives instructions through the redundant CAN bus for servo control, the hand-eye camera data triggers the visual pipeline through an interrupt event, the visual processing, motion planning and joint control are distributed to reduce the calculation pressure, the double buffering mechanism is adopted to overlap the execution of the current frame visual processing and the last frame motion control, and the real-time scheduling and parallel calculation of the task are completed; All intermediate data are encapsulated as octFeature objects, stored in feature database, hot data cached in Memcached in-memory database, cold data archived to HDFS and migrated through LRU strategy, data type conversion automatically derived and executed based on feature type registry, conditional branch execution triggered based on real-time sensor data.

[0006] Preferably, the host computer planning layer runs periodic task scheduling triggered by the host computer, the joint controller of the lower machine execution layer receives instructions through the redundant CAN bus for servo control, the hand-eye camera data triggers the vision pipeline through interrupt events, the vision processing, motion planning and joint control are distributed to relieve computing pressure, double buffering mechanism is used to overlap the current frame vision processing and the last frame motion control, real-time scheduling and parallel computing are completed, including the following steps: The system architecture is divided into a host computer planning layer and a lower machine execution layer, which work cooperatively through a standardized communication interface, the host computer planning layer is deployed on an x86 architecture host computer, and the lower machine execution layer is composed of a joint controller cluster based on ARM Cortex-R series processors; The host computer planning layer adopts a periodic scheduling strategy, periodically triggers task scheduling cycles, and controls tasks to be completed within time constraints based on a priority-driven real-time scheduling algorithm; The lower machine execution layer adopts a distributed real-time control architecture, each joint controller is based on an ARM Cortex-R series processor, and receives control instructions from the host computer through a redundant CAN bus network; The hand-eye camera data starts the vision pipeline processing through an interrupt event triggering mechanism, a hardware interrupt is triggered when the camera sensor detects that a new frame of data has arrived, and the real-time scheduling program processes the vision data first; A heterogeneous computing resource optimization allocation strategy is implemented, computing resources are allocated according to task characteristics, and overall control tasks are decomposed into multiple independent computing nodes.

[0007] Preferably, the host computer planning layer adopts a periodic scheduling strategy, periodically triggers task scheduling cycles, and controls tasks to be completed within time constraints based on a priority-driven real-time scheduling algorithm, including executing a trajectory planning function in each scheduling period, using a spline interpolation-based trajectory generation algorithm processing logic, calculating intermediate trajectory points that are smooth and meet the dynamics characteristics according to the target pose sequence and motion constraint conditions, then performing inverse kinematics solving, and through the processing logic of a numerical iterative optimization algorithm, the required configuration parameters of each joint are back calculated from the target end pose.

[0008] Preferably, the lower machine execution layer network topology adopts a dual-channel redundant design, and through memory access and hardware timer interrupt mechanisms, the real-time performance of control instructions is ensured, and a PDO real-time transmission protocol is supported for transmission of key joint angle instructions and state feedback information, and the PDO real-time transmission protocol adopts a fixed-cycle cyclic communication mode.

[0009] Preferably, a heterogeneous computing resource optimization allocation strategy is implemented, and computing resources are allocated according to task characteristics, including the following steps: For a computing-intensive task, a GPU acceleration mechanism is adopted, and parallel computing tasks are allocated to multiple computing cores of a graphics processing unit through processing logic of CUDA kernel parallel computing. For a logic-intensive task, general computing processing of a CPU is used.

[0010] Preferably, the underlying drivers of devices of different brands are packaged into a unified interface, corresponding plug-ins are dynamically loaded through device IDs at runtime, visual processing, pose calculation, and force control adjustment are packaged into standardized operators, the topological relationship between standardized operators is defined through a configuration file, a directed acyclic graph execution pipeline is constructed, operator dynamic libraries are dynamically loaded with the help of dlopen(), a preset operator library is combined, and dynamic scheduling of a task flow is completed, including the following steps: A hardware abstraction layer is constructed, octDevice and octRobot modules are used to realize standardized interface packaging of multiple brands of devices, underlying drivers of different devices are abstracted, and a unified device operation interface specification is formed; Standardized image acquisition interfaces and point cloud generation interfaces are defined for visual devices, and trajectory execution interfaces and joint state feedback interfaces are defined for robotic arm devices, and corresponding device driver plug-ins are developed for each device based on these standard interfaces; Plug-ins are dynamically loaded through device IDs at runtime to realize hybrid deployment capability of cameras and robotic arms; Each operator complies with the specification of the OperatorBase abstract base class, realizes the execute() method, and defines input / output feature slot mechanisms; A JSON configuration file is used to define the topological relationship between operators to construct an execution pipeline in the form of a directed acyclic graph; At runtime, a dlopen() dynamic loading mechanism is used to load corresponding operator dynamic link libraries in real time according to the operator types specified in the configuration file.

[0011] Preferably, the operators include a point cloud segmentation operator, a 3D registration operator, and a force control adjustment operator.

[0012] Preferably, the point cloud segmentation operator processes the original point cloud data, extracts the point cloud region of the target object through the processing logic based on normal vector analysis and Euclidean distance clustering, and outputs the segmented point cloud data. The 3D registration operator receives the segmented point cloud data and the target model point cloud, adopts the processing logic of the iterative closest point algorithm, calculates the optimal pose transformation matrix of the current point cloud and the target model, and outputs the control instruction. The force control adjustment operator processes the force sensor data, dynamically adjusts the clamping force or contact force of the mechanical arm based on the processing logic of the force feedback control algorithm, and outputs the control instruction.

[0013] Preferably, each of the operators receives upstream data through a defined input slot, and after internal processing logic, transmits the processing result to the downstream operator through an output slot, forming a data flow dependency relationship.

[0014] The application also provides a hand-eye collaborative robot control system based on dynamic operator arrangement, which comprises a dynamic arrangement module, a scheduling and calculation module, and an execution module. The dynamic arrangement module encapsulates the bottom layer drivers of different brands of equipment into a unified interface, dynamically loads the corresponding plug-in through the device ID at runtime, encapsulates the visual processing, pose calculation, and force control adjustment into standardized operators, defines the topological relationship between the standardized operators through a configuration file, constructs a directed acyclic graph execution pipeline, dynamically loads the operator dynamic library by means of dlopen(), and combines the pre-set operator library. The scheduling and calculation module: the host computer planning layer runs the main control computer cycle to trigger task scheduling, the joint controller of the lower computer execution layer receives instructions through the redundant CAN bus for servo control, the hand-eye camera data triggers the visual pipeline through an interrupt event, the visual processing, motion planning, and joint control are distributed for calculation to reduce the pressure, and a double buffering mechanism is adopted to overlap the execution of the current frame visual processing and the last frame motion control. The execution module: all intermediate data are encapsulated into octFeature objects and stored in a feature database, hot data are cached in a Memcached in-memory database, cold data are archived to HDFS and migrated through an LRU strategy, data type conversion is automatically derived and executed based on a feature type register, and conditional branch execution is triggered according to real-time sensor data.

[0015] In the above technical solution, the application provides the following technical effects and advantages: The application solves the compatibility problem of traditional hand-eye systems (such as ROSMoveIt) by packaging the underlying drivers of different brand devices into a unified interface and dynamically loading plug-ins, realizing plug-and-play mixed deployment of 7 brands of cameras and 5 models of mechanical arms; At the same time, the core functions such as visual processing and pose calculation are atomized into standardized operators, and the task pipeline can be constructed through JSON configuration instead of code rewriting (such as power cabinet knob operation, which is shortened from traditional 3 person-days development to 2 hours), so that the system has zero coding task adaptation capability.

[0016] In terms of real-time scheduling and parallel computing, the hierarchical control model (upper computer periodic scheduling + lower computer μs level servo) cooperates with the visual pipeline driven by the interrupt event (actual measurement delay < 50ms), combined with the double buffering mechanism (throughput is improved by 2.1 times) and the calculation load dispersion strategy (GPU acceleration point cloud segmentation / CPU processing logic task), the operation delay of traditional system > 200ms is reduced to < 50ms, the resource utilization is improved by 40%, and the problem of inefficient data processing is effectively solved.

[0017] The application eliminates the repeated calculation of visual data and pose information in the traditional scheme through global data sharing of unified feature database (hot data < 1ms delay / cold data LRU migration) and chain type conversion (zero copy optimization); Based on the automatic derivation mechanism of feature type registration table (such as point cloud→pose ICP registration) and the event-driven conditional branch execution (such as force sensor triggering resistance adjustment), the system supports artificial intervention rate < 5% in complex tasks such as 245 busbar cabinet closing and opening, and can reuse JSON configuration to reduce 90% development cost in scenarios such as automobile production line change, and finally realizes adaptive control across fields (industrial / medical). BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0019] Figure 1 The mind map of the identification method of the present application. DETAILED DESCRIPTION

[0020] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0021] Embodiment: The embodiment provides a hand-eye collaborative robot control method based on dynamic operator orchestration. Please refer to FIG. 1. Figure 1 The control method comprises the following steps: First, dynamic orchestration of the task flow is performed. Through the plug-in framework of the hardware abstraction layer (octDevice / octRobot), the bottom-layer drivers of different brands of devices (such as OBI medium light cameras and UR robot arms) are packaged into a unified interface. The corresponding plug-in can be dynamically loaded through the device ID at runtime, thereby supporting the mixed deployment of 7 brands of cameras and 5 models of robot arms and avoiding the trouble of customizing and developing drivers for a single device. Meanwhile, core functions such as visual processing (such as point cloud segmentation), pose calculation (such as 3D registration) and force control adjustment (such as force control self-adaptation) are packaged into standardized operators. The standardized operators inherit the abstract class OperatorBase, and the execute() method and input / output feature slots Slot are implemented compulsorily. The topological relationship between the operators is defined through a JSON configuration file, a directed acyclic graph (DAG) execution pipeline is constructed, and the operator dynamic library is dynamically loaded by means of dlopen(). Without rewriting the code, the pre-existing operator library or new operators can be flexibly combined to complete the dynamic orchestration of the task flow.

[0022] After the dynamic orchestration of the task flow is completed, a real-time task scheduling and parallel computing phase based on a hierarchical distributed architecture is entered. The real-time task scheduling and parallel computing phase focuses on solving the problems of low data processing efficiency and high operation delay in traditional systems, and ensures efficient execution of tasks. The upper machine planning layer runs on a host computer of an x86 architecture, triggers task scheduling with a period of no more than 10 ms, and is mainly responsible for trajectory planning and inverse kinematics solving. The joint controller of the lower machine execution layer adopts an ARM Cortex-R series, receives instructions through a redundant CAN bus, realizes μs-level servo control, and also supports PDO real-time transmission of joint angle instructions. Hand-eye camera data triggers a vision pipeline through an interrupt event, and the priority of the hand-eye camera data is higher than that of the periodic task. In terms of utilization of computing resources, computationally intensive tasks such as point cloud segmentation are allocated to GPU acceleration (CUDA kernel), and logical tasks such as path planning are processed by the CPU. The vision processing (hand-eye controller), motion planning (host computer), and joint control (six independent controllers) are distributed to share the computing pressure, avoiding a single point bottleneck. A double buffering mechanism (Double Buffering) is adopted to overlap the execution of the current frame vision processing and the last frame motion control, so that the throughput is improved by 2.1 times, thereby realizing real-time scheduling of tasks and optimized utilization of resources, and completing real-time scheduling and parallel computing of tasks.

[0023] After the real-time scheduling and parallel computing of the task are completed, event-driven dynamic response and global data sharing based on a unified feature database become a key step, solving the problems of scattered data storage, repeated calculation and rigid flow in traditional systems, and further improving the effect and adaptability of task execution. All intermediate data such as point clouds, pose matrices and joint angle sequences are encapsulated as octFeature objects and stored in a unified feature database (octFeatureDatabase). Hot data such as real-time poses are cached in a Memcached in-memory database with a read-write delay of <1 ms, and cold data is archived to HDFS and migrated through an LRU strategy. Through a chain type conversion engine, data type conversion (such as ICP registration of point clouds to poses) is automatically inferred and executed based on a feature type registry. The conversion process uses memory mapping to realize a zero-copy shared buffer. At the same time, event-driven dynamic response is supported, and conditional branch execution is triggered according to real-time sensor data (such as force sensor detection of excessive knob resistance), for example, a force control adaptive operator adjusts the gripping force of the robot arm in real time. Moreover, the event triggering mechanism (such as the depth camera data ready to trigger the point cloud segmentation operator) ensures that the data flow flows according to actual needs, breaking through the limitations of traditional linear flow.

[0024] The embodiment also provides a hand-eye collaborative robot control system based on dynamic operator orchestration, including a dynamic orchestration module, a scheduling and computing module, and an execution module. Dynamic orchestration module: encapsulate the underlying drivers of different brands of equipment into a unified interface, dynamically load the corresponding plug-ins through the device ID at runtime, encapsulate visual processing, pose calculation, force control adjustment into standardized operators, define the topological relationship between standardized operators through the configuration file, construct a directed acyclic graph (DAG) execution pipeline, dynamically load operator dynamic libraries with dlopen(), combine the preset operator library, complete the dynamic orchestration of the task flow and send it to the scheduling and computing module; Scheduling and computing module: the host computer planning layer runs the master computer periodically to trigger task scheduling, the joint controller of the lower computer execution layer receives instructions through the redundant CAN bus for servo control, the hand-eye camera data triggers the visual pipeline through the interrupt event, the visual processing, motion planning, and joint control are distributed for computing pressure, the double buffering mechanism is used to overlap the current frame visual processing and the last frame motion control for execution, the real-time scheduling and parallel computing of the task are completed and sent to the execution module; Execution module: all intermediate data are encapsulated as octFeature objects and stored in the feature database, hot data are cached in the Memcached in-memory database, cold data are archived to HDFS and migrated through the LRU strategy, data type conversion is automatically derived and executed based on the feature type registry, conditional branch execution is triggered according to real-time sensor data.

[0025] First, dynamic orchestration of the task flow is required, through the plug-in framework of the hardware abstraction layer (octDevice / octRobot), the underlying drivers of different brands of equipment (such as OBI light camera, UR robot arm, etc.) are encapsulated into a unified interface, and the corresponding plug-ins can be dynamically loaded through the device ID at runtime, thereby supporting the mixed deployment of 7 brands of cameras and 5 models of robot arms, avoiding the trouble of customizing the development of drivers for a single device. At the same time, core functions such as visual processing (such as point cloud segmentation), pose calculation (such as 3D registration), force control adjustment (such as force control adaptation), etc. are encapsulated into standardized operators, which inherit the OperatorBase abstract class and are forced to implement the execute() method and input / output feature slots Slot. The topological relationship between operators is defined through a JSON configuration file to construct a directed acyclic graph (DAG) execution pipeline. With the help of dlopen() to dynamically load operator dynamic libraries, it is not necessary to rewrite the code to flexibly combine the preset operator library or add new operators to complete the dynamic orchestration of the task flow.

[0026] A Hardware Abstraction Layer (HAL) is constructed to realize the standardized interface encapsulation of multi-brand devices through octDevice and octRobot modules. The design adopts a plug-in architecture mode to abstract the bottom-layer drivers of devices from different manufacturers (such as Orbbec RGB-D camera, Intel RealSense, UR collaborative robot, KUKA industrial robot, etc.), and forms a unified device operation interface specification.

[0027] Specifically, the application defines a standardized image acquisition interface (such as octCamera::grabImage()) and a point cloud generation interface (such as octCamera::getPointCloud()) for visual devices, and defines a trajectory execution interface (such as octRobot::executeTrajectory()) and a joint state feedback interface (such as octRobot::getJointState()) for robot devices. Device manufacturers need to develop corresponding device driver plug-ins based on these standard interfaces, for example, Orbbec camera corresponds to OrbbecPlugin, and UR robot corresponds to liboctRobot_UR.so. These plug-ins are dynamically loaded (using the dlopen() dynamic library loading mechanism) through the device ID (such as camera_orbbec_001) at runtime, realizing the mixed deployment capability of 7 mainstream brand cameras and 5 models of robots, and effectively avoiding the cumbersome process of developing a separate driver interface for each device in the traditional scheme.

[0028] The core functional modules such as visual processing, pose calculation and force control adjustment are atomized and encapsulated to form standardized functional operators. Each operator strictly follows the specification of the abstract base class OperatorBase and must implement the execute() core method and define a clear input / output feature slot (Slot) mechanism.

[0029] Typical operator designs include: The point cloud segmentation operator (PointCloudSegmentation) processes the original point cloud data (octFeature), extracts the point cloud region of the target object (such as the operation knob) through the processing logic based on normal vector analysis and Euclidean distance clustering, and outputs the segmented point cloud data (octFeature).

[0030] The 3D registration operator (ICPRegistration) receives the segmented point cloud data (octFeature) and the target model point cloud (such as the knob CAD model), adopts the processing logic of the Iterative Closest Point (ICP) algorithm, and calculates the optimal pose transformation matrix (octFeature) of the current point cloud and the target model.

[0031] ForceControlAdapter processes force sensor data (octFeature), based on the processing logic of force feedback control algorithm, dynamically adjusts the gripping force or contact force of the robot arm, and outputs control instructions (octFeature).

[0032] 1) Processing logic and application examples of point cloud segmentation operator: When processing the original point cloud data, the point cloud segmentation operator adopts the processing logic based on normal vector analysis and Euclidean distance clustering. In the actual industrial scene, such as the grabbing operation of automobile parts in the automobile manufacturing workshop, the original point cloud data collected by the 3D camera equipped on the robot contains a large number of points of the entire working scene, which come from the automobile parts themselves and other objects such as surrounding desktops and clamps. The point cloud segmentation operator first calculates the normal vector of each point in the original point cloud, which reflects the orientation information of the surface where the point is located. By analyzing the direction and change of the normal vector, the points with different surface characteristics can be preliminarily distinguished. For example, the surface normal vector of the automobile parts usually has certain regularity, while the normal vector of the surrounding miscellaneous environment objects may be more chaotic. Then, the operator uses the method of Euclidean distance clustering. It will set a suitable distance threshold, and the points with similar normal vectors and spatial distance less than the threshold will be clustered together to form different clusters. In this example of grabbing automobile parts, the points belonging to the surface of automobile parts will be clustered into the same cluster due to their similar normal vectors and relatively concentrated spatial positions, while the points of other irrelevant objects in the surrounding environment will form independent clusters. In this way, the point cloud segmentation operator can extract the point cloud area of the target object (automobile parts) from the complex original point cloud data, and output the segmented point cloud data, providing accurate target point cloud information for subsequent 3D registration and other operations.

[0033] 2) Processing logic and application examples of 3D registration operator: 3D registration operator receives segmented point cloud data (e.g. point cloud of the car part mentioned above) and target model point cloud (i.e. standard CAD model point cloud of the car part obtained beforehand), and uses the processing logic of iterative closest point algorithm to calculate the optimal pose transformation matrix of the current point cloud and the target model. In the electronic product manufacturing factory, when the tiny electronic chip needs to be accurately assembled, the robot obtains the actual point cloud data of the electronic chip through the vision system, and at the same time, there is the standard model point cloud of the electronic chip. The core idea of the iterative closest point algorithm is to constantly find the closest point pairs in the current point cloud and the target model point cloud, and gradually adjust the position and pose of the point cloud according to the information of these closest point pairs, so that the two point clouds can be as coincident as possible. At the beginning of the algorithm, a pose transformation matrix is randomly initialized to transform the current point cloud. Then, for each point in the current point cloud, find the closest point in the target model point cloud in the transformed current point cloud and the target model point cloud, forming a series of closest point pairs. Then, according to the information of these closest point pairs, an error function is calculated, which reflects the degree of mismatch between the two point clouds. By constantly iterating and adjusting the pose transformation matrix, the error function gradually decreases until a preset convergence condition is reached or the maximum number of iterations is reached. In this example of electronic chip assembly, after multiple iterations, the 3D registration operator finally calculates the optimal pose transformation matrix of the current electronic chip point cloud and the target model point cloud, which describes how to translate and rotate the actual electronic chip point cloud to make it completely coincide with the standard model point cloud. The robot can accurately grasp and place the electronic chip according to this optimal pose transformation matrix, achieving accurate assembly.

[0034] 3) Processing logic and application examples of force control adjustment operator The force control adjustment operator processes force sensor data based on the processing logic of the force feedback control algorithm to dynamically adjust the gripping force or contact force of the robotic arm and outputs control instructions. In the food processing industry, robots need to grasp food of different shapes and textures, such as cakes and breads. When the robotic arm grips the food, the force sensor detects the contact force between the robotic arm and the food in real time. The force feedback control algorithm calculates and judges based on the preset force threshold and the current force sensor data. If it is detected that the current gripping force is too small, the food may fall off, and the force control adjustment operator will calculate a control instruction to increase the gripping force according to the processing logic of the force feedback control algorithm, and by adjusting the driving force of the joints of the robotic arm, the gripping force of the robotic arm is slightly increased to ensure that the food can be stably grasped. Conversely, if the gripping force is too large, it may cause the food to be squeezed and deformed, and the force control adjustment operator will calculate a control instruction to reduce the gripping force to appropriately relax the grip of the robotic arm and protect the integrity of the food. For example, when the robot grasps a soft cake, the force sensor detects that the initial gripping force is normal, but during movement, the force sensor data changes due to slight shaking of the cake, etc., indicating that the gripping force is increasing. The force control adjustment operator adjusts in time according to the force feedback control algorithm to dynamically reduce the gripping force, ensuring that the shape and quality of the cake are not affected during grasping and movement, and achieving precise and safe grasping of different foods.

[0035] Each operator receives upstream data through a well-defined input slot (such as input_point_cloud) and passes the processing result to downstream operators through an output slot (such as output_segmented_cloud) after internal processing logic, forming a data flow dependency relationship.

[0036] A JSON configuration file is used to define the topology relationship between operators to build an execution pipeline in the form of a Directed Acyclic Graph (DAG). This configuration method allows users to define operator execution order and data dependency relationships through declarative language without writing program code.

[0037] The configuration file explicitly specifies the type identifier of each operator (such as PointCloudSegmentation), the mapping of input feature slots (such as mapping the original point cloud collected by the camera to the input_point_cloud slot of the operator), and the dependency relationship between operators (such as the ICP registration operator must wait for the point cloud segmentation operator to complete and output the segmentation result). The system parsing engine constructs a complete DAG execution plan based on these configuration information and determines the optimal execution order of the operators through a topological sorting algorithm.

[0038] At runtime, the dlopen() dynamic loading mechanism is used to load the corresponding operator dynamic link library (such as libPointCloudSegmentation.so) according to the operator type specified in the configuration file. This design allows users to flexibly combine the function modules in the pre-installed operator library or integrate newly developed operators by simply modifying the JSON configuration file, thereby achieving rapid reconstruction and customized development of the task flow.

[0039] The dynamic operator orchestration technology implemented in this stage has significant engineering application value. Through standardized operator interfaces and flexible configuration mechanisms, different application scenarios and task requirements can be quickly adapted without modifying the core code. For example, for the power cabinet knob operation task, only the standard process of point cloud acquisition → point cloud segmentation → 3D registration → force control adjustment → mechanical arm execution needs to be defined in the JSON configuration, and the system configuration and debugging can be completed within 2 hours (the traditional customized development scheme usually requires more than 3 person-days of development effort).

[0040] In addition, this technical architecture also supports the continuous expansion and upgrade of the operator library. When new visual algorithms (such as improved point cloud segmentation methods) or control strategies (such as adaptive force control algorithms) need to be introduced, only the new operator module needs to be developed according to the OperatorBase specification and integrated into the operator library. By updating the JSON configuration file, the new function can be applied without the need for large-scale modification of the existing system architecture.

[0041] After the dynamic orchestration of the task flow is completed, the real-time task scheduling and parallel computing phase based on a hierarchical distributed architecture is entered. This real-time task scheduling and parallel computing phase focuses on solving the problems of low data processing efficiency and high operation delay in traditional systems, and ensures efficient execution of tasks. The upper machine planning layer runs on an x86 architecture master computer, triggers task scheduling with a period of no more than 10 ms, and is mainly responsible for trajectory planning and inverse kinematics solving. The joint controller of the lower machine execution layer adopts an ARM Cortex-R series, receives instructions through a redundant CAN bus, realizes μs-level servo control, and also supports PDO real-time transmission of joint angle instructions. Hand-eye camera data triggers a vision pipeline through an interrupt event, and its priority is higher than that of periodic tasks. In terms of computing resource utilization, computationally intensive tasks such as point cloud segmentation are allocated to GPU acceleration (CUDA kernel), and logical tasks such as path planning are processed by CPU. Through node-level parallel design, the computing pressure is distributed among vision processing (hand-eye controller), motion planning (master computer), and joint control (6 independent controllers), avoiding single-point bottlenecks. Double buffering is used to overlap the execution of current frame vision processing and last frame motion control, which improves the throughput by 2.1 times, thereby realizing real-time scheduling of tasks and optimized utilization of resources, and completing real-time scheduling and parallel computing of tasks.

[0042] The system architecture is divided into two core levels: the upper machine planning layer and the lower machine execution layer, which work cooperatively through standardized communication interfaces. The upper machine planning layer is deployed on a high-performance master computer with an x86 architecture, and is responsible for computationally intensive and decision-intensive tasks, mainly including trajectory planning, inverse kinematics solving, and other core functions. The lower machine execution layer is composed of a joint controller cluster based on the ARM Cortex-R series, and is focused on high real-time servo control tasks. Through real-time guarantee at the hardware level, precise motion control is realized.

[0043] The upper machine planning layer adopts a deterministic periodic scheduling strategy, triggering the task scheduling cycle with a fixed period of no more than 10 ms. This scheduling mechanism is based on a priority-driven real-time scheduling algorithm, ensuring that critical control tasks can be completed within strict time constraints. In each scheduling period, the system prioritizes trajectory planning functions, using a spline interpolation-based trajectory generation algorithm to calculate smooth and dynamic characteristic intermediate trajectory points based on target pose sequences and motion constraint conditions. Subsequently, inverse kinematics solving is performed through the processing logic of numerical iterative optimization algorithms such as the Jacobian transpose method or pseudo-inverse method, which reversely calculates the required configuration parameters of each joint from the target end pose, ensuring that the robot arm can accurately reach the predetermined position.

[0044] The host computer planning layer employs a strict periodic scheduling strategy (typically 5-10ms), using a priority-driven real-time scheduling algorithm to ensure critical tasks are completed within a strict time window. Taking the assembly of precision electronic components by an industrial robot as an example (such as inserting a tiny connector into a designated position on a PCB board), the system first executes the trajectory planning function within each scheduling cycle. This process uses a trajectory generation algorithm based on cubic spline interpolation: upon receiving the target pose sequence (including the starting point P0 and intermediate transition points P1-P) from the upper-level system... n and endpoint P n+1 When dealing with Cartesian coordinates and attitude quaternions, the algorithm first performs time parameterization on discrete pose points (e.g., dividing them into n segments evenly according to the total motion time T). Then, it calculates the polynomial coefficients of the spline curve by solving a system of tridiagonal matrix equations (ensuring continuity of position, velocity, and acceleration), ultimately generating a smooth joint space or Cartesian space trajectory. This trajectory not only satisfies preset motion constraints (such as maximum velocity v) but also... max =200mm / s, maximum acceleration It can also automatically optimize acceleration and deceleration curves by adjusting the weight parameters of interpolation points, thus avoiding mechanical shocks.

[0045] After completing trajectory planning, inverse kinematics solution is performed, using a numerical iterative optimization algorithm based on the Jacobian pseudo-inverse: When the target end-effector pose (including position [x, y, z] and orientation [rx, ry, rz]) exceeds the current workspace of the robotic arm, the system first calls the reachability analysis module to determine feasibility (estimates the workspace boundary through Monte Carlo sampling). For feasible targets, the algorithm uses the current joint angle θ... k Using the initial values, the Jacobian matrix J (a matrix describing the partial derivatives of the end-effector pose with respect to the joint angles) is calculated analytically. Then, the incremental Δθ = J is solved using damped least squares. + ·Δx (where J + (where Δx is the pseudo-inverse matrix and Δx is the error vector between the target pose and the current pose). To avoid numerical instability near singular points, the algorithm dynamically adjusts the damping coefficient. (Based on the minimum singular value σ of SVD decomposition) min Adaptive adjustment =k·σ min, k is an empirical coefficient), ensuring the convergence of the iteration process. For example, in an assembly task, when the robot needs to move a connector from the initial position (x = 100 mm, y = 50 mm, z = 30 mm) to a tiny slot on a PCB board (x = 100.1 mm, y = 50.05 mm, z = 29.9 mm, attitude requirement Rx = 5°, Ry = 2°, Rz = 1°), the system calculates the exact angular increment (e.g. Δθ1 = 0.5°, Δθ2 = -0.3°) that each joint (e.g. ) needs to rotate through the above algorithm, finally outputting joint configuration parameters that meet the accuracy requirements (position error < 0.02 mm, attitude error < 0.1°), guiding the robotic arm to complete sub-millimeter precision assembly operations. This process strictly follows the time constraints of the scheduling period (e.g. inverse kinematics solving must be completed within 3 ms), ensuring the real-time performance of the entire control loop.

[0046] The lower machine execution layer adopts a distributed real-time control architecture, with each joint controller based on an ARM Cortex-R series processor. It receives control instructions from the upper machine through a redundant CAN bus network. The network topology uses a dual-channel redundant design to ensure system reliability in the event of a single-channel failure. The joint controller implements μs-level (microsecond-level) servo control through direct memory access (DMA) and hardware timer interrupt mechanisms, ensuring the real-time performance of control instructions. In particular, the system supports the PDO (Process Data Object) real-time transmission protocol for transmitting critical joint angle instructions and state feedback information. This protocol uses a fixed-cycle loop communication mode that is not affected by other network communication traffic, ensuring the most stringent real-time requirements.

[0047] Hand-eye camera data uses an interrupt event triggering mechanism to start the vision pipeline processing, which gives vision processing tasks higher priority than regular periodic tasks. When the camera sensor detects new frame data, a hardware interrupt is triggered, and the system's real-time scheduler prioritizes vision data processing, ensuring the timeliness of critical perception information. This design effectively solves the delay problem of vision processing affected by periodic task scheduling in traditional systems, ensuring that vision feedback can participate in control decision-making cycles in a timely manner.

[0048] The system implements a heterogeneous computing resource optimization allocation strategy, which reasonably allocates computing resources according to the characteristics of the task. For computationally intensive tasks (such as point cloud segmentation, feature extraction, etc.), a GPU acceleration mechanism is adopted, and through the processing logic of CUDA kernel parallel computing, large-scale parallel computing tasks are allocated to the multiple computing cores of the graphics processing unit (GPU), significantly improving the processing speed. For logically intensive tasks (such as path planning, task decision, etc.), the general computing core of the CPU is used for processing, taking advantage of its complex control logic processing capability. Through this matching optimization of computing tasks and hardware resources, the system maximizes computing efficiency.

[0049] The node-level parallel design is adopted to decompose the overall control task to multiple independent computing nodes, effectively distributing the computing pressure. Specifically, the visual processing task is undertaken by a dedicated hand-eye controller node, the motion planning task is handled by the host computer node, and the joint control task is allocated to 6 independent joint controller nodes, each node focusing on the computing of a specific functional domain to avoid single-point bottleneck problems. At the same time, the system introduces a double buffering mechanism (Double Buffering), which maintains two parallel data buffers to realize the overlapping execution of the current frame visual processing and the last frame motion control. This mechanism allows the visual processing pipeline to process newly collected frame data, while the motion control module executes control instructions based on the processing results of the previous frame, effectively hiding the computing delay and improving the overall throughput of the system by 2.1 times, ensuring that control tasks with strict real-time requirements can be stably executed.

[0050] After completing the real-time scheduling and parallel computing of the task, event-driven dynamic response and global data sharing based on a unified feature database become key steps, solving the problems of scattered data storage, repeated computation, and rigid process in traditional systems, further improving the effectiveness and adaptability of task execution. All intermediate data (such as point cloud, pose matrix, joint angle sequence) are encapsulated as octFeature objects and stored in the unified feature database (octFeatureDatabase). Hot data (such as real-time pose) is cached in the Memcached in-memory database with a read-write delay of <1ms, and cold data is archived to HDFS and migrated through the LRU strategy. Through the chain type conversion engine, data type conversion (such as point cloud → pose ICP registration) is automatically inferred and executed based on the feature type registry, and the conversion process uses memory mapping to realize zero-copy shared buffer. At the same time, event-driven dynamic response is supported, and conditional branch execution is triggered according to real-time sensor data (such as force sensor detecting excessive knob resistance), such as force control adaptive operator adjusting the gripping force of the robot arm in real time. Moreover, the event triggering mechanism (such as depth camera data ready triggering point cloud segmentation operator) ensures that the data flow flows according to actual needs, breaking through the limitations of traditional linear processes.

[0051] In this phase, a centralized data management system based on the unified feature database (octFeatureDatabase) is constructed, which standardizes and stores all intermediate calculation results (including point cloud data, pose matrix, joint angle sequence, etc. Key information). The system uses the octFeature object model as the basis for data encapsulation, and each data object contains feature type identification, timestamp, data validity and associated meta information, ensuring global consistency and traceability of the data. This architecture solves the synchronization difficulties and version inconsistency problems caused by scattered data storage in traditional systems, providing a reliable foundation for subsequent data sharing and dynamic response.

[0052] In terms of data storage strategy, the system implements a hierarchical storage management mechanism, which differentiates processing according to data access frequency and real-time requirements. Hot data (such as the current joint pose of the robot, the real-time pose of the end effector, etc. High-frequency access data) is cached in the Memcached in-memory database, using the LRU (Least Recently Used) cache replacement strategy to ensure that read and write delays are less than 1 millisecond, meeting the stringent requirements of real-time control for data access timeliness. Cold data (such as historical trajectory records, intermediate results of completed tasks, etc. Low-frequency access data) is automatically archived to HDFS (Hadoop Distributed File System), which uses distributed storage technology to efficiently manage large amounts of data, and uses an LRU migration strategy based on access frequency to dynamically adjust the distribution of data between memory and disk, optimizing overall storage resource utilization.

[0053] The system integrates a chain type conversion engine, which realizes automatic identification and conversion of data types through a feature type registry. This registry maintains a conversion relationship graph between all feature types, and when data type conversion is needed (for example, converting point cloud data to pose matrix, such as in the ICP registration process), the system automatically deduces the optimal conversion path based on the type registry and executes the corresponding conversion logic. This process uses a chain call of processing logic, first identifying the source feature type (such as octFeature) and the target feature type (such as octFeature), then finding a matching converter (such as the ICP registration converter) in the type registry, and finally performing the conversion operation and generating the target feature object.

[0054] In particular, the memory mapping zero-copy technology is implemented on the data sharing mechanism. By means of memory-mapped file, multiple operators can directly access data in the same physical memory, avoiding deep copy operations in the traditional data transfer process. This technology establishes a mapping relationship between the virtual memory address space and the physical memory, allowing multiple processes or threads to share the same data buffer. When an operator modifies the data, other operators can immediately see the updated results, ensuring data consistency and significantly improving data transmission efficiency, eliminating the performance overhead and delay caused by data replication in traditional systems.

[0055] The event-based dynamic response mechanism is implemented, which can dynamically adjust the control flow and execution strategy according to real-time sensor input (such as force sensors, vision sensors, etc.). When the sensor detects an abnormal situation (for example, the force sensor detects that the knob resistance exceeds the preset threshold, indicating that there may be mechanical jamming or over-tightening), the system will immediately trigger the conditional branch execution logic. Taking the force control adjustment scene as an example, when the force control sensor data exceeds the safety threshold, the system will dynamically activate the force control adaptive operator. Based on the current force feedback value, the operator calculates and adjusts the gripping force or contact force of the robot arm in real time through the processing logic of the feedback control algorithm (such as PID control or impedance control), ensuring the safety and stability of the operation process.

[0056] The event-driven mechanism is not limited to abnormal handling, but is widely used in dynamic optimization of normal processes. For example, when the depth camera data is ready (detected through hardware interrupts or data ready flags), the system will automatically trigger the execution of the point cloud segmentation operator without waiting for a fixed period or manual intervention. Similarly, when the vision processing result (such as the pose estimation of the target object) reaches the expected accuracy, the system will trigger the subsequent path planning or grasping operation, ensuring that the entire control flow closely follows the actual environmental changes. This dynamic response capability based on actual data state enables the system to break through the rigid limitations of traditional linear processes and achieve truly adaptive control of perception-decision-execution closed loop.

[0057] Based on the event-driven mechanism, the adaptive flow control capability is further implemented, which can dynamically adjust the control strategy according to the runtime environment changes and task execution state. When certain conditions are detected (such as the pose change of the target object exceeding the threshold, the movement of the robot arm being blocked, the change of the environmental lighting conditions, etc.), the system can dynamically re-plan the task flow and select the most suitable processing path for the current situation. For example, in the power cabinet knob operation task, if the initial visual positioning fails, the system can automatically trigger a re-scan or use a backup positioning scheme instead of simply reporting an error or terminating the task.

[0058] This mechanism also supports fusion decision-making of multi-modal sensor data, by comprehensively analyzing information from different sensors (such as visual cameras, force sensors, tactile sensors, etc.), the system can make more accurate and robust control decisions. For example, when grabbing fragile objects, the system not only considers visual positioning results, but also combines force feedback information, dynamically adjusts the grabbing force and contact strategy, to ensure the safety and success rate of the operation. Through this global data sharing and event-driven dynamic response capability, the system realizes the transformation from traditional fixed process control to intelligent adaptive control, significantly improving the robustness, adaptability and overall effect of task execution in complex environments.

[0059] In the description of the present specification, the description referring to the terms "one embodiment", "an example", "a specific example" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0060] The preferred embodiments of the application disclosed above are only used to help explain the application. The preferred embodiments do not describe all the details, nor limit the application to the specific embodiments. Obviously, many modifications and variations can be made according to the content of the present specification. The present specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the application, so that those skilled in the art can well understand and utilize the application. The application is limited only by the claims and their full scope and equivalents.

Claims

1. A hand-eye collaborative robot control method based on dynamic operator orchestration, characterized in that: The control method comprises the following steps: The bottom layer driver of different brand devices is packaged into a unified interface, the corresponding plug-in is dynamically loaded through the device ID at runtime, visual processing, pose calculation and force control adjustment are packaged into standardized operators, the topological relationship between the standardized operators is defined through a configuration file, a directed acyclic graph execution pipeline is constructed, a dynamic operator library is dynamically loaded, a preset operator library is combined, and dynamic scheduling of a task flow is completed; The host computer planning layer triggers task scheduling periodically, the joint controller of the lower computer execution layer receives instructions through the redundant CAN bus for servo control, the hand-eye camera data triggers the visual pipeline through an interrupt event, the visual processing, motion planning and joint control are distributed to reduce the calculation pressure, the double buffering mechanism is used to overlap the current frame visual processing and the last frame motion control for execution, and real-time scheduling and parallel calculation of the task are completed. All intermediate data are packaged into objects and stored in a feature database, hot data are cached in a memory database, cold data are archived and migrated through a strategy, data type conversion is automatically derived and executed based on a feature type registry, and conditional branch execution is triggered according to real-time sensor data.

2. The dynamic operator-based hand-eye coordination robotic control method of claim 1, wherein: The host computer planning layer triggers task scheduling periodically, the joint controller of the lower computer execution layer receives instructions through the redundant CAN bus for servo control, the hand-eye camera data triggers the visual pipeline through an interrupt event, the visual processing, motion planning and joint control are distributed to reduce the calculation pressure, the double buffering mechanism is used to overlap the current frame visual processing and the last frame motion control for execution, and real-time scheduling and parallel calculation of the task are completed, including the following steps: The system architecture is divided into a host computer planning layer and a lower computer execution layer, and the two layers work cooperatively through a standardized communication interface; the host computer planning layer is deployed on an x86 architecture host computer, and the lower computer execution layer is composed of a joint controller cluster based on an ARM Cortex-R series processor; The host computer planning layer adopts a periodic scheduling strategy, triggers a task scheduling cycle periodically, and controls the task to be completed within a time constraint based on a priority-driven real-time scheduling algorithm; The lower computer execution layer adopts a distributed real-time control architecture, each joint controller is based on an ARM Cortex-R series processor, and receives control instructions from the host computer through a redundant CAN bus network; The hand-eye camera data starts the visual pipeline processing through an interrupt event triggering mechanism, a hardware interrupt is triggered when the camera sensor detects that a new frame of data arrives, and the real-time scheduling program processes the visual data preferentially; A heterogeneous computing resource optimization allocation strategy is implemented, computing resources are allocated according to the task characteristics, and the overall control task is decomposed into multiple independent computing nodes.

3. The dynamic operator-based hand-eye coordination robotic control method of claim 2, wherein: The host planning layer adopts a periodic scheduling strategy, triggers a task scheduling cycle periodically, and controls the task to be completed within a time constraint based on a priority-driven real-time scheduling algorithm, including executing a trajectory planning function in each scheduling cycle, adopting the processing logic of a spline interpolation-based trajectory generation algorithm, calculating intermediate trajectory points that are smooth and meet the dynamic characteristics according to a target pose sequence and a motion constraint condition, and then performing inverse kinematics solving, and through the processing logic of a numerical iterative optimization algorithm, the configuration parameters required by each joint are back calculated from the target end pose.

4. The dynamic operator-based hand-eye coordination robotic control method of claim 3, wherein: The network topology of the lower machine execution layer adopts a dual-channel redundant design, ensures the real-time performance of control instructions through a memory access and hardware timer interrupt mechanism, supports a PDO real-time transmission protocol, and uses the PDO real-time transmission protocol to transmit key joint angle instructions and state feedback information, and the PDO real-time transmission protocol adopts a fixed-cycle cyclic communication mode.

5. The dynamic operator-based hand-eye coordination robotic control method of claim 3, wherein: A heterogeneous computing resource optimization allocation strategy is implemented, and computing resources are allocated according to task characteristics, including the following steps: For a computing-intensive task, a GPU acceleration mechanism is adopted, and through the processing logic of CUDA kernel parallel computing, the parallel computing task is allocated to multiple computing cores of a graphics processing unit; For a logic-intensive task, general computing of a CPU is used.

6. The dynamic operator-based hand-eye coordination robotic control method of claim 1, wherein: Different brands of device underlying drivers are packaged into a unified interface, and corresponding plug-ins are dynamically loaded through device IDs at runtime, visual processing, pose calculation and force control adjustment are packaged into standardized operators, the topological relationship between standardized operators is defined through a configuration file, a directed acyclic graph execution pipeline is constructed, and by means of dynamically loading operator dynamic libraries, a preset operator library is combined to complete task flow dynamic scheduling, including the following steps: A hardware abstraction layer is constructed, and the octDevice and octRobot modules are used to realize standardized interface packaging of multiple brands of devices, the underlying drivers of different devices are abstracted to form a unified device operation interface specification; Standardized image acquisition interfaces and point cloud generation interfaces are defined for vision devices, and trajectory execution interfaces and joint state feedback interfaces are defined for robot arms, and corresponding device driver plug-ins are developed for each device based on these standard interfaces; Plug-ins are dynamically loaded through device IDs at runtime to realize hybrid deployment capability of cameras and robot arms; Each operator complies with the specification of the OperatorBase abstract base class, implements the execute() method, and defines the input / output feature slot mechanism; A JSON configuration file is used to define the topological relationship between operators to construct an execution pipeline in the form of a directed acyclic graph; At runtime, the dlopen() dynamic loading mechanism is used to load corresponding operator dynamic link libraries in real time according to the operator types specified in the configuration file.

7. The dynamic operator-based hand-eye coordination robotic control method of claim 6, wherein: The operators include a point cloud segmentation operator, a 3D registration operator and a force control adjustment operator.

8. The dynamic operator-based hand-eye coordination robotic control method of claim 7, wherein: The point cloud segmentation operator processes original point cloud data, extracts the point cloud region of a target object through the processing logic based on normal vector analysis and Euclidean distance clustering, and outputs the segmented point cloud data; The 3D registration operator receives segmented point cloud data and target model point cloud, adopts the processing logic of the iterative closest point algorithm, and calculates an optimal pose transformation matrix of the current point cloud and the target model; The force control adjustment operator processes force sensor data, dynamically adjusts the clamping force or contact force of the mechanical arm based on the processing logic of the force feedback control algorithm, and outputs a control instruction.

9. The dynamic operator-based hand-eye coordination robotic control method of claim 8, wherein: Each operator receives upstream data through a defined input slot, processes the data through internal logic, and passes the processing results to downstream operators through an output slot, forming a data flow dependency relationship.

10. A hand-eye collaborative robot control system based on dynamic operator orchestration for implementing the control method of any one of claims 1-9, characterized by: It includes a dynamic arrangement module, a scheduling and calculation module, and an execution module. The dynamic arrangement module encapsulates the underlying drivers of different brands of devices into a unified interface, dynamically loads the corresponding plug-ins through the device ID at runtime, encapsulates visual processing, pose calculation, and force control adjustment into standardized operators, defines the topological relationship between standardized operators through a configuration file, constructs a directed acyclic graph execution pipeline, dynamically loads operator libraries using dlopen(), and combines preloaded operator libraries. The scheduling and calculation module: the host computer planning layer triggers task scheduling periodically, the joint controller of the lower computer execution layer receives instructions through the redundant CAN bus for servo control, and the hand-eye camera data triggers the visual pipeline through an interrupt event to distribute the calculation pressure of visual processing, motion planning, and joint control, and uses a double buffering mechanism to overlap the execution of the current frame visual processing and the last frame motion control. The execution module: all intermediate data are encapsulated as octFeature objects and stored in a feature database, hot data are cached in a Memcached in-memory database, cold data are archived to HDFS and migrated through an LRU strategy, data type conversion is automatically derived and executed based on a feature type registry, and conditional branch execution is triggered based on real-time sensor data.

Images