A mobile robot task scheduling method and system based on a heterogeneous computing architecture

Abstract

This application provides a mobile robot task scheduling method and system based on a heterogeneous computing architecture, relating to the field of intelligent scheduling technology. The method includes: constructing a workspace geometric model based on the geometric structure of a planting row; obtaining the motion state parameters of the mobile robot using a first computing unit; determining the look-ahead compensation amount by combining the statistical value of the inference time of the target recognition task by a second computing unit; determining a spatiotemporal pre-aiming slice in the visual field region based on the look-ahead compensation amount and the workspace geometric model, and cropping image data located within the slice from the original image data; the second computing unit performing target recognition on the image data, generating a work instruction corresponding to the recognized target, and controlling the lateral actuator to perform the work according to the work instruction. This can alleviate the spatiotemporal asynchrony problem between perception results and execution position in a moving work scenario, reduce missed and misaligned tasks, and reduce the communication and computational load of the heterogeneous computing system.

Description

Technical Field

[0001] This application relates to the field of intelligent scheduling technology, and more specifically, to a mobile robot task scheduling method and system based on a heterogeneous computing architecture. Background Technology

[0002] Currently, mobile robots equipped with forward imaging devices and lateral actuators are increasingly being used in mobile operations such as fruit and vegetable harvesting, targeted spraying, and pest and disease identification. To improve the accuracy and real-time performance of target recognition, many solutions have introduced heterogeneous computing architectures, including FPGAs and NPUs, to reduce the burden on the host computer and shorten the overall latency of the perception-decision-execution link by completing image preprocessing, target detection, and trajectory planning at the edge.

[0003] In existing technologies, mobile robots typically travel continuously between rows in structured planting. A forward imaging device acquires visual images along the row direction. An FPGA or other preprocessing unit performs simple cropping, scaling, or format conversion on the images, and then transmits the processed images or regions of interest to an NPU for target recognition. Subsequently, based on the recognition results, the lateral actuators are controlled to perform operations such as grasping or spraying. This type of solution can basically meet the operational requirements under stationary or low-speed conditions. However, in scenarios where the robot maintains a certain travel speed and the objects being worked on are densely distributed, the NPU inference in the perception link inevitably introduces millisecond-level or even longer inference time.

[0004] Because the mobile robot continues to move along the planting row during NPU inference, existing solutions often directly drive the lateral actuators based on the image coordinates at the time of capture. This makes it difficult to compensate for the robot's displacement during inference in a timely and accurate manner. As a result, there is a temporal and spatial deviation between the visual position of the target and the physical working envelope of the lateral actuators, leading to a lag in recognition results compared to the actual execution position, resulting in problems such as missed or misaligned operations. To mitigate these risks, some solutions "wait" for recognition results by reducing the travel speed or frequently pausing, but this significantly impacts the overall row operation efficiency.

[0005] On the other hand, in heterogeneous computing architectures, continuously transmitting entire frames or large areas of image data to the NPU for processing consumes significant bus bandwidth and computing resources. This makes it difficult to maintain sufficient frame rates and resolutions under limited power consumption constraints, resulting in heavy communication and computational loads and impacting the system's scalability in complex agricultural scenarios. In summary, existing technologies for mobile robot operations still face the challenge of overcoming the spatiotemporal asynchrony problem caused by the time-consuming inference process of the NPU, which lags behind the physical execution position, while simultaneously reducing the communication and computational load of heterogeneous computing systems. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this application provides a mobile robot task scheduling method and system based on a heterogeneous computing architecture.

[0007] Firstly, this application provides a mobile robot task scheduling method based on a heterogeneous computing architecture, including:

[0008] On a mobile robot equipped with a forward imaging device and a lateral actuator, a work space geometric model is constructed based on the geometric structure of a structured planting row. The work space geometric model represents the spatial correspondence between the visual field of view of the forward imaging device and the physical work envelope of the lateral actuator.

[0009] The motion state parameters of the mobile robot are obtained by the first computing unit, and the look-ahead compensation amount is determined based on the motion state parameters and the statistical value of the inference time of the target recognition task by the second computing unit. The first computing unit communicates with the forward imaging device, and the second computing unit communicates with the first computing unit. The motion state parameters include the instantaneous linear velocity and / or angular velocity along the planting row direction.

[0010] The first computing unit uses the look-ahead compensation amount and the workspace geometric model to determine a spatiotemporal preview slice in the visual field of view;

[0011] The first computing unit uses the acquired raw image data to crop out the image data located within the spatiotemporal preview slice, and sends the cropped image data to the second computing unit;

[0012] The second computing unit performs target recognition on the image data within the spatiotemporal pre-aiming slice, generates a work instruction corresponding to the recognized target, and controls the lateral execution mechanism to perform the work according to the work instruction.

[0013] Optionally, the look-ahead compensation amount is used to characterize the row displacement of the mobile robot moving along the planting row within a predetermined look-ahead time, and the predetermined look-ahead time is determined based on the statistical value of the inference time of the second computing unit for the target recognition task.

[0014] Optionally, determining the spatiotemporal pre-aiming slice includes:

[0015] In the work space geometric model, the physical work envelope area is translated in the opposite direction of the movement direction of the mobile robot by the look-ahead compensation amount to obtain the translated work area;

[0016] The translated work area is projected onto the imaging plane of the forward imaging device, and the projected area serves as the spatiotemporal preview slice.

[0017] Optionally, the first computing unit is an FPGA unit, and the second computing unit is an NPU unit.

[0018] Optionally, controlling the lateral actuator to perform the operation according to the operation instruction includes:

[0019] Based on the work space geometric model, it is determined that the identified target is located within the physical work envelope area;

[0020] In response to the determination that the location is within the physical operation envelope, the lateral actuator is triggered to execute the operation command.

[0021] Optionally, determining the spatiotemporal pre-aiming slice includes:

[0022] Within the time window corresponding to the predetermined look-ahead time, the first computing unit simultaneously collects the high-frequency sequence of the drive current of the mobile robot chassis motor and the point cloud flow output by the 4D radar sensing unit associated with the forward imaging device for sensing the spatial structure of the working lane, while the mobile robot travels along the working lane defined by the multi-layer three-dimensional planting rack.

[0023] The first computing unit determines the instantaneous jitter offset vector of the forward imaging device based on the high-frequency sequence of the driving current, and determines the sliding pose compensation amount of the mobile robot along the planting row direction based on the point cloud flow and the encoder speed of the chassis motor.

[0024] Using the first calculation unit, based on the instantaneous jitter offset vector and the slip pose compensation amount, the visual field area corresponding to the working lane in the working space geometric model is corrected, and the spatiotemporal preview slice after jitter and slip compensation is determined based on the corrected working space geometric model.

[0025] Optionally, determining the instantaneous jitter offset vector of the forward imaging device includes:

[0026] The first computing unit performs wavelet packet transform on the high-frequency sequence of the driving current to extract the current ripple component that matches the mechanical resonance frequency band.

[0027] Based on a preset current-end jitter transfer function, the current ripple component is mapped to the instantaneous jitter offset vector of the forward imaging device at the imaging moment.

[0028] Optionally, determining the sliding pose compensation amount of the mobile robot along the planting row direction based on the point cloud flow and the encoder speed of the chassis motor includes:

[0029] Using the first calculation unit, based on the pitch angle information of the point cloud flow, a height band corresponding to the height range of the multi-layer three-dimensional planting rack column is selected in the vertical direction, and the horizontal slice point cloud distributed along the planting row direction within the height band is extracted.

[0030] Doppler velocity field analysis is performed on the horizontal slice point cloud, and an environmental reference velocity matrix is ​​constructed based on the column echo clusters distributed within a preset spacing range along the working roadway.

[0031] The environmental reference velocity matrix and the encoder velocity are cross-correlated to generate a nonlinear slip coefficient sequence.

[0032] The slip pose compensation amount is obtained by performing time-domain integration on the nonlinear slip coefficient sequence along the planting row direction using the first computing unit.

[0033] Optionally, the preset current-end jitter transfer function is obtained in the following manner:

[0034] During the static calibration phase when the mobile robot is stationary at the typical working lane position corresponding to the structured planting row, the first computing unit applies a step excitation signal to the lateral actuator, and the forward imaging device acquires the displacement attenuation curve of the calibration plate.

[0035] The motor current response sequence caused by the step excitation signal is recorded synchronously.

[0036] A mapping model between the motor current response sequence and the displacement attenuation curve is established using a system identification algorithm, and the parameters of the mapping model are stored in the on-chip storage resources of the first computing unit as the current-end jitter transfer function.

[0037] Secondly, this application provides a mobile robot task scheduling system based on a heterogeneous computing architecture, comprising:

[0038] A construction module is used to construct a work space geometric model on a mobile robot equipped with a forward imaging device and a lateral actuator, based on the geometric structure of a structured planting row. The work space geometric model represents the spatial correspondence between the visual field of view of the forward imaging device and the physical work envelope of the lateral actuator.

[0039] The processing module uses a first computing unit to obtain the motion state parameters of the mobile robot, and determines the look-ahead compensation amount based on the motion state parameters and the statistical value of the inference time of the second computing unit for the target recognition task. The first computing unit communicates with the forward imaging device, and the second computing unit communicates with the first computing unit. The motion state parameters include the instantaneous linear velocity and / or angular velocity along the planting row direction.

[0040] The first computing unit uses the look-ahead compensation amount and the workspace geometric model to determine a spatiotemporal preview slice in the visual field of view;

[0041] The first computing unit uses the acquired raw image data to crop out the image data located within the spatiotemporal preview slice, and sends the cropped image data to the second computing unit;

[0042] The second computing unit performs target recognition on the image data within the spatiotemporal pre-aiming slice, generating a job instruction corresponding to the recognized target.

[0043] The scheduling module is used to control the lateral actuator to perform the operation according to the operation instruction.

[0044] Compared to existing technologies, this application, focusing on a structured planting row scenario, first constructs a geometric model of the working space between the visual field of view of the forward imaging device and the physical working envelope of the lateral actuator. It introduces a spatial correspondence oriented towards a specific row direction, enabling unified modeling of the visual perception coordinates and the actual reachable area of ​​the actuator within the same geometric framework. Based on this, the first computing unit acquires the directional motion state parameters of the mobile robot, and combines this with the estimation of the inference time for the target recognition task by the second computing unit to obtain a look-ahead compensation amount characterizing the robot's directional displacement during the inference time, explicitly linking the robot's kinematics with the perception delay.

[0045] By mapping the look-ahead compensation amount to the geometric model of the work space, a spatiotemporal pre-aiming slice corresponding to the future physical work envelope area is determined within the visual field of view of the forward imaging device. Image data is cropped only within this pre-aiming slice and sent to the second computing unit for target recognition. This ensures that the recognized target enters the physical work envelope area of ​​the lateral actuator precisely at the moment of inference completion. Thus, while maintaining robot movement, the perception result and execution position are coordinated in time and space, significantly reducing the probability of missed and misaligned tasks due to recognition delay. Simultaneously, since recognition processing is only performed on the image region related to the future work window, this application reduces the size of the image data transmitted to the second computing unit and the amount of unnecessary computation, reducing the communication and computational load of the heterogeneous computing system. This is beneficial for maintaining a high work cycle and recognition performance under power-constrained edge hardware conditions, improving overall inter-row work efficiency and system robustness. Attached Figure Description

[0046] Figure 1 A flowchart illustrating a mobile robot task scheduling method based on a heterogeneous computing architecture, provided for embodiments of this application;

[0047] Figure 2A flowchart illustrating a method for determining spatiotemporal preview slices provided in this application embodiment;

[0048] Figure 3 A flowchart illustrating a method for controlling a lateral actuator to perform operations, as provided in this application embodiment;

[0049] Figure 4 This is a schematic diagram of a mobile robot task scheduling system based on a heterogeneous computing architecture, provided as an embodiment of this application. Detailed Implementation

[0050] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0051] See Figure 1 The flowchart shown is a process for scheduling mobile robot tasks based on a heterogeneous computing architecture, according to an embodiment of this application. It includes steps S101 to S105, wherein:

[0052] S101: On a mobile robot equipped with a forward imaging device and a lateral actuator, a work space geometric model is constructed based on the geometric structure of the structured planting row. The work space geometric model represents the spatial correspondence between the visual field area of ​​the forward imaging device and the physical work envelope area of ​​the lateral actuator.

[0053] S102: The motion state parameters of the mobile robot are obtained by the first computing unit, and the look-ahead compensation amount is determined based on the motion state parameters and the statistical value of the inference time of the second computing unit for the target recognition task. The first computing unit communicates with the forward imaging device, and the second computing unit communicates with the first computing unit. The motion state parameters include the instantaneous linear velocity and / or angular velocity along the planting row direction.

[0054] S103: Using the first computing unit, based on the look-ahead compensation amount and the work space geometric model, determine the spatiotemporal pre-aiming slice in the visual field of view;

[0055] S104: The first computing unit uses the acquired raw image data to crop out the image data located within the spatiotemporal preview slice, and sends the cropped image data to the second computing unit;

[0056] S105: The second computing unit performs target recognition on the image data within the spatiotemporal pre-aiming slice, generates a work instruction corresponding to the recognized target, and controls the lateral execution mechanism to perform the work according to the work instruction.

[0057] Regarding the above S101:

[0058] In practical implementation, S101 is used to establish a workspace geometric model that can be directly called by the control program in the current working environment of the mobile robot, so that the visual field of view of the forward imaging device and the physical work envelope of the lateral actuator have a clear spatial correspondence in a unified coordinate system. The mobile robot can be an inter-row operation robot, a lane-type transport robot, or a three-dimensional planting rack lane operation robot, with a forward imaging device and a lateral actuator mounted on its chassis. The forward imaging device can be a monocular camera, a binocular camera, or a depth camera, used to periodically output forward image frames; the lateral actuator can be a laterally extending robotic arm, a lateral telescopic pick-and-place mechanism, or a lateral gripping mechanism, and the end effector can be a gripper, a suction cup, or a shear. The first computing unit can be integrated into the chassis main control board or an edge computing board that communicates with it, and is used to perform the geometric modeling process.

[0059] For example, the first computing unit is used to construct and store the workspace geometric model and establish a data communication link with the forward imaging device to obtain information such as the calibration parameters of the imaging device and / or image frame timestamps. The first computing unit can be a programmable logic device, a system-on-a-chip, or an industrial control computing unit. For example, it can be implemented using a heterogeneous SoC including programmable logic resources and a general-purpose processing core, wherein the programmable logic part is used to perform geometric operations such as coordinate transformation and region intersection in parallel, and the general-purpose processing core is used to manage configuration parameters and data structures.

[0060] For example, the first computing unit can be implemented using an FPGA development board or an FPGA+ARM SoC platform, and connected to the forward imaging device via MIPI-CSI, LVDS, or Gigabit Ethernet; the first computing unit maintains the data structure of the workspace geometric model in memory, the data structure including at least the global coordinate system definition, the imaging device extrinsic parameter matrix, the lateral actuator base extrinsic parameter matrix, and the boundary parameter set of the visual field region and the physical work envelope region, the boundary parameter set can be represented by any of the following: polyhedron vertex set, bounding box parameters, or meshed occupancy set.

[0061] In this embodiment, the structured planting row refers to a row of crops, a row of vertical planting racks, or a row of shelves arranged approximately parallel to each other along the row direction, forming a working aisle that can be accessed by robots. Its geometric structure includes at least parameters such as the row direction, the center line of the aisle, the position of the left and right boundaries of the aisle, and the width of the aisle. In the case of a vertical planting rack, the geometric structure may also include the spacing of the columns along the row direction and the vertical height range of the multi-level storage locations. The geometric structure of the structured planting row can be obtained in two ways: one is offline configuration before deployment, such as importing parameters such as row spacing, aisle width, and column spacing from greenhouse layout diagrams, planting rack installation diagrams, or warehouse planning data; the other is online perception during operation, such as the first computing unit detecting row boundaries from camera images and obtaining the aisle boundary lines through straight line fitting, or extracting the point clusters of corresponding columns from 4D radar point clouds and determining the main direction of the row through principal component analysis.

[0062] In specific implementations, edge detection and Hough transform line detection can be performed on the forward image to obtain the parameters of the left and right boundary lines; clustering and principal component analysis can also be performed on the point cloud to obtain the tunnel direction. These image processing and point cloud processing algorithms are well-known methods in the field, and existing library functions can be directly called in the implementation.

[0063] The operational space geometric model is a geometric data structure used to describe the spatial region related to the operation in a unified coordinate system. It includes at least the transformation relationships between the global coordinate system, the robot body coordinate system, the forward imaging device coordinate system, and the lateral actuator base coordinate system, as well as the spatial boundary information of the visual field of view and the physical operational envelope represented in this coordinate system. The global coordinate system can be defined with the channel centerline as the travel axis, with the travel axis defined as the X-axis, the lateral direction as the Y-axis, and the vertical direction as the Z-axis. The robot body coordinate system is fixed at the robot chassis reference point. The pose matrices of the forward imaging device coordinate system and the lateral actuator base coordinate system are obtained relative to the robot body coordinate system through installation position calibration. These pose matrices can be obtained through conventional hand-eye calibration or mechanical installation dimension calculations. For example, camera extrinsic parameters can be obtained using Zhang Zhengyou's camera calibration and extrinsic parameter solving method, and actuator base extrinsic parameters can be obtained using rigid body transformation based on measured dimensions.

[0064] It is understood that the working space geometric model is a geometric data structure built based on calibration parameters and structured planting row geometric parameters. It is not a machine learning model. Its generation process does not require training. It is obtained by reading configuration parameters, imaging device calibration parameters and actuator parameters and performing geometric operations.

[0065] In the specific process of constructing the geometric model of the workspace, the first computational unit can execute the following steps: First, read the configuration file or perception results to obtain the row direction unit vector, channel centerline parameters, and left and right boundary line parameters of the structured planting rows. For example, the channel width can be configured within the range of 1.0m to 2.5m. Second, read the intrinsic parameters (focal length, principal point coordinates, distortion coefficients) and extrinsic parameters (translation and attitude relative to the body coordinate system) of the forward imaging device and convert them into a camera model for 3D projection. Third, read the mechanism parameters and installation pose of the lateral actuator, calculate the reachable range of its end effector in the body coordinate system, and generate an approximate boundary of the physical workspace envelope in the global coordinate system. For example, a cuboid or convex polyhedron can be used to enclose the reachable space of the end effector. Feasible reachable space calculations can be achieved through conventional forward and inverse kinematics solutions of the robotic arm and multi-pose sampling. For example, Denavit-Hartenberg parametric modeling can be used in conjunction with a numerical solver to generate a set of end effector trajectory points, and then the envelope of the point set can be calculated.

[0066] Based on the above, the visual field of view can be represented as the effective observation space of the forward imaging device in the global coordinate system. For example, the first calculation unit generates a truncated cone based on the camera's field of view angle and a preset effective observation distance range (e.g., 0.5m near end and 5m far end), and intersects this truncated cone with the spatial region defined by the left and right boundaries of the channel to obtain the visual field of view restricted within the planting row channel. The physical operation envelope is represented as the set of spaces that the end effector of the lateral actuator can reach and perform operations in the global coordinate system, such as a spatial region within a certain range in the horizontal direction (0.3m to 1.0m), vertical direction (0.4m to 1.5m), and forward and backward of the row. By storing the spatial boundary expressions of the visual field of view and the physical operation envelope, as well as the projection / reverse projection relationship between the camera coordinate system and the global coordinate system, in the operation space geometric model, the first calculation unit can calculate the corresponding spatial position of the image region in subsequent steps, or project the physical operation envelope onto the imaging plane, thereby realizing the spatial correspondence between the two. The specific solution to the above correspondence can be achieved through a standard three-dimensional projection model and a homogeneous coordinate transformation matrix, which will not be elaborated here.

[0067] Regarding S102 above:

[0068] In an optional embodiment, S102 is used during the operation of the mobile robot while it is moving to generate a look-ahead compensation amount that can be directly used by the control program, based on the robot's current motion state and the computational delay of the target recognition task. The look-ahead compensation amount can be understood as "the expected displacement of the mobile robot along the planting row direction before the target recognition result is generated and can be used for control," serving as the temporal-spatial alignment basis for subsequent image region selection and task triggering. The "motion state parameters" here include at least the instantaneous linear velocity and / or angular velocity along the planting row direction. The instantaneous linear velocity characterizes the robot's translational speed along the row direction, and the angular velocity characterizes the robot's rotational rate about the vertical axis; when the robot exhibits slight serpentine movement or steering corrections, introducing angular velocity helps to more accurately characterize the changing trend of the row displacement.

[0069] The first calculation unit can acquire motion state parameters using conventional chassis odometer or inertial measurement methods. For example, the first calculation unit can read wheel speed encoder counts from the chassis drive controller, calculate the instantaneous wheel speed based on wheel diameter parameters and encoder resolution, and then combine this with the attitude projection along the planting row to obtain the instantaneous linear velocity. Simultaneously, the first calculation unit can read gyroscope outputs from the inertial measurement unit to obtain angular velocity. The above readings can be achieved via CAN bus, RS485, or Ethernet, with a sampling period set from 5ms to 20ms to meet the real-time requirements of in-motion operations. If further improvement in speed and angular velocity stability is needed, the first calculation unit can filter or fuse the raw wheel speed and gyroscope data, for example, by using complementary filtering or extended Kalman filtering to fuse the wheel speed odometer and inertial measurement data. Such filtering and fusion are conventional techniques in the field and can be implemented directly using existing algorithm libraries.

[0070] The statistical values ​​of the inference time for the target recognition task by the second computing unit are used to characterize the estimated time from the readiness of input data to the availability of the output recognition result. The second computing unit can set a uniform timing boundary for the target recognition task and perform statistics and estimation within the same timing boundary.

[0071] As an example, after receiving a frame of image data to be recognized, the second computing unit uses a high-precision timer to record the inference start timestamp and records the inference end timestamp when the inference output is generated. The difference between the two timestamps is taken as the actual inference time for that frame. The second computing unit further performs statistical analysis on the actual inference time of the most recent several frames, outputs the statistical value of the inference time, and sends it to the first computing unit. The statistical method can be any of the following: moving average, exponentially weighted moving average, or quantile estimation. For example, the moving average of the most recent 8 frames can be used as the statistical value, or an exponentially weighted moving average can be used with a weighting coefficient set to 0.1 to 0.3 to improve the responsiveness to fluctuations in computing power; alternatively, the higher quantile of the inference time of the most recent several frames can be used as the statistical value to improve the conservatism of the time limit. The timing and statistical methods described above can be directly implemented by the second computing unit without the need for additional hardware.

[0072] Communication between the second computing unit and the first computing unit can be achieved using shared memory, PCIe, AXI bus, or Ethernet. To ensure consistency between motion state parameters and inference time statistics in the time dimension, the second computing unit can attach a timestamp or frame number corresponding to the inference time statistics when sending them. Upon receiving the statistics, the first computing unit aligns them with the sampling time of the current motion state parameters and selects the inference time statistics that match the current processing frame. This time alignment process can be achieved through a unified clock source, timestamp conversion, or frame number matching, which are conventional implementation methods in this field.

[0073] For example, the second computing unit can be a computing unit with neural network inference acceleration capabilities. It establishes a data communication link with the first computing unit to receive image data forwarded or cropped by the first computing unit and to send back inference time statistics and metadata of the target recognition result. As an example, the second computing unit can be implemented using an edge inference module with an integrated neural network processor, such as an NPU accelerator card, a SoC module with an NPU, or an industrial computing unit with an AI inference accelerator. It runs a target recognition inference program at the software level, and the inference program can be deployed based on the inference engine of a commonly used deep learning inference framework, such as TensorRT, ONNXRuntime, or OpenVINO. The above frameworks are only examples, and the second computing unit can also be implemented using other equivalent inference engines and accelerated runtime environments.

[0074] After obtaining the motion state parameters and inference time statistics, the first computing unit determines the look-ahead compensation amount. When the look-ahead compensation amount is used to characterize the directional displacement within a predetermined look-ahead time, the predetermined look-ahead time can be directly given by the inference time statistics, or a fixed margin can be added to the inference time statistics to cover scheduling and communication overhead. The margin can be set to a range of 5ms to 30ms. The look-ahead compensation amount can be obtained by predicting the directional displacement: when only the instantaneous linear velocity of the directional displacement is used, the look-ahead compensation amount can be calculated as the product of the instantaneous linear velocity of the directional displacement and the predetermined look-ahead time; when angular velocity is used simultaneously, the first computing unit can also perform discrete integration prediction of the directional displacement within the predetermined look-ahead time, for example, by numerically integrating the velocity with an integration step size of 5ms to 20ms, thereby obtaining a directional displacement prediction value that better matches the steering correction condition. The above calculation process is completed on the first computing unit side, and the output look-ahead compensation amount is stored in the form of distance or directional displacement for subsequent steps to call.

[0075] Regarding the above S103:

[0076] In specific implementation, the inputs of S103 include: the operational space geometric model established and stored by S101, the look-ahead compensation amount obtained by S102, and the time information corresponding to the current image frame; the output of S103 is the "spatiotemporal preview slice parameter" that can be directly called by the control program. The spatiotemporal preview slice parameter is used to uniquely determine an image region in the imaging plane, and may include fields such as the coordinates of the polygon vertices or the coordinates of the bounding rectangle of the image region, the corresponding timestamp or frame number, etc.

[0077] To facilitate subsequent hardware / software collaboration, the spatiotemporal preview slice parameters can be expressed in integer form in pixel coordinates, such as a rectangular window representation with the coordinates of the top left corner and the width and height, or a convex quadrilateral window representation with four vertices. When using an FPGA for fast geometric calculations, the above coordinates can be stored in on-chip memory resources in fixed-point format.

[0078] After receiving the motion state parameters and look-ahead compensation amount at the current moment during operation, the first computing unit first determines the look-ahead time reference for spatial extrapolation. For example, the first computing unit can regard the look-ahead compensation amount as the displacement in the direction of travel within a predetermined look-ahead time, and combine it with the current unit vector of travel and angular velocity to make a short-term prediction of the robot's pose: in the scenario that only considers the translation of the direction of travel, a constant velocity model can be used, that is, the look-ahead compensation amount is translated along the direction of travel of the robot reference point in the global coordinate system; in the scenario that considers angular velocity, the first computing unit can use a constant angular velocity model to predict the yaw angle increment, and update the robot's attitude in the global coordinate system accordingly.

[0079] The above predictions are conventional kinematic extrapolation methods in this field. In specific implementation, they can be completed by discrete updates, such as discretizing the look-ahead time into step sizes of 5ms to 20ms and performing numerical integration, or directly using yaw angle increments for equivalent updates when small angle approximation is valid. This application is not limited to the above extrapolation methods, and can also be implemented using a bicycle model or an IMU-based short-time integration model.

[0080] After obtaining the predicted pose at the look-ahead moment, the first computational unit maps the physical work envelope of the lateral actuator to the global coordinate system at the look-ahead moment based on the coordinate transformation relationships pre-stored in the workspace geometric model. The workspace geometric model includes the extrinsic parameter matrix from the lateral actuator base coordinate system to the body coordinate system, and the pose matrix from the body coordinate system to the global coordinate system; the first computational unit then performs a rigid body transformation on the boundary point set or enclosing volume parameters of the physical work envelope to obtain the "look-ahead work area".

[0081] To reduce computational complexity, the physical operation envelope can be represented using a convex polyhedron or a cuboid bounding box. For example, the envelope can be configured as an accessible space range of 0.3m to 1.0m horizontally and 0.4m to 1.5m vertically, and limited in the row direction to a preset length range covering the end-operation window. The specific values ​​can be determined by the actuator model, installation height, and height of the work object. The boundary representation of the envelope is not limited; it can also be represented using a gridded occupancy set or a point set envelope.

[0082] Subsequently, the first computational unit determines the imaging region corresponding to the look-ahead working area within the visual field of view. In implementation, the first computational unit can project the boundary points of the look-ahead working area onto the imaging plane based on the imaging model (intrinsic and extrinsic parameters) of the forward imaging device stored in the working space geometric model, obtaining a set of projected points. Projection can be achieved based on a standard pinhole camera model and homogeneous coordinate transformation, and the conventional projection calculation process of "3D point to pixel coordinates" can be directly adopted. If the look-ahead working area is represented by a polyhedron, its vertices can be projected and the convex hull of the projected point set can be calculated to obtain the corresponding polygonal region of the image; if the look-ahead working area is represented by a bounding box, the eight corner points of the bounding box can be projected and the circumscribed rectangle can be obtained accordingly. To ensure that the constraint of "determined within the visual field of view" holds, the first computational unit can intersect the above-mentioned projected area with the effective imaging range of the visual field of view on the imaging plane, and further crop it with the image boundary to form the final spatiotemporal preview slice. As an example, if the image resolution is 1280×720 or 1920×1080, the first calculation unit can limit the slice boundary to the pixel range and output an empty slice flag when the slice is too small or empty, so that subsequent steps can process it according to a preset strategy (e.g., skip the pre-aimed slice calculation of this frame or keep the slice parameters of the previous frame).

[0083] In an optional implementation, to improve real-time performance on the first computing unit side, the projection and intersection processes can employ a layered approximation. Specifically, when the working tunnel floor is approximately flat and the pitch angle of the forward imaging device changes little in a short time, the first computing unit can convert the projected boundary of the forward-looking working area on the ground reference plane into a two-dimensional polygon, and map the two-dimensional polygon onto the imaging plane based on a pre-calibrated planar homography matrix, thereby obtaining a spatiotemporal pre-aiming slice. The planar homography matrix can be calculated from the camera's extrinsic and intrinsic parameters and ground plane parameters, which is a conventional visual geometry method in the field. In implementation, the homography mapping process in commonly used vision libraries can be referenced without elaborating its mathematical derivation. To adapt to hardware implementation, homography mapping can be accelerated using fixed-point matrix multiplication and table lookup methods, and the matrix coefficients can be pre-stored in the configuration register or on-chip memory of the first computing unit.

[0084] Regarding parameter configuration, the look-ahead time can be of the same dimension as the inference time statistics and can be superimposed with a time margin to cover communication and scheduling overhead; for example, the margin can be configured in the range of 5ms to 30ms. The output format of the spatiotemporal preview slice can include a timestamp or frame number field to establish a consistent binding with the input image frame; for example, an index field consistent with the frame number output by the forward imaging device can be used, or a timestamp field under a unified clock source can be used, which facilitates subsequent modules to call the image data in frame alignment. In this way, the first computing unit can determine the spatiotemporal preview slice within the visual field of view based on the look-ahead compensation amount and the working space geometric model without introducing manual intervention, and use it as the direct basis for subsequent image data selection.

[0085] Regarding S104 above:

[0086] In an optional embodiment, after completing S103, the first computing unit obtains the spatiotemporal preview slice parameters corresponding to the current image frame, and performs cropping and transmission of the original image data accordingly. The original image data is single-frame image data or image stream data output by the forward imaging device, which can be a common progressive scan format, such as YUV420 (NV12 / NV21) or RGB / BGR format; the image resolution can be configured according to hardware capabilities, such as 1280×720, 1920×1080, etc. The spatiotemporal preview slice parameters are preferably represented by region parameters in pixel coordinate system, including at least the pixel coordinates of the upper left corner of the slice and the slice width and height, or including a set of vertex coordinates used to define the convex quadrilateral slice; at the same time, the spatiotemporal preview slice parameters and the original image data are bound by frame sequence number or timestamp, thereby ensuring that the first computing unit performs cropping of "which slice" on "which frame image" is logically determined.

[0087] In executing the cropping operation, the first computing unit can employ either a frame buffering method or a streaming method. Taking the frame buffering method as an example, the first computing unit first writes the raw image data output from the forward imaging device into a preset image frame buffer, which can be located in the external memory or off-chip memory of the first computing unit. Subsequently, the first computing unit calculates the starting address and row step size of the cropping region in the frame buffer based on the spatiotemporal pre-aiming slice parameters, and copies the pixel blocks corresponding to the cropping region to the cropping output buffer. The above copying can be implemented by the DMA controller in a two-dimensional transfer method, that is, continuous transfer by row and jump by row step size at the end of each row, in order to reduce the involvement of general logic and improve real-time performance. Common hash / aggregate DMA or two-dimensional DMA transfer mechanisms can be used for this purpose.

[0088] Taking the streaming method as an example, when the first computing unit receives the row-by-row pixel data of the image stream, it gates the pixels according to the row and column ranges of the slice, and only writes to the pixels that fall within the slice range. The remaining pixels are directly discarded or not put into the buffer, thereby completing the "receive and crop at the same time" streaming output on the side of the first computing unit. This implementation is suitable for scenarios where the first computing unit is directly connected to the imaging device and has row counting, column counting and row buffer resources.

[0089] To ensure robustness of the cropping process, the first computational unit preferably performs boundary normalization on the slice parameters before cropping. This includes cropping the slice coordinates to within the effective range of the image and handling abnormal cases where the slice width and height are zero or the slice falls completely outside the image range. If the cropping output uses DMA transfer, the first computational unit can also perform alignment processing on the slice width to meet bus or storage alignment constraints, such as rounding down or up to 8 pixels, 16 pixels, or 32 bytes, and updating the span parameter of the output buffer accordingly. This type of alignment is a conventional constraint processing and can be directly configured by the control program according to platform constraints. For NV12 equal-component planar format, the first computational unit can perform consistent ROI cropping on the luminance and chrominance components respectively. The cropping coordinates of the chrominance component can be converted according to the sampling rate relationship. The conversion rules are well-known image format processing methods in the art and will not be elaborated here.

[0090] In terms of transmission, the first computing unit sends the cropped image data along with necessary metadata to the second computing unit. The metadata may include at least the frame sequence number or timestamp, slice region parameters, and image format identifier, which the second computing unit uses to perform queue sorting and frame alignment processing on the receiving side. Transmission between the first and second computing units can be achieved using communication methods such as shared memory, PCIe, AXI bus, or Ethernet.

[0091] Taking shared memory as an example, the first computing unit writes the cropped output buffer to a preset circular queue and updates the write pointer; the second computing unit polls or triggers an interrupt to read the read pointer, retrieving the cropped image data and its metadata from the circular queue. Taking PCIe or AXI as examples, the first computing unit can write the physical address or descriptor of the cropped output buffer to a message queue and trigger a doorbell interrupt to notify the second computing unit to initiate a read; both the message queue and the doorbell mechanism are conventional heterogeneous communication methods and can be directly implemented using the driver interface provided by the platform.

[0092] Through the above process, the first computing unit can perform cropping of the original image data in accordance with the spatiotemporal pre-aiming slice during runtime, and send the cropped image data to the second computing unit in an aligned and frame-by-frame traceable manner, thus meeting the requirements of subsequent target recognition processing for input data consistency and real-time performance.

[0093] Regarding the above S105:

[0094] In one optional embodiment, after receiving the cropped image data sent by the first computing unit, the second computing unit writes it into the inference input buffer and reads the frame number, timestamp, and cropped region parameters carried in the packet for input alignment and result backtracking in the inference task. The processing flow of the second computing unit performing target recognition inference may include input preprocessing, model inference, and result postprocessing. Input preprocessing may include conventional operations such as image scaling, normalization, and channel rearrangement; for example, scaling the cropped image to a network input size of 224×224, 320×320, or 416×416, and performing normalization according to model requirements.

[0095] The model inference can be implemented using existing object detection or object recognition networks. Inference models based on convolutional neural networks or lightweight detection networks can be selected and executed in the inference acceleration environment of the second computing unit through inference runtime. For example, it can be deployed based on inference engines such as TensorRT, ONNX Runtime or OpenVINO. The model type and engine are only examples and do not constitute a limitation.

[0096] The second computational unit performs post-processing on the inference output to generate structured recognition results. The recognition results may include target category identifiers, confidence scores, and target location parameters. Target location parameters can be represented as bounding boxes with the coordinates of the top-left and bottom-right corners, or as the coordinates of the center point and width and height parameters. To ensure the usability of the recognition results, the second computational unit can perform threshold filtering on the confidence scores, with the threshold set between 0.3 and 0.7; and can perform non-maximum suppression on multi-target candidate results to remove overlapping candidate boxes, with the cross-union ratio (CUI) threshold set between 0.4 and 0.6.

[0097] After obtaining the identification result, the second computing unit generates a work instruction corresponding to the identified target and sends the work instruction to the control program or controller interface of the lateral actuator. The work instruction preferably adopts the form of a structured data packet, which includes at least the target category identifier, target position parameters, and frame sequence number or timestamp bound to the input frame.

[0098] Optionally, the work instruction also includes a work type identifier to instruct the lateral actuator to perform actions such as gripping, picking up, cutting, or transporting. If the control program of the lateral actuator requires coordinate input, the second calculation unit can further provide the target pose parameters in the execution coordinate system when generating the work instruction.

[0099] For example, the second calculation unit can restore the center point of the bounding box of the identified target to the pixel coordinates of the original imaging plane based on the cropping region parameters. Then, combining the imaging model and coordinate transformation relationship defined in the S101 work space geometric model, it performs intersection or projection transformation between the spatial ray corresponding to the pixel coordinates and the physical work space constraints to obtain the position parameters of the target in the robot base coordinate system or the actuator base coordinate system, and writes these position parameters into the work instruction. The above conversion from image coordinates to execution coordinates can be implemented using the standard pinhole imaging model and homogeneous transformation matrix, which is a conventional technique and will not be elaborated here.

[0100] When the lateral actuator performs a task according to the work instruction, the mobile robot control program reads the target category identifier and target position parameters from the work instruction, generates the action sequence of the lateral actuator, and sends it to the actuator drive interface. The action sequence can include sub-actions such as pre-alignment, extension, end effector, and retraction. The control variables can be pose or joint parameters, and the control cycle can be set from 5ms to 20ms to match the real-time requirements of the robot's operation. The motion control, gripping control, or cutting control of the actuator can be implemented using existing industrial control methods, such as position servo control, speed servo control, or force control / gripping threshold control, which are conventional control methods and will not be elaborated here. To avoid false triggering, the control program can also perform gating based on the confidence level or validity flag in the work instruction. When the confidence level is lower than the threshold or the recognition result is inconsistent with the input frame number, the work action is not generated and the work instruction is discarded.

[0101] Furthermore, during mobile robot operations, the target recognition inference latency fluctuates, and when the robot's posture undergoes slight yaw or serpentine correction, if the look-ahead quantity is represented solely by "time" or "body velocity modulus," it is easy for the compensation amount used for subsequent slice calculations to be inconsistent with the actual displacement along the direction of travel, thereby causing the locking area of ​​the spatiotemporal pre-aiming slice to shift or jitter.

[0102] Optionally, the look-ahead compensation amount is used to characterize the row displacement of the mobile robot moving along the planting row within a predetermined look-ahead time, which is determined based on the reasoning time estimate of the target recognition task by the second computing unit.

[0103] In an optional embodiment, the look-ahead compensation is a distance quantity used to characterize the expected displacement of the mobile robot along the direction of the structured planting row within a predetermined look-ahead time. The direction of travel can be given by the S101 workspace geometry model, for example, using the unit vector of the channel centerline direction as the unit vector of travel. The first computing unit samples motion state parameters during operation. The motion state parameters include at least the instantaneous linear velocity along the travel direction, and optionally include angular velocity; wherein, the instantaneous linear velocity can be directly converted from the wheel speed odometer and projected according to the unit vector of travel to obtain the travel velocity component, and the angular velocity can be obtained from the gyroscope output of the inertial measurement unit.

[0104] The predetermined look-ahead time is determined based on the inference time estimate of the target recognition task by the second computing unit. The timing boundary of the inference time can be uniformly defined as: from the moment the second computing unit begins to perform target recognition inference on a frame of input image data, until the moment the recognition result corresponding to that frame is written to the output buffer and marked as readable. As an example, the second computing unit records the inference start timestamp and inference end timestamp for each frame of input data, and the difference between the two is taken as the actual inference time for that frame; and statistically analyzes the actual inference time of the most recent several frames to obtain the statistical value of the inference time. The statistical method can be any of the following: moving average, exponentially weighted moving average, or quantile estimation. For example, the moving window length can be set to 4 frames, 8 frames, or 16 frames; the weight coefficient of the exponentially weighted moving average can be set to 0.1 to 0.3. The second computing unit sends the statistical value of the inference time along with its corresponding frame number or timestamp to the first computing unit, and the first computing unit determines the predetermined look-ahead time matching the current processing frame accordingly.

[0105] After determining the predetermined look-ahead time, the first computing unit calculates the look-ahead compensation. Optionally, the predetermined look-ahead time can be directly taken as the statistical value of inference time consumption; alternatively, a preset margin can be added to the statistical value of inference time consumption to cover data transmission and scheduling overhead, and the margin can be set to 5ms to 30ms. When the look-ahead compensation is used to characterize the directional displacement, the first computing unit can obtain it in any of the following ways: when only the instantaneous linear velocity of the directional displacement is used, the instantaneous linear velocity of the directional displacement is multiplied by the predetermined look-ahead time to obtain the look-ahead compensation; when there are velocity fluctuations within the sampling period, the first computing unit performs discrete integration on the directional velocity sequence within the time window covered by the predetermined look-ahead time to obtain the look-ahead compensation, and the integration step size can be consistent with the sampling period of the motion state parameters, for example, 5ms to 20ms. If angular velocity is collected simultaneously, the first calculation unit can update the heading change based on the angular velocity in conjunction with the heading direction in the operational space geometric model. Within each integration step, the body velocity is projected according to the angle between the current heading and the heading direction, thus obtaining a heading displacement prediction value that better matches the conditions of slight serpentine driving or steering correction. The look-ahead compensation amount calculated above is stored by the first calculation unit and used in subsequent steps.

[0106] In this way, the look-ahead compensation is defined as the displacement along the planting row direction, and the predetermined look-ahead time is determined by the inference time estimation. This makes the compensation consistent with the geometric direction of the working channel, which can stabilize the calculation of the look-ahead distance under the conditions of inference time delay fluctuation and slight steering correction. This improves the spatiotemporal alignment accuracy of the spatiotemporal pre-aiming slice and reduces the additional communication and computing load caused by expanding the clipping area to cover uncertainty.

[0107] In moving operation scenarios, target recognition involves time-consuming inference while the robot continuously moves along the path. If the clipping region is fixed or determined empirically only in the image domain, it can easily lead to a discrepancy between the clipping region and the moment the target enters the physical operation envelope of the lateral actuator, resulting in spatiotemporal pre-aiming slice offset, missed detection, or missed operation windows. Blindly expanding the clipping region or performing full-image inference to avoid this offset would significantly increase the data transmission volume between the first and second computing units and the inference load of the second computing unit.

[0108] Optional, see Figure 2 The flowchart of a method for determining spatiotemporal pre-aiming slices provided in this application embodiment includes steps S201 to S202, wherein:

[0109] S201: In the geometric model of the work space, the physical work envelope area is translated in the opposite direction of the movement direction of the mobile robot by the look-ahead compensation amount to obtain the translated work area;

[0110] S202: Project the translated work area onto the imaging plane of the forward imaging device, and the projected area serves as the spatiotemporal preview slice.

[0111] In one optional embodiment, the determined spatiotemporal pre-aiming slice is used to perform a reverse translation of the physical operation envelope region in the operation space geometric model in space according to the look-ahead compensation amount, and then map the translated spatial region onto the imaging plane of the forward imaging device to obtain a two-dimensional slice region that can be directly called for subsequent image cropping. The operation space geometric model pre-stores at least the following information: the pose transformation relationship between the global coordinate system and the forward imaging device coordinate system, the imaging model parameters of the forward imaging device, and the boundary representation of the physical operation envelope region in the global coordinate system. The boundary representation can adopt any one of the following: convex polyhedron vertex set, cuboid bounding box parameters, or meshed occupancy set; for the convenience of subsequent calculations, this embodiment uses "the envelope region is a convex polyhedron or a bounding box" as an example.

[0112] The first calculation unit first determines the unit vector of the motion direction. This motion direction can be given by the motion state parameters in S102: when only the instantaneous linear velocity of the heading is obtained, the unit vector of the channel centerline direction in the operational space geometric model can be directly used as the unit vector of the motion direction; when the angular velocity is obtained at the same time and slight yaw needs to be reflected, the first calculation unit can update the aircraft heading angle based on the angular velocity at the current moment, and use the unit vector of the updated heading direction in the global coordinate system as the unit vector of the motion direction. Subsequently, the first calculation unit records the look-ahead compensation as the distance d, and constructs the translation vector Δp ​​= -d·u, where u is the unit vector of the motion direction, and "-" indicates reverse translation. For the physical task envelope represented by a set of vertices, the first computation unit performs a translation operation on the coordinates p_i of each vertex in the envelope to obtain the translated vertex coordinates p_i′=p_i+Δp, thus obtaining the translated vertex set of the task area. For the physical task envelope represented by bounding box parameters, the first computation unit performs the same translation on the coordinates of the center point of the bounding box, while keeping the size parameters of the bounding box unchanged, thereby obtaining the translated task area. The look-ahead compensation amount d can be obtained from S102, and its typical range is related to the robot speed and inference time. For example, when the traveling speed is 0.3m / s to 1.5m / s and the inference time is 30ms to 200ms, the corresponding d can be in the range of 0.01m to 0.30m. The above values ​​are only used to illustrate the range, and the specific values ​​are determined by the actual system configuration and operating status.

[0113] After obtaining the translated work area, the first computing unit projects the translated work area onto the imaging plane of the forward imaging device to obtain a spatiotemporal preview slice. The projection process can be implemented based on the pinhole imaging model and homogeneous coordinate transformation: The first calculation unit first transforms the translated vertices of the working area from the global coordinate system to the camera coordinate system. The transformation relationship can be expressed as P_c=R·P_w+t, where P_w is a 3D point in the global coordinate system, P_c is a 3D point in the camera coordinate system, and R and t are provided by the working space geometric model. R is the rotation matrix from the global coordinate system to the camera coordinate system, and t is the translation vector from the global coordinate system to the camera coordinate system. R and t together constitute the extrinsic parameters of the forward imaging device relative to the global coordinate system. R and t can be obtained by the installation and calibration of the imaging device or given by the coordinate system association parameters in the working space geometric model. Then, the 3D points in the camera coordinate system are projected onto the imaging plane using the camera intrinsic parameter matrix K to obtain the pixel coordinates (u,v). The standard form (u,v,1)^T∝K·(x / z,y / z,1)^T can be used, where x, y, and z are the point coordinates in the camera coordinate system and z is the depth value. Where K is the camera intrinsic parameter matrix, used to characterize the focal length, principal point position, and pixel scale; the symbol "∝" represents the proportional equivalence relationship under homogeneous coordinates, that is, the two vectors differ by a non-zero scaling factor. Thus, the calculation form of pixel coordinates (u,v) can be obtained, for example, u=f_x·(x / z)+c_x, v=f_y·(y / z)+c_y, where f_x and f_y are the focal length per pixel, c_x and c_y are the principal point coordinates; z is the depth value, used to normalize the 3D points in the camera coordinate system to the imaging plane. In an optional implementation, the coordinate transformation can also be represented by a homogeneous transformation matrix T, where T is a 4×4 matrix, to facilitate unified composite operations of rotation and translation. After the first calculation unit performs the above projection on multiple boundary points of the translated working area (e.g., vertices of a polyhedron or the 8 corner points of a bounding box), it obtains a set of pixel points. The "projection region" can be determined in any of the following ways: First, calculate the two-dimensional convex hull of the pixel set, and use the convex hull polygon as the projection region; Second, take the minimum / maximum pixel coordinates of the pixel set in the horizontal and vertical directions to obtain an axis-aligned rectangle (u_min, u_max, v_min, v_max) as the projection region, where (u_min, u_max, v_min, v_max) are the boundary parameters of the axis-aligned rectangle of the projection region in the pixel coordinate system, u_min and u_max represent the minimum / maximum horizontal pixel coordinates of the pixels covered by the projection region, and v_min and v_max represent the minimum / maximum vertical pixel coordinates, respectively; the rectangle can be obtained by taking the extreme values ​​of the pixel set in the two directions and used to limit the clipping window; Third, rasterize the projection polygon into a pixel mask, and use the pixel set within the mask as the projection region.For the convenience of subsequent direct cropping in S104, in this embodiment, an "axis-aligned rectangular box" can be selected as the output form of the spatio-temporal preview slice, and (u_min, u_max, v_min, v_max) are written into the shared buffer or the register mapping area for the image cropping logic to read. At the same time, boundary cropping is performed on the rectangular box to ensure that it falls within the valid range of the image. For example, when the image resolution is 1280×720, the constraints are 0 ≤ u_min < u_max < 1280 and 0 ≤ v_min < v_max < 720. In an optional implementation, the cropping window is defined as [u_min, u_max) × [v_min, v_max) in a semi-open interval to avoid double counting of boundary pixels.

[0114] Through the above calculations of reverse translation and projection, the spatio-temporal preview slice is defined as the two-dimensional corresponding area of the translated working area on the imaging plane, so that this slice corresponds in the time sense to the spatial range that will enter the physical working envelope area after a predetermined preview time, and in terms of data form, it can be directly called by the first computing unit for subsequent image cropping processing.

[0115] In this way, the physical working envelope area is translated backward by the preview compensation amount along the movement direction and projected onto the imaging plane to determine the slice, so that the slice corresponds in definition to the spatial target of "the area that will enter the working envelope area after a predetermined preview time", thereby improving the spatio-temporal alignment accuracy between the perception result and the execution position. Since the slice range is determined by the projection of the working envelope area, the cropping area can be compressed on the premise of meeting the working requirements, reducing the communication bandwidth occupancy and the inference calculation load.

[0116] In an optional implementation manner, the first computing unit is an FPGA unit and the second computing unit is an NPU unit, mainly aiming at the technical problem of "uncontrollable deterministic delay and bandwidth occupancy before the perception data enters the inference link" under the heterogeneous computing architecture. If the first computing unit uses a general-purpose processor to perform pre-processing such as image acquisition, timestamp alignment, slice projection calculation, and cropping and transfer, it is easy to cause fluctuations in the pre-processing delay due to operating system scheduling jitter, cache misses, or bus contention, so that the calculation basis of the preview compensation amount and the spatio-temporal preview slice is unstable, and it is further amplified into the drift of the slice locking area or the conservative strategy of needing to expand the cropping area, increasing the communication and inference load.

[0117] Specifically, when the first computing unit uses an FPGA unit, the FPGA unit can be directly connected to the forward imaging device through a hardware interface. For example, it receives the image stream through MIPI-CSI, LVDS, or a parallel pixel bus, and completes the geometric calculations and data cropping operations related to S101, S103, and S104 in a pipelined manner on the chip.

[0118] For example, the FPGA unit can be configured with a line buffer and block RAM to achieve real-time cropping line by line. The region of interest corresponding to the spatiotemporal pre-targeting slice is written to the cropping control register in the form of pixel coordinate range or mask. When the image resolution is 1280×720 or 1920×1080 and the frame rate is 20fps to 60fps, the FPGA unit can complete cropping and packaging without buffering the entire frame, and write the cropped image block to the external transmission buffer via DMA. The communication link between the FPGA unit and the second computing unit can be PCIe, AXI interconnect, gigabit / 10 gigabit Ethernet, etc. The cropped image block can be accompanied by a frame sequence number and timestamp field for subsequent inference time statistics and timing alignment.

[0119] When the second computing unit uses an NPU unit, the NPU unit is used to perform target recognition inference in S105 and output the inference time statistics and recognition results.

[0120] For example, the NPU unit can be a SoC module with a neural network inference accelerator or a standalone NPU accelerator card. During runtime, it loads the inference engine of the target recognition network (e.g., based on TensorRT, ONNX Runtime, or OpenVINO runtime), preprocesses the slice data from the FPGA unit according to a fixed input size, and then performs inference. The slice input size can be determined according to the model configuration, such as 224×224, 320×320, 640×640, etc. The NPU unit records timestamps at the start and end of each frame of inference to form the inference time for that frame, and performs moving average, exponential weighted moving average, or quantile statistics on the most recent frames to obtain the inference time statistics. The statistics are then sent back to the FPGA unit to support the look-ahead compensation calculation of S102.

[0121] This approach allows image acquisition, spatiotemporal pre-aiming slice calculation, and cropping to be completed on the hardware pipeline with deterministic latency, while target recognition is performed on dedicated inference hardware. This reduces the disruption to the inference link caused by preprocessing and bus transmission, and lowers the communication and computational load caused by expanding the cropping area to combat latency uncertainties. Furthermore, since the first computing unit is directly connected to the imaging device and can perform cropping and packaging at the hardware level, it reduces external memory bandwidth usage and improves the overall real-time performance and stability of the system.

[0122] Optional, see Figure 3 The flowchart below illustrates a method for controlling a lateral actuator to perform an operation, as provided in this application embodiment, including steps S301 to S302, wherein:

[0123] S301: Based on the work space geometric model, determine that the identified target is located within the physical work envelope area;

[0124] S302: In response to the determination that the result is within the physical operation envelope area, the lateral actuator is triggered to execute the operation instruction.

[0125] During mobile robot operations, target recognition results may include targets located within the visual field of view but beyond the reach of the lateral actuators. If execution is triggered without constraints, invalid actions, collision risks, or idle travel of the actuators can easily occur, reducing cycle stability and increasing wear on the mechanisms. By determining the consistency of the spatial relationship between the identified target and the physical operation envelope before triggering, erroneous triggering of unreachable targets can be avoided, and the executed actions can be kept consistent with the mechanism's reachability boundaries, thereby improving the reliability and controllability of operation triggering.

[0126] In an optional embodiment, after the second computing unit completes target recognition in S105, it outputs spatial positioning metadata related to the identified target while outputting the work instruction. The metadata includes at least pixel region information of the target in the imaging plane and the corresponding frame number or timestamp. The pixel region information can be the pixel coordinates of the upper left and lower right corners of the bounding box, or a set of pixels of the target outline. After receiving the work instruction, the first computing unit first maps the pixel region to the target spatial position in a unified coordinate system based on the work space geometry model, then determines whether the target spatial position falls within the physical work envelope of the lateral actuator, and finally triggers the actuator to execute the work instruction only when it is determined that the target position falls within the physical work envelope of the lateral actuator.

[0127] Specifically, the first calculation unit can select representative points from the pixel region as positioning points, such as taking the pixel coordinates (u0, v0) of the bounding box center point or the pixel coordinates of the centroid of the contour; and convert the pixel coordinates into a normalized line-of-sight vector in the camera coordinate system based on the intrinsic parameter matrix of the forward imaging device. To obtain the target's position in three-dimensional space, the first calculation unit can obtain depth or equivalent distance information in any of the following ways: When the imaging device is a depth camera or a binocular camera, the depth value corresponding to the representative point is obtained using a depth map or disparity matching; when the environment is a structured planting row and the target is mainly distributed near a known crop strip or shelf sidewall, the intersection of the representative point's line of sight and the geometric constraint is determined using the pre-configured "target distribution geometric constraint" in the work space geometric model as the target position. For example, the crop strip is approximated as a plane or strip-shaped curved surface parallel to the row direction and fixed laterally offset from the center line of the channel, and the intersection of the line of sight and the plane or curved surface is obtained; when the system integrates a 4D radar or other ranging sensing unit, the distance information output by it can be associated with the pixel area through timestamps and extrinsic parameter calibration to obtain the equivalent depth of the representative point.

[0128] Among the aforementioned depth acquisition methods, disparity matching, depth map reading, ray-plane intersection, and multi-sensor timestamp association are all conventional implementation methods in this field and can be completed by the first computing unit calling the corresponding algorithm module.

[0129] After obtaining the target's three-dimensional coordinates in the camera coordinate system, the first computing unit uses the external parameter transformation relationships stored in the operational space geometric model to transform the target coordinates to a unified expression in the global coordinate system or the machine coordinate system. The physical operational envelope can be represented in the operational space geometric model as a polyhedron, bounding box, or set of inequality constraints; for example, with the machine coordinate system as a reference, the envelope can be defined by the horizontal range [xmin, xmax], the horizontal range [ymin, ymax], and the vertical range [zmin, zmax]. Each boundary is determined by the structural parameters of the actuator and the installation pose. The boundaries can be configured according to the mechanism specifications as several reachable ranges ranging from tens of centimeters to one meter. xmin and xmax correspond to the minimum and maximum values ​​of the horizontal range, respectively, and the meanings of other parameters are deduced accordingly. When the first calculation unit performs envelope determination on the target coordinates, it can use a point-in-box or point-in-polytope determination algorithm: when the envelope region is an axis-aligned bounding box, it directly compares whether the target coordinate components fall within the boundary interval of each component; when the envelope region is a convex polyhedron, it calculates the target coordinates and substitutes the signs of each half-space inequality and determines whether it is located inside the polyhedron. This determination algorithm can use either the half-space intersection determination or the ray method.

[0130] When the determination result indicates that the target is located within the physical operation envelope, the first calculation unit triggers the lateral actuator to execute the operation command. The triggering method can be to send an "execution allowed" flag and target parameters to the actuator controller, and the actuator controller will complete the grasping, clamping or shearing actions according to the preset action sequence; the target parameters may include the target's expected end pose in the body coordinate system, the target category identifier and the corresponding frame number.

[0131] In addition, to avoid boundary jitter caused by attitude perturbations during travel, a safety margin can be introduced in the envelope determination, that is, the boundary of the envelope area is contracted inward by a preset margin before determination. The margin can be set to 10mm to 50mm to improve the robustness of the trigger. When the determination result is that it is not within the envelope area, the first calculation unit can discard the operation instruction or mark it as unexecutable without triggering the execution mechanism, so as to ensure that the execution action is consistent with the accessibility of the mechanism.

[0132] In an optional embodiment, to address the issue of slippage and structural vibration occurring when the mobile robot operates within a work lane defined by a multi-layered, three-dimensional planting rack, leading to deviations between odometry-based displacement prediction and instantaneous imaging attitude, and consequently causing offsetting of the spatiotemporal pre-aiming slice locking area, this embodiment introduces joint compensation for jitter and slippage during the determination of the spatiotemporal pre-aiming slice. By simultaneously utilizing high-frequency information from the chassis motor drive current and environmental observation information from 4D radar point clouds within a predetermined look-ahead time window, the geometric model of the work space is dynamically corrected. This improves the spatiotemporal alignment stability of the slices without expanding the clipping area, thereby reducing additional communication and inference load.

[0133] Specifically, when the mobile robot travels along the direction of travel within the working tunnel, the first computing unit simultaneously collects two data streams and performs time alignment within a time window corresponding to a predetermined look-ahead time. One example is that the first computing unit samples the phase current or bus current output by the chassis motor drive controller to form a high-frequency sequence of drive current, and adds a sampling timestamp to the sequence; simultaneously, it receives a point cloud stream from a 4D radar sensing unit associated with the forward imaging device and adds a frame timestamp. The sampling frequency of the drive current can be set to 5kHz to 20kHz, and the point cloud frame rate of the 4D radar can be set to 10Hz to 30Hz; time alignment can be achieved using a unified clock source or a timestamp conversion method, with an allowable alignment error controlled within 5ms. The point cloud stream here can contain the target point's distance, azimuth, pitch, and Doppler velocity information. Since the working tunnel is defined by a multi-layered, three-dimensional planting frame, the point cloud typically contains stable echo components representing the tunnel's spatial structure, facilitating subsequent observation of relative motion to the ground or environment.

[0134] After obtaining the synchronization data, the first calculation unit determines the instantaneous jitter offset vector of the forward imaging device based on the high-frequency sequence of the driving current. The jitter offset vector is used to characterize the instantaneous disturbance of the imaging device relative to its nominal mounting pose at the moment of imaging. It can be represented as a pixel offset on the imaging plane, or as a small-angle attitude disturbance or a small displacement disturbance in the imaging device coordinate system.

[0135] The nominal installation pose is the reference pose determined during the installation calibration phase and used as the external parameters R and t of the workspace geometric model.

[0136] One optional implementation involves the first computing unit performing frequency domain or time-frequency domain analysis on the high-frequency sequence of the driving current to extract current ripple features, and converting these features into a jitter offset vector through a preset mapping relationship. The time-frequency domain analysis can use wavelet packet decomposition or short-time Fourier transform as examples, and the mapping relationship can use a pre-calibrated transfer function or regression model of the current and end-point jitter as examples. To ensure that the jitter estimation is consistent with the imaging time, the first computing unit can select the current subsequence closest to the imaging frame within a time window based on the imaging frame timestamp. The subsequence length can be set to 20ms to 80ms to cover the main structural vibration cycle.

[0137] Simultaneously, the first computational unit determines the sliding pose compensation amount of the mobile robot along the travel direction based on the point cloud flow and the encoder speed of the chassis motor. The encoder speed is used to characterize the nominal rolling speed of the wheel system, but it may deviate from the actual ground speed under conditions such as high humidity or uneven ground. The sliding pose compensation amount is used to correct the travel displacement error caused by the accumulation of this deviation within a predetermined look-ahead time. One example is that the first computational unit extracts a set of points related to the spatial structure of the alley or the relative stability of the ground from the point cloud flow, constructs an environmental reference velocity observation based on the Doppler velocity statistics of these points, and aligns and compares the environmental reference velocity observation with the encoder speed to obtain the sliding velocity difference. Then, it integrates the sliding velocity difference within a window corresponding to the predetermined look-ahead time to obtain the sliding pose compensation amount. The velocity alignment can use cross-correlation or least squares alignment as examples, and the integration can use discrete integration with an integration step size of 5ms to 20ms, consistent with the encoder speed update cycle. The slip pose compensation amount can be expressed as a heading displacement correction amount, and can also be extended to a pose correction amount that includes a small heading correction if necessary. However, the heading displacement correction amount should at least be able to be used for heading alignment in subsequent slice determination.

[0138] After obtaining the instantaneous jitter offset vector and the slip pose compensation amount, the first calculation unit corrects the visual field of view region corresponding to the work lane in the work space geometric model, and determines the spatiotemporal pre-aiming slice after jitter and slip compensation based on the corrected work space geometric model. Specifically, the first calculation unit can update the extrinsic parameters or projection parameters related to the imaging device in the work space geometric model to convert the instantaneous jitter offset vector into an instantaneous correction of the imaging device pose; at the same time, it updates the directional displacement prediction amount related to the robot motion to superimpose or cancel the slip pose compensation amount into the directional displacement corresponding to the look-ahead compensation amount, so that subsequent slice calculations use the corrected camera projection relationship and the corrected directional displacement prediction. On this basis, according to the determination process of S103, the first calculation unit re-locks the image region corresponding to the physical work envelope region within the visual field of view region after a predetermined look-ahead time, as the spatiotemporal pre-aiming slice after jitter and slip compensation, and writes the pixel boundary parameters of the slice into the cache for subsequent steps to call.

[0139] In one optional embodiment, the first computing unit aligns the high-frequency sequence of the driving current using the image frame as a time reference. Specifically, the first computing unit obtains the imaging time information corresponding to each frame of the image from the forward imaging device (e.g., the correspondence between the frame number and the frame acquisition time, or the time stamp carried in the frame header), and selects the current segment corresponding to the imaging time of that frame in the high-frequency sequence of the driving current as an analysis window. The analysis window can be set as a "symmetrical window centered on the imaging time" or a "preceding window ending at the imaging time" to cover several vibration cycles of the mechanical structure within the resonant frequency band. As an example, the driving current sampling frequency can be set to 5kHz to 20kHz, and the analysis window duration can be set to 40ms to 120ms; if there is a fixed delay between the exposure and readout of the imaging device, the first computing unit can compensate for the imaging time according to a preset delay amount before truncating the current segment, thereby improving the consistency between the current segment and the imaging moment.

[0140] After window capture, the first computational unit performs wavelet packet transform on the current segment to extract the current ripple component that matches the mechanical resonance frequency band. Specifically, the first computational unit performs wavelet packet decomposition on the current segment at a preset number of levels to obtain a set of coefficients corresponding to multiple sub-band nodes, and maps each node to the corresponding frequency sub-band according to the sampling frequency and decomposition level. The mechanical resonance frequency band can be given by preset parameters, such as 10Hz to 60Hz, 20Hz to 80Hz, etc., as selectable frequency band ranges. The first computational unit selects the set of nodes whose frequency sub-bands overlap with the resonance frequency band from the wavelet packet decomposition results, and reconstructs or converges the energy of the selected nodes to obtain the current ripple component. Wavelet packet transform can be directly implemented using existing signal processing libraries. For example, in the Python environment, the WaveletPacket interface of the PyWavelets library can be called to complete the decomposition and reconstruction (e.g., using the "db4" or "sym4" wavelet basis), and in the Matlab environment, wpdec and wprcoef can be called to achieve decomposition and reconstruction. The above functions are only examples, and equivalent wavelet packet filter banks can also be used to implement them.

[0141] After obtaining the current ripple component, the first computational unit extracts ripple characteristic quantities for mapping. For example, the ripple characteristic quantities can be at least one of the root mean square value, peak-to-peak value, energy value, or envelope mean value of the current ripple component within the analysis window. To reduce the impact of transient spikes on the estimation, the first computational unit can first perform full-wave rectification on the current ripple component and use low-pass filtering to obtain the envelope curve, and then extract statistics from the envelope curve; this rectification and filtering are conventional digital signal processing techniques and can be implemented using a first-order low-pass or finite impulse response filter.

[0142] Subsequently, the first calculation unit calls a preset current-end jitter transfer function to map the ripple characteristic or current ripple component to the instantaneous jitter offset vector at the imaging moment. The current-end jitter transfer function is used to characterize the mapping relationship between "current ripple characteristics" and "imaging device jitter," and it can be represented as any one of a discrete gain matrix, a discrete-time transfer function, or a set of finite impulse response coefficients.

[0143] For example, if the jitter offset vector is represented by pixel offset in the imaging plane, the first calculation unit can map the ripple feature quantity into horizontal and vertical pixel offsets according to a preset gain relationship; if the jitter offset vector is represented by camera attitude disturbance, the first calculation unit can map the ripple feature quantity into instantaneous yaw and pitch disturbances, and convert the attitude disturbances into pixel offsets on the imaging plane based on the imaging device intrinsic parameters. If the transfer function is given in the form of a discrete filtering model, the first calculation unit can perform discrete convolution or recursive calculation on the current ripple component to obtain a jitter response value aligned with the imaging time, and take this response value as the instantaneous jitter offset vector.

[0144] Finally, the first calculation unit writes the instantaneous jitter offset vector into a data structure associated with the current image frame. The data structure includes at least the frame number, the imaging time marker, and the instantaneous jitter offset vector, so that it can be directly called when performing corrections on the projection area of ​​the imaging plane or the spatial geometric model.

[0145] In one optional embodiment, a slip pose compensation amount based on point cloud flow and encoder speed is introduced to address the slip error problem that easily occurs in wheel speed odometers under conditions such as high humidity, mulch, water accumulation, or soft ground, where "the wheel rotates but the vehicle does not move forward proportionally." This slip error causes the predicted directional displacement obtained from motion state parameters to deviate from the actual ground displacement, resulting in a systematic drift of the locking position of subsequent spatiotemporal pre-aiming slices. By using the multi-layer three-dimensional planting rack columns in the working tunnel as absolute environmental references, an environmental reference velocity is constructed in the point cloud flow and cross-correlated with the encoder speed for verification. This can suppress slip error without relying on ground texture or lighting conditions. Furthermore, integrating the obtained slip coefficient sequence into a pose compensation amount can stabilize the directional displacement prediction and reduce the additional transmission and inference load caused by the expansion of the clipping area due to uncertainty.

[0146] Specifically, the first calculation unit continuously acquires point cloud streams from the 4D radar sensing unit within a time window corresponding to a predetermined look-ahead time. The point cloud stream may contain information such as the distance, azimuth, elevation angle, and Doppler velocity of each echo point. Based on the elevation angle information, the first calculation unit selects a height band in the vertical direction corresponding to the height range of the multi-layer three-dimensional planting rack columns, and extracts horizontal slices of point cloud distributed along the planting row direction within this height band from the point cloud stream. As an example, the height band may be set to 0.3m to 1.8m above the ground to filter out ground clutter (e.g., below 0.2m) and echoes from the top facilities (e.g., above 2.0m); the upper and lower limits of the height band may be given by the offline configured dimensions of the planting rack columns, or adaptively updated by the point cloud height distribution statistically analyzed during operation.

[0147] After obtaining the horizontal slice point cloud, the first computing unit performs Doppler velocity field analysis on it to extract environmental reference velocities that can be used to characterize ground motion. For example, the first computing unit can first identify column echo clusters distributed within a preset spacing range along the working roadway in the horizontal slice point cloud: the point cloud locations can be converted from radar polar coordinates to local rectangular coordinates with the roadway as the axis, then clustering is performed on the point cloud to obtain multiple echo clusters, and the centroid position and velocity statistics of each echo cluster are calculated; clustering can be implemented using Euclidean distance-based clustering or density-based clustering methods, such as calling EuclideanClusterExtraction in the PCL environment, calling the DBSCAN interface of the sklearn library in the Python environment, and using a distance threshold-based clustering function in the Matlab environment. The above are just examples. To meet the constraint of "approximately fixed spacing distribution", the first calculation unit can project the centroids of each echo cluster along the row direction and sort them, calculate the row spacing of adjacent centroids and check the consistency with the preset column spacing; the column spacing can be given by the scene configuration, for example, 0.8m to 1.5m, and the allowable error can be set to ±10% to ±25%, thereby eliminating echo clusters formed by random obstacles or non-structural components.

[0148] After identifying the pillar echo clusters, the first calculation unit constructs an environmental reference velocity matrix based on the pillar echo clusters. For example, at each sampling time, the first calculation unit can aggregate the Doppler velocity of each pillar echo cluster, such as by taking the mean, median, or weighted mean, and combine this with the azimuth and elevation relationship of the echo cluster relative to the radar to convert the Doppler velocity into velocity components along the tunnel direction; subsequently, the environmental reference velocity matrix is ​​organized with "echo cluster index" as rows and "sampling time" as columns. To improve robustness, the first calculation unit can smooth the velocity sequence of each row in the time dimension, such as by using a moving median or first-order low-pass filtering, and impose consistency constraints on the velocity components of multiple clusters in the spatial dimension, such as removing abnormal clusters that deviate from the median by more than a preset threshold.

[0149] Subsequently, the first calculation unit obtains the encoder speed of the chassis motor and performs a cross-correlation operation with the environmental reference speed matrix to generate a nonlinear slip coefficient sequence. For example, the first calculation unit can first aggregate the environmental reference speed matrix along the cluster dimension to obtain an environmental reference speed sequence, such as taking the median of each column to form a time series, and then perform a cross-correlation operation with the encoder speed sequence under the same time reference to determine the optimal alignment delay. The cross-correlation operation can be implemented directly by calling existing numerical libraries, such as numpy.correlate or scipy.signal.correlate in Python, and xcorr in Matlab; the above are just examples. After alignment, the first calculation unit can generate slip coefficients based on the difference or ratio between the encoder speed and the environmental reference speed, and input them into a preset nonlinear mapping function to obtain a nonlinear slip coefficient sequence. The nonlinear mapping function can be a piecewise linear function, a saturation function, or a sigmoid function, used to suppress noise amplification in the low-speed range and limit the coefficient range in the high-speed range.

[0150] After obtaining the nonlinear slip coefficient sequence, the first computational unit performs time-domain integration along the planting row direction to obtain the slip pose compensation amount.

[0151] For example, the first computing unit can perform trapezoidal integration on the sequence of "encoder velocity multiplied by slip coefficient" within a time window covered by a predetermined look-ahead time, using the sampling period of the motion state parameters as the integration step size, to obtain the row displacement compensation amount. The integration can be implemented through conventional numerical integration methods, such as calling numpy.trapz in the Python environment or performing cumulative integration on the sequence in the Matlab environment, and calling cumtrapz in the Matlab environment. The above are just examples. The slip pose compensation amount can be represented as the displacement correction amount along the row direction, or further combined with angular velocity to form a pose compensation amount including translation and heading corrections, and stored for subsequent use in the correction process of the working space geometry model.

[0152] In an optional embodiment, to ensure that the process of estimating the instantaneous jitter offset vector of the forward imaging device based on motor current ripple has sufficient physical basis and adapts to the actual installation stiffness and damping characteristics of the corresponding working roadway in the structured planting row, a transfer function between current and end-effector jitter can be established offline during the static calibration phase. By applying controlled excitation to the lateral actuator at a typical working roadway location and simultaneously acquiring the end-effector displacement attenuation curve and the motor current response sequence, a mapping model between the two can be obtained using a system identification algorithm, and the parameters of this model are embedded within the first computing unit. During operation, the first computing unit can directly call this transfer function to convert the real-time current ripple into a jitter offset vector in the camera coordinate system, thereby improving the accuracy and stability of jitter compensation.

[0153] Specifically, a static calibration process can be performed when the mobile robot is stationary at a typical work aisle position corresponding to the structured planting row. A typical work aisle position refers to a location where the robot is situated in a representative aisle section, such as a position where the aisle width and planting rack height are consistent with the normal working area, so that the structural vibration characteristics match the operating conditions during operation. During calibration, the lateral actuators can be extended to one of several preset working postures, such as extending to a position corresponding to the depth of the middle layer or planting tray. A calibration plate with obvious feature points, such as a checkerboard pattern or a rigid calibration plate with a dot array, is placed within the field of view of the forward imaging device, positioning the calibration plate in the working area near the end of the lateral actuator. The first computing unit controls the chassis to remain stationary, stops wheel speed output, and shuts down other actuators that might introduce vibration, retaining only the lateral actuators for excitation.

[0154] During the static calibration phase, the first computing unit applies a step excitation signal to the lateral actuator via the control interface. For example, in servo drive position control mode, a small step change can be applied to a given joint position, or in torque control mode, a step torque can be applied. The step amplitude can be configured according to the mechanism stiffness and safety limits, for example, the joint position step angle cannot exceed 5% to 10% of its stroke. Simultaneously with issuing the step command, the first computing unit controls the forward imaging device to switch to a higher frame rate operating mode (e.g., 50 frames / second or 100 frames / second), continuously acquiring image sequences containing the calibration board. To obtain the displacement attenuation curve, the first computing unit can extract feature points on the calibration board in each frame and calculate the pixel displacement of these feature points relative to the initial frame. Further, combining camera intrinsic and extrinsic parameters, the pixel displacement is converted into a translation sequence in the global coordinate system or the camera coordinate system. Feature point extraction and tracking can be implemented using existing image processing libraries. For example, in a C++ or Python environment, the feature detection and optical flow tracking interfaces of the OpenCV library can be called, such as findChessboardCorners or goodFeaturesToTrack combined with functions like calcOpticalFlowPyrLK. In a Matlab environment, relevant feature detection and optical flow functions can be called, which will not be elaborated on here. By averaging or projecting the multi-point displacement along the principal direction, the end displacement curve decaying over time can be obtained. This curve reflects the modal vibration response of the lateral actuator in this posture.

[0155] Synchronous with image acquisition, the first computing unit also needs to record the motor current response sequence caused by the step excitation. Specifically, the first computing unit can acquire the phase current or DC bus current of the target joint motor at a sampling frequency higher than the image frame rate through the analog acquisition channel or the current feedback interface provided by the driver. For example, the sampling frequency can be set in the range of 1kHz to 10kHz, and a precise timestamp can be attached to each sampling point. The timestamp can be aligned with the image acquisition clock, either by the first computing unit uniformly issuing a time reference to the imaging device and the current acquisition module, or by recording and converting the sampling time at the software level. After acquisition, the first computing unit can align the current response sequence and the displacement decay curve to a unified time axis through interpolation or resampling, providing synchronized input and output data for subsequent system identification.

[0156] After obtaining the motor current response sequence and the corresponding end displacement decay curve, the first calculation unit can establish a mapping model between the two using a system identification algorithm. The mapping model can take the form of a linear time-invariant system, such as a discrete transfer function, state-space model, or impulse response model, treating the current signal as the system input and the end displacement as the system output.

[0157] For example, in the Matlab environment, the System Identification Toolbox can be used to select a class of second-order or higher-order linear models based on the collected input and output data, and functions such as tfest, ssest, or arc can be called to estimate the model parameters. In the Python environment, the Control Systems Library or the signal processing module in SciPy can be used to fit the state-space model or impulse response model based on least squares or subspace identification algorithms. These functions and algorithms are all existing mature tools, and parameter selection can be based on a trade-off between fitting error and model complexity. In some implementations, the above calibration process can be executed separately for calibration attitudes with different extension strokes or different layer depths to obtain multiple sets of transfer functions, which can then be organized into a stroke-stiffness gain scheduling table; such extensions can be configured together with the aforementioned modal vibration component extraction section.

[0158] The mapping model obtained through system identification can be regarded as a transfer function between current and end jitter. The first computing unit can extract key parameters of the model from the identification results, such as the numerator and denominator coefficients of the discrete transfer function, the system matrix of the state-space model, and the coefficient vector of the impulse response sequence. These parameters are written into the on-chip storage resources of the first computing unit in the form of a coefficient table, serving as the current-end jitter transfer function during operation. If the first computing unit adopts an FPGA or SoC platform, the above parameters can be configured in the on-chip RAM, and the corresponding difference equations or convolution operations can be implemented in the hardware logic to realize the online transformation of the current signal to the jitter offset. During operation, when the first computing unit extracts the current ripple component that matches the mechanical resonance frequency band from the high-frequency sequence of the driving current, it can call the transfer function to perform forward inference, mapping the current ripple to the instantaneous jitter offset vector of the forward imaging device at the imaging moment, thereby maintaining the consistency and reliability of jitter estimation in complex working tunnels and multi-stiffness conditions.

[0159] For example, the current-end jitter transfer function can be implemented using a second-order difference equation: taking the current ripple component sequence as input and the jitter displacement observation as output, the output value is recursively calculated according to a fixed sampling period, and the output value is used as a scalar representation of the imaging jitter; then, combined with the camera mounting direction, the scalar representation is converted into a two-dimensional offset vector. For example, the difference equation can be written as "the current output is a weighted superposition of the previous two outputs, plus the weighted terms of the current and historical inputs", and its coefficients are identified by the above system and fixed in the first calculation unit; to facilitate the explanation of the implementation, a set of example coefficients can be used for the fixed structure, for example, the output recursion coefficients can be 1.6 and -0.65, and the input weighting coefficients can be 0.002, 0 and -0.002, to reflect the dynamic response characteristics under light damping resonance; this combination of coefficients is only used to illustrate the implementation of the difference recursion, and the actual coefficients are subject to the calibration and identification results. The direction of the two-dimensional offset vector can be determined by the main direction of the corner displacement during calibration, or it can be preset to be consistent with the horizontal axis of the camera. When it is necessary to characterize the horizontal and vertical jitter separately, two sets of parallel transfer function parameters can be established to output the horizontal and vertical components respectively.

[0160] The parameter storage of the mapping model may include the recursive coefficients of the difference equation, input and output scaling factors, sampling period, and effective frequency band range. The first computing unit can write these parameters into on-chip storage resources and perform recursive updates according to the sampling period during runtime. If the first computing unit is implemented using an FPGA, the above coefficients can be quantized into a fixed-point format and recursive calculations can be performed through a multiply-accumulator array. The fixed-point format and quantization bit width can be set according to the current sampling amplitude and jitter range, for example, using 16-bit or 24-bit fixed-point representation to balance accuracy and resource consumption.

[0161] Based on the same inventive concept, this application also provides a mobile robot task scheduling system based on a heterogeneous computing architecture, corresponding to a mobile robot task scheduling method based on a heterogeneous computing architecture. Since the principle of the system in this application is similar to the mobile robot task scheduling method based on a heterogeneous computing architecture described above, the implementation of the system can refer to the implementation of the method, and the repeated parts will not be described again.

[0162] Reference Figure 4 The diagram shown is a schematic of a mobile robot task scheduling system based on a heterogeneous computing architecture provided in an embodiment of this application. The system includes:

[0163] The construction module 10 is used to construct a work space geometric model on a mobile robot equipped with a forward imaging device and a lateral actuator, based on the geometric structure of a structured planting row. The work space geometric model represents the spatial correspondence between the visual field of view of the forward imaging device and the physical work envelope of the lateral actuator.

[0164] Processing module 20 uses a first computing unit to obtain motion state parameters of the mobile robot, and determines a look-ahead compensation amount based on the motion state parameters and the statistical value of the inference time of the second computing unit for the target recognition task. The first computing unit communicates with the forward imaging device, and the second computing unit communicates with the first computing unit. The motion state parameters include instantaneous linear velocity and / or angular velocity along the planting row direction.

[0165] The first computing unit uses the look-ahead compensation amount and the workspace geometric model to determine a spatiotemporal preview slice in the visual field of view;

[0166] The first computing unit uses the acquired raw image data to crop out the image data located within the spatiotemporal preview slice, and sends the cropped image data to the second computing unit;

[0167] The second computing unit performs target recognition on the image data within the spatiotemporal pre-aiming slice, generating a job instruction corresponding to the recognized target.

[0168] The scheduling module 30 is used to control the lateral actuator to perform the operation according to the operation instruction.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A mobile robot task scheduling method based on a heterogeneous computing architecture, characterized in that, include: On a mobile robot equipped with a forward imaging device and a lateral actuator, a work space geometric model is constructed based on the geometric structure of a structured planting row. The work space geometric model represents the spatial correspondence between the visual field of view of the forward imaging device and the physical work envelope of the lateral actuator. The motion state parameters of the mobile robot are obtained by the first computing unit, and the look-ahead compensation amount is determined based on the motion state parameters and the statistical value of the inference time of the target recognition task by the second computing unit. The first computing unit communicates with the forward imaging device, and the second computing unit communicates with the first computing unit. The motion state parameters include the instantaneous linear velocity and / or angular velocity along the planting row direction. The first computing unit uses the look-ahead compensation amount and the workspace geometric model to determine a spatiotemporal preview slice in the visual field of view; The first computing unit uses the acquired raw image data to crop out the image data located within the spatiotemporal preview slice, and sends the cropped image data to the second computing unit; The second computing unit performs target recognition on the image data within the spatiotemporal pre-aiming slice, generates a work instruction corresponding to the recognized target, and controls the lateral execution mechanism to perform the work according to the work instruction.

2. The mobile robot task scheduling method based on a heterogeneous computing architecture according to claim 1, characterized in that, The look-ahead compensation amount is used to characterize the row displacement of the mobile robot moving along the planting row within a predetermined look-ahead time, and the predetermined look-ahead time is determined based on the statistical value of the reasoning time of the second computing unit for the target recognition task.

3. A mobile robot task scheduling method based on a heterogeneous computing architecture according to claim 1 or 2, characterized in that, The determination of the spatiotemporal pre-aiming slice includes: In the work space geometric model, the physical work envelope area is translated in the opposite direction of the movement direction of the mobile robot by the look-ahead compensation amount to obtain the translated work area; The translated work area is projected onto the imaging plane of the forward imaging device, and the projected area serves as the spatiotemporal preview slice.

4. A mobile robot task scheduling method based on a heterogeneous computing architecture according to claim 1 or 2, characterized in that, The first computing unit is an FPGA unit, and the second computing unit is an NPU unit.

5. The mobile robot task scheduling method based on a heterogeneous computing architecture according to claim 1, characterized in that, The step of controlling the lateral actuator to perform the operation according to the operation instruction includes: Based on the work space geometric model, it is determined that the identified target is located within the physical work envelope area; In response to the determination that the location is within the physical operation envelope, the lateral actuator is triggered to execute the operation command.

6. A mobile robot task scheduling method based on a heterogeneous computing architecture according to claim 2, characterized in that, The determination of the spatiotemporal pre-aiming slice includes: Within the time window corresponding to the predetermined look-ahead time, the first computing unit simultaneously collects the high-frequency sequence of the drive current of the mobile robot chassis motor and the point cloud flow output by the 4D radar sensing unit associated with the forward imaging device for sensing the spatial structure of the working lane, while the mobile robot travels along the working lane defined by the multi-layer three-dimensional planting rack. The first computing unit determines the instantaneous jitter offset vector of the forward imaging device based on the high-frequency sequence of the driving current, and determines the sliding pose compensation amount of the mobile robot along the planting row direction based on the point cloud flow and the encoder speed of the chassis motor. Using the first calculation unit, based on the instantaneous jitter offset vector and the slip pose compensation amount, the visual field area corresponding to the working lane in the working space geometric model is corrected, and the spatiotemporal preview slice after jitter and slip compensation is determined based on the corrected working space geometric model.

7. A mobile robot task scheduling method based on a heterogeneous computing architecture according to claim 6, characterized in that, Determining the instantaneous jitter offset vector of the forward imaging device includes: The first computing unit performs wavelet packet transform on the high-frequency sequence of the driving current to extract the current ripple component that matches the mechanical resonance frequency band. Based on a preset current-end jitter transfer function, the current ripple component is mapped to the instantaneous jitter offset vector of the forward imaging device at the imaging moment.

8. A mobile robot task scheduling method based on a heterogeneous computing architecture according to claim 6, characterized in that, The determination of the sliding pose compensation amount of the mobile robot along the planting row direction based on the point cloud flow and the encoder speed of the chassis motor includes: Using the first calculation unit, based on the pitch angle information of the point cloud flow, a height band corresponding to the height range of the multi-layer three-dimensional planting rack column is selected in the vertical direction, and the horizontal slice point cloud distributed along the planting row direction within the height band is extracted. Doppler velocity field analysis is performed on the horizontal slice point cloud, and an environmental reference velocity matrix is ​​constructed based on the column echo clusters distributed within a preset spacing range along the working roadway. The environmental reference velocity matrix and the encoder velocity are cross-correlated to generate a nonlinear slip coefficient sequence. The slip pose compensation amount is obtained by performing time-domain integration on the nonlinear slip coefficient sequence along the planting row direction using the first computing unit.

9. A mobile robot task scheduling method based on a heterogeneous computing architecture according to claim 7, characterized in that, The preset current-end jitter transfer function is obtained in the following way: During the static calibration phase when the mobile robot is stationary at the typical working lane position corresponding to the structured planting row, the first computing unit applies a step excitation signal to the lateral actuator, and the forward imaging device acquires the displacement attenuation curve of the calibration plate. The motor current response sequence caused by the step excitation signal is recorded synchronously. A mapping model between the motor current response sequence and the displacement attenuation curve is established using a system identification algorithm, and the parameters of the mapping model are stored in the on-chip storage resources of the first computing unit as the current-end jitter transfer function.

10. A mobile robot task scheduling system based on a heterogeneous computing architecture, characterized in that, include: A construction module is used to construct a work space geometric model on a mobile robot equipped with a forward imaging device and a lateral actuator, based on the geometric structure of a structured planting row. The work space geometric model represents the spatial correspondence between the visual field of view of the forward imaging device and the physical work envelope of the lateral actuator. The processing module uses a first computing unit to obtain the motion state parameters of the mobile robot, and determines the look-ahead compensation amount based on the motion state parameters and the statistical value of the inference time of the second computing unit for the target recognition task. The first computing unit communicates with the forward imaging device, and the second computing unit communicates with the first computing unit. The motion state parameters include the instantaneous linear velocity and / or angular velocity along the planting row direction. The first computing unit uses the look-ahead compensation amount and the workspace geometric model to determine a spatiotemporal preview slice in the visual field of view; The first computing unit uses the acquired raw image data to crop out the image data located within the spatiotemporal preview slice, and sends the cropped image data to the second computing unit; The second computing unit performs target recognition on the image data within the spatiotemporal pre-aiming slice, generating a job instruction corresponding to the recognized target. The scheduling module is used to control the lateral actuator to perform the operation according to the operation instruction.

Images