A method and system for dynamic obstacle avoidance and working path recovery of unmanned agricultural machines based on deep vision

Abstract

This invention discloses a method and system for dynamic obstacle avoidance and operation path recovery of unmanned agricultural machinery based on depth vision, belonging to the interdisciplinary field of agricultural machinery automation and computer vision. It achieves parallel deep dynamic perception of multiple types of obstacles and real-time identification and geometric modeling of crop row boundaries based on a heterogeneous computing platform; it integrates scene cognition for intelligent judgment of path obstruction status; when obstructed, it initiates path planning with safe obstacle avoidance and operation recovery as the collaborative optimization objectives, dynamically generating obstacle avoidance detour paths; it determines the optimal regression point and solves control parameters through an intelligent decision-making model based on an attention mechanism; it executes path tracking control to achieve precise alignment with crop rows at the regression point; it incrementally constructs and dynamically maintains a global semantic operation map, intelligently managing unoperated areas caused by obstacle avoidance to ensure full operation coverage. This invention systematically solves the problems of continuity, accuracy, and full coverage of unmanned agricultural machinery operations in complex field environments.

Description

Technical Field

[0001] This invention relates to the fields of agricultural machinery automation technology and computer vision interdisciplinary technology, specifically to a method and system for handling obstacles in unmanned field operations based on depth vision. Background Technology

[0002] In the process of agricultural modernization and intelligentization, machine vision-based autonomous navigation and operation of unmanned agricultural machinery has become a key technology for improving agricultural production efficiency and achieving precision agriculture. Among them, using visual sensors to perceive the environment, identify obstacles, and plan paths to replace manual driving in completing tasks such as tilling, sowing, and plant protection is an important development direction in the field of intelligent agricultural equipment.

[0003] However, existing vision-based navigation and obstacle avoidance technologies for unmanned agricultural machinery still face numerous technical bottlenecks and practical challenges when dealing with complex and unstructured field operation environments. Most current systems overemphasize front-end environmental perception and back-end path planning, failing to construct a closed-loop intelligent system that tightly couples scene understanding and task objectives. Due to limitations in the singularity of perceived information, the one-sidedness of scene models, and the constraints of decision-making logic, existing solutions are significantly inadequate in achieving safe, efficient, and continuous operation without omissions.

[0004] The aforementioned technical defects severely restrict the reliability and widespread application of unmanned agricultural machinery in actual production, specifically manifested in the following ways:

[0005] (1) Insufficient perception dimension and weak dynamic understanding ability: Most solutions use traditional image processing and basic deep learning models, which can only achieve binary detection of the "presence / absence" of obstacles, and seriously lack the ability to estimate the precise depth and real-time motion state of obstacles. This makes the system unable to distinguish between stationary field ridges and moving livestock, and also unable to determine the width and depth of ditches, thus failing to provide sufficiently rich and accurate information input for subsequent decision-making;

[0006] (2) Lack of contextual information in the work scenario: Existing technologies usually treat "obstacle avoidance" and "work" as two isolated tasks. The system only constructs an "obstacle map" centered on the agricultural machinery, while completely ignoring the continuous identification and modeling of the crop row structure. This makes the agricultural machinery "know the obstacles, but not the work," and after avoiding obstacles, it loses the original work baseline, making it difficult to automatically and accurately return to the correct work position, resulting in skewed work rows, repeated work, and missed work, which seriously affects the quality of agronomy;

[0007] (3) The path planning strategy is simplistic and lacks task-level intelligence: Existing obstacle avoidance path planning often takes "safe bypass" as the sole objective and adopts general local planning algorithms. The paths generated by this method are often "dead ends," only guiding the agricultural machinery to bypass obstacles, but failing to integrate "returning to the original work line" as an equally important optimization objective. For fixed obstacles, the system lacks the ability to manage the overall progress of the work task globally and cannot intelligently mark and complete the unworked areas caused by detours;

[0008] (4) The contradiction between complex algorithms and the real-time performance of mobile platforms is prominent: Realizing deep visual perception, multi-task deep learning models, and complex planning algorithms requires huge computing resources. Some existing solutions cannot run in real time on the vehicle-mounted computing unit due to their high algorithm complexity, resulting in decision delays; others oversimplify the model to meet real-time requirements, sacrificing the system's performance and robustness. This constitutes an engineering bottleneck restricting the application of advanced intelligent algorithms on unmanned agricultural machinery.

[0009] Although existing research has attempted to integrate multiple sensors, it has generally failed to construct a complete technological closed loop that integrates deep dynamic perception, scene semantic modeling, and dual-objective optimization decision-making, with the continuity of the entire operation process and agronomic compliance as its core. Therefore, there is an urgent need for an intelligent obstacle avoidance and path recovery method and system that possesses high-level scene understanding capabilities, balances obstacle avoidance safety and operational efficiency, and ensures real-time operation on edge devices, in order to truly achieve autonomous, reliable, and high-quality continuous operation of unmanned agricultural machinery in complex fields. Summary of the Invention

[0010] This invention addresses the core problems of existing vision-based autonomous agricultural machinery operation systems in complex field environments, namely poor operational continuity and low intelligence levels due to the disconnect between perception, understanding, and decision-making processes, as well as the separation between obstacle avoidance objectives and operational tasks. Specifically, existing technologies generally suffer from the following technical challenges: First, the perception and modeling processes are separated; obstacle detection and farmland scene recognition are often processed sequentially and independently, failing to provide a unified, real-time scene understanding that integrates obstacle status and operational geographic information for decision-making. Second, the decision-making objective is singular; path planning focuses solely on "safe obstacle avoidance," neglecting the agronomic requirement of "precise alignment with crop rows" as a parallel core optimization objective, leading to difficulties in returning to the correct path after obstacle avoidance, skewed operation rows, and interruptions. Finally, there is a significant contradiction between the computational resource requirements of the aforementioned complex algorithms and the computational limitations of the onboard mobile platform, making it difficult to guarantee the real-time performance and reliability of the system in real-world operations.

[0011] To address this, this invention proposes a method and system for dynamic obstacle avoidance and operation path recovery of unmanned agricultural machinery based on depth vision. By constructing a real-time processing closed loop of "synchronous perception-modeling-decision" and innovatively implementing a path planning strategy of "obstacle avoidance and boundary alignment dual-objective optimization", the invention deeply integrates depth vision information, the high-efficiency computing power of the Ascend processor and intelligent decision-making algorithms, thereby achieving an immediate and thorough understanding of the operating environment and integrated intelligent decision-making for obstacle avoidance and recovery actions. Ultimately, this invention overcomes the technical bottleneck of continuous, precise, and autonomous operation of unmanned agricultural machinery in unstructured field scenarios.

[0012] This invention provides a depth vision-based method for dynamic obstacle avoidance and work path recovery for unmanned agricultural machinery, addressing the problems of work interruption, low efficiency, and difficulty in regression caused by the disconnect between perception and decision-making, and the separation of obstacle avoidance targets and work tasks in existing technologies. The technical solution of this invention constructs a real-time closed loop of "synchronous perception-modeling-decision" and innovatively proposes a dual-objective optimized path planning strategy of "obstacle avoidance and boundary alignment." Combined with the high-performance Ascend AI computing platform, this effectively improves the ability of unmanned agricultural machinery to operate continuously, accurately, and autonomously in complex, unstructured field environments.

[0013] The technical solution of this invention is: a method for dynamic obstacle avoidance and operation path recovery of unmanned agricultural machinery based on depth vision, comprising the following steps:

[0014] Step S1: Synchronously acquire RGB images and depth information of the field scene using a stereo vision system to construct a synchronous data stream containing texture and three-dimensional structure;

[0015] Step S2: Based on the heterogeneous computing platform, the synchronous data stream is processed in parallel to realize the deep dynamic perception of multiple types of obstacles and the real-time identification and geometric modeling of crop row boundaries, and output a structured scene cognition containing three-dimensional information of obstacles and geometric model of crop rows.

[0016] Step S3: Integrate the structured scene cognition with the agricultural machinery's own status to make intelligent judgments on the path obstruction status, and output a judgment decision package that includes the obstruction status, obstacle classification, and target crop row reference line;

[0017] Step S4: When the path is determined to be blocked, path planning is initiated with 'safe obstacle avoidance' and 'job return' as the collaborative optimization objectives. Based on the judgment decision package, obstacle avoidance detour paths are dynamically generated by solving the integrated optimization problem that integrates the two objectives, so as to achieve integrated protection of safety and job continuity.

[0018] Step S5: Optimize the end of the obstacle avoidance path. Using an intelligent decision-making model based on an attention mechanism, determine the optimal regression point under the constraint of the dual objectives through arbitration, and calculate the control parameters required to achieve precise alignment with the crop row.

[0019] Step S6: Based on the obstacle avoidance path and the control parameters, generate low-level control commands to control the steering and drive system of the agricultural machinery to perform motion and path tracking, so that the agricultural machinery can achieve precise alignment with the crop row at the optimal return point;

[0020] Step S7: During the operation, incrementally build and dynamically maintain a global semantic operation map containing the operation progress layer, agronomic structure layer and environmental memory layer for closed-loop task management. By comparing the planned path with the actual path, automatically identify and manage the 'already bypassed but to be filled' areas caused by obstacle avoidance to ensure full operation coverage.

[0021] Furthermore, in step S2, the heterogeneous computing platform is a heterogeneous computing platform integrating a central processing unit (CPU) and a dedicated neural network processor (NPU); the parallel processing includes:

[0022] A first deep learning model and a second deep learning model, after model compression, are run in parallel on the NPU; the first deep learning model is an object detection network for obstacle detection, and the second deep learning model is an encoder-decoder network for semantic segmentation.

[0023] The first deep learning model is used to detect obstacles based on RGB images, and combines depth information to convert the detection results into three-dimensional spatial coordinates and estimate the motion speed;

[0024] The second deep learning model is used to perform semantic segmentation of crop rows based on RGB images, and to perform geometric fitting on the segmentation results to obtain a mathematical model of the crop row boundary lines.

[0025] Furthermore, in step S3, the intelligent determination of the path obstruction state includes:

[0026] Unify all perceived information into the vehicle coordinate system;

[0027] The estimated collision time between the computer and the obstacle, based on the predicted collision time model;

[0028] The estimated collision time is compared with a dynamic safety threshold to determine whether immediate thwarting is possible.

[0029] Based on the semantic category and motion state of the obstacle, it is classified into dynamic temporary obstacle, static permanent obstacle, or unknown obstacle, and the corresponding handling strategy is marked.

[0030] Furthermore, in step S4, the bi-objective optimization path planning is achieved by solving a constrained optimization problem, the objective function J of which is defined as the weighted sum of the obstacle avoidance cost term Ca and the regression alignment cost term Cr: J = wa*Ca + wr*Cr; where wa and wr are the weight coefficients of the two terms, respectively.

[0031] Furthermore, an improved hybrid A* algorithm is employed to solve the optimization problem. The heuristic function h(n) of this algorithm incorporates the cost h to the geometric objective point. goal (n) and the estimated alignment cost h align (n): h(n) = h goal (n)+γ*h align (n), where γ is the weight coefficient of the alignment heuristic.

[0032] Further, in step S5, the intelligent decision-making model is a regression decision-making model based on spatiotemporal attention mechanism and multi-objective reinforcement learning, which includes an encoder unit for feature encoding, a spatiotemporal attention unit for focusing key information, and a multi-objective evaluation unit for outputting multi-dimensional evaluation scores; the process of determining the optimal regression point includes:

[0033] Define candidate regression windows at the end of the obstacle avoidance and detour path;

[0034] Construct a multivariate input tensor that includes visual features, geometric constraints, agricultural machinery status, and path shape;

[0035] The candidate regression points are evaluated using the regression decision model, and the visual-geometric alignment score, motion smoothness score, and work efficiency score are output.

[0036] The comprehensive value of each candidate point is calculated based on the scores, and the point with the highest comprehensive value is selected as the optimal regression point.

[0037] Furthermore, in step S6, a hierarchical model predictive controller is used to achieve path tracking and terminal alignment. The cost function of this controller includes a terminal alignment cost term J. terminal This is used to enhance the matching degree between the predicted trajectory endpoint and the optimal regression point in terms of position and heading.

[0038] Furthermore, in step S7, the global semantic job map is a multi-layered fusion structure, including at least:

[0039] The work progress layer is used to record the ground status as "not yet done", "in progress", "completed" and "already bypassed but awaiting replacement".

[0040] The agronomic structure layer is used to persistently store the global crop row network model;

[0041] An environmental memory layer is used to persistently record information about identified static, permanent obstacles.

[0042] The method further includes: by comparing the original planned operation path with the actual detour path, automatically identifying and marking the "detoured and to be completed" area, generating the task unit to be completed, and inserting it into the global task queue for dynamic scheduling.

[0043] This invention discloses a dynamic obstacle avoidance and operation path recovery system for unmanned agricultural machinery based on depth vision, the system comprising:

[0044] The synchronous sensing module is configured to synchronously acquire RGB images and depth information of the field scene through a stereo vision system mounted on agricultural machinery, and construct a synchronous data stream;

[0045] The scene understanding module is configured to perform parallel processing on the synchronous data stream based on a heterogeneous computing platform and output structured scene cognition.

[0046] The intelligent judgment module is configured to integrate the structured scene cognition with the agricultural machinery status to judge the path obstruction status and classify obstacles;

[0047] The path planning module is configured to perform path planning with 'safe obstacle avoidance' and 'job return' as the collaborative optimization objectives when the path is blocked. By solving the integrated optimization problem that combines the two objectives, an obstacle avoidance detour path is generated.

[0048] The regression decision module is configured to optimize the end of the obstacle avoidance path. It uses an intelligent decision model based on an attention mechanism to arbitrate under the constraints of the two objectives to determine the optimal regression point and solve the control parameters.

[0049] The motion control module is configured to generate control signals for driving the steering and travel mechanisms of the agricultural machinery based on the obstacle avoidance path and the control parameters, so as to control the agricultural machinery to track the path and achieve precise alignment at the optimal return point;

[0050] The task management module is configured to incrementally build and maintain a global semantic job map with a multi-layered structure to achieve closed-loop task management and automatically identify and manage 'already bypassed and waiting to be filled' areas; wherein, the scene understanding module, intelligent judgment module, path planning module, regression decision module and motion control module constitute a real-time processing closed loop.

[0051] Furthermore, the heterogeneous computing platform is an in-vehicle heterogeneous computing unit that integrates a central processing unit (CPU) and a dedicated neural network processor (NPU); the scene understanding module is deployed on the NPU of this platform and includes a parallel-running obstacle depth dynamic perception submodule and a crop row real-time recognition and geometric modeling submodule.

[0052] The beneficial effects of this invention are as follows: By constructing a real-time closed loop of "synchronous perception-modeling-decision" and a dual-objective optimization mechanism of "obstacle avoidance and boundary alignment," the system systematically solves the problems of low operational efficiency, incomplete coverage, and reduced agronomic quality caused by obstacle avoidance interruptions in complex field environments for unmanned agricultural machinery, achieving fully autonomous operation from environmental perception and intelligent decision-making to precise execution. This solution, based on the Ascend processor, achieves parallel dynamic perception of obstacle depth and crop row boundary modeling, providing precise scene cognition that integrates semantics and geometry for decision-making. Its dual-objective path planning algorithm and the introduced intelligent decision-making model for regression points ensure that the agricultural machinery can adaptively, smoothly, and accurately return to the target crop row after safely avoiding obstacles, fundamentally guaranteeing the continuity of operations and agronomic compliance. By constructing and dynamically maintaining a global semantic operation map, the system has the ability to automatically mark and intelligently supplement operations in "already bypassed but not yet operated" areas, achieving closed-loop management and full coverage guarantee of operation tasks. Ultimately, relying on the Ascend full-stack AI software and hardware collaborative optimization, the entire set of complex algorithms achieved millisecond-level real-time processing closed loop on the vehicle-mounted mobile platform, enabling this high-level intelligent system to meet the stringent performance requirements of actual operations and powerfully promoting the industrial technology evolution of agricultural machinery towards full autonomy and high reliability. Attached Figure Description

[0053] Figure 1 A flowchart for the intelligent decision-making and execution closed-loop process of dynamic obstacle avoidance and path recovery for unmanned agricultural machinery.

[0054] Figure 2 This is a flowchart of the intelligent regression decision-making module.

[0055] Figure 3 A schematic diagram of static obstacle handling based on the dual objective of "obstacle avoidance-regression": global replanning and intelligent marking of the work area, taking a ditch as an example. Detailed Implementation

[0056] The invention will now be described in detail with reference to a practical application scenario to better understand its technical principles and practical application. This embodiment is only used to illustrate the technical solution of the invention and does not constitute a limitation on the scope of protection of the invention. This embodiment is based on the dynamic obstacle avoidance and path recovery process of an unmanned harvester used for autonomous harvesting in corn fields.

[0057] In this embodiment, to fully illustrate the technical principles and implementation details of the present invention, the Huawei Ascend AI processor and its supporting software ecosystem (such as the MindSpore framework and CANN architecture) will be used as a specific implementation example of the "heterogeneous computing platform." Those skilled in the art should understand that the "heterogeneous computing platform integrating a central processing unit (CPU) and a dedicated neural network processor (NPU)" as defined in the claims of this invention is not limited to the Ascend platform. Any embedded or automotive computing platform with a similar heterogeneous computing architecture (e.g., integrating a CPU with a GPU, DSP, or other dedicated AI acceleration units) can implement the "synchronous perception-modeling-decision" closed-loop and "obstacle avoidance-regression" dual-objective optimization method disclosed in this invention by adapting its corresponding software toolchain (such as model conversion and deployment tools). The following detailed steps aim to reveal the core process and algorithmic ideas of the invention, and its implementation can be engineered and ported based on different hardware foundations.

[0058] The system of this invention is deployed on the vehicle-mounted Ascend heterogeneous computing platform, and the implementation of each functional module is as follows:

[0059] Synchronous perception module: Composed of a stereo vision system (including two high-resolution industrial cameras and a synchronization controller) and underlying driver software, it is responsible for the synchronous acquisition and preprocessing of data.

[0060] Scene understanding module: a deep learning inference engine deployed on the Ascend NPU, containing two parallel neural network models: an obstacle detection model and a crop row segmentation model, which output structured scene cognition; it is scheduled through the Ascend CANN runtime.

[0061] Intelligent Judgment Module: A logic judgment unit deployed on the CPU, based on a collision detection algorithm and rule engine implemented in C++ / Python, running on a real-time operating system. It performs path obstruction status judgment and obstacle classification.

[0062] Path planning module: A planning algorithm unit deployed on the CPU, employing an improved hybrid A* algorithm and optimization solver implemented in C++. It performs path planning with 'safe obstacle avoidance' and 'job regression' as collaborative optimization objectives, generating obstacle avoidance and detour paths by solving an integrated optimization problem that combines these two objectives.

[0063] Regression Decision Module: A lightweight neural network model deployed on the NPU, loaded and inferred via the Ascend MindSpore framework. An attention-based intelligent decision model arbitrates the optimal regression point and calculates the control parameters under the constraints of the dual objectives.

[0064] Motion control module: A real-time control unit deployed on the CPU, containing a model predictive controller (MPC) algorithm, communicating with the agricultural machinery chassis controller via a CAN bus. It generates control signals to drive the agricultural machinery's steering and travel mechanisms, controlling the machinery to track the path and achieve precise alignment at the optimal regression point.

[0065] Task Management Module: A task scheduling unit deployed on the CPU, maintaining a global semantic map database and task queue, implemented using C++ / SQLite. It is used to incrementally build and maintain a multi-layered global semantic job map to achieve closed-loop task management, automatically identifying and managing 'bypassed and pending completion' areas.

[0066] The modules exchange data through shared memory and message queues, forming a loosely coupled distributed system architecture.

[0067] like Figure 1-3 As shown, the present invention discloses a method and system for dynamic obstacle avoidance and operation path recovery of unmanned agricultural machinery based on depth vision. The method includes the following steps:

[0068] Step S1: Synchronously acquire RGB images and depth information of the field scene using a stereo vision system to construct a synchronous data stream containing texture and three-dimensional structure;

[0069] Step S2: Based on the heterogeneous computing platform, the synchronous data stream is processed in parallel to realize the deep dynamic perception of multiple types of obstacles and the real-time identification and geometric modeling of crop row boundaries, and output a structured scene cognition containing three-dimensional information of obstacles and geometric model of crop rows.

[0070] Step S3: Integrate the structured scene cognition with the agricultural machinery's own status to make intelligent judgments on the path obstruction status, and output a judgment decision package that includes the obstruction status, obstacle classification, and target crop row reference line;

[0071] Step S4: When the path is determined to be blocked, path planning is initiated with 'safe obstacle avoidance' and 'job return' as the collaborative optimization objectives. Based on the judgment decision package, obstacle avoidance detour paths are dynamically generated by solving the integrated optimization problem that integrates the two objectives, so as to achieve integrated protection of safety and job continuity.

[0072] Step S5: Optimize the end of the obstacle avoidance path. Using an intelligent decision-making model based on an attention mechanism, determine the optimal regression point under the constraint of the dual objectives through arbitration, and calculate the control parameters required to achieve precise alignment with the crop row.

[0073] Step S6: Based on the obstacle avoidance path and the control parameters, generate low-level control commands to control the steering and drive system of the agricultural machinery to perform motion and path tracking, so that the agricultural machinery can achieve precise alignment with the crop row at the optimal return point;

[0074] Step S7: During the operation, incrementally build and dynamically maintain a global semantic operation map containing the operation progress layer, agronomic structure layer and environmental memory layer for closed-loop task management. By comparing the planned path with the actual path, automatically identify and manage the 'already bypassed but to be filled' areas caused by obstacle avoidance to ensure full operation coverage.

[0075] To achieve a millisecond-level real-time processing closed loop from perception to control, this embodiment utilizes the Ascend full-stack AI toolchain to deeply optimize and collaboratively schedule the deep learning models in steps S2 and S5, and the planning and control algorithms in steps S4 and S6. Specifically, this includes: using Ascend CANN for model transformation and graph optimization, deploying lightweight models on the NPU; and achieving pipelined parallelism of perception, planning, and control tasks through heterogeneous task scheduling. Ultimately, the system's end-to-end latency can be stably controlled within 200-300 milliseconds, meeting the rapid response requirements of high-speed agricultural machinery to dynamic obstacles.

[0076] Step 1 involves constructing a high-quality environmental data stream to provide synchronized perceptual input that integrates texture and 3D structure for subsequent intelligent processing. The specific construction and execution process of the synchronized data stream is as follows:

[0077] Hardware configuration and data acquisition mechanism:

[0078] To achieve deep perception of the work scene, the unmanned agricultural machinery is equipped with a calibrated and hardened stereo binocular vision system. This system consists of two high-resolution industrial cameras with a fixed horizontal distance (called the stereo baseline, denoted as B). In each processing cycle, the system instructs the left and right cameras to perform strict hardware-synchronized exposure, thereby capturing the left view I of the field scene at the same moment. left (x,y) and right view I right (x, y) form a pair of 3D images;

[0079] In addition to stereo vision, the system is also compatible with an active RGB-D camera as a supplement to adapt to different lighting conditions. Its core output is a time-stamped RGB color image and depth information.

[0080] Real-time calculation and generation of depth information:

[0081] For a stereo binocular system, the acquisition of depth information is a computational process. After receiving a pair of synchronized images, the system immediately executes a stereo matching algorithm on the preprocessing unit of the on-board computing unit. The goal of this algorithm is to find the corresponding point in the right image for each pixel (x,y) in the left image. The difference in pixel coordinates in the horizontal direction is defined as the disparity, denoted as d.

[0082] According to the principles of perspective geometry, the perpendicular distance from a point in the scene to the camera, i.e., the depth value Z, is calculated using the following formula to obtain the depth calculation model:

[0083] Z = (f * B) / d

[0084] In the formula: Z represents the calculated depth value, which is the real spatial distance from the measured point to the camera plane, in meters (m). This is the key 3D information for subsequent obstacle localization and scene modeling; f represents the camera's focal length, in pixels. This is an internal parameter obtained in advance through camera calibration, reflecting the camera's imaging characteristics; B represents the stereo baseline, which is the horizontal distance between the optical centers of the left and right cameras, in meters (m). The larger the value of B, the higher the theoretical depth accuracy that can be measured, but the overlapping area of ​​the field of view will be reduced; d represents the parallax, in pixels. It is obtained through a stereo matching algorithm and is the core variable connecting the 2D image and 3D space. The larger the parallax d, the closer the corresponding point is to the camera (the smaller Z is).

[0085] By traversing the images, the system ultimately generates an image that is identical to the left view I. left The depth map D(x, y) corresponds to each pixel, where the value of each pixel is the depth Z calculated by Formula 1;

[0086] Data stream construction and transmission:

[0087] Ultimately, the output of this step is a structured synchronization packet, each packet containing:

[0088] RGB image frames: High dynamic range color images used for subsequent object recognition, semantic segmentation, and texture analysis;

[0089] Depth frame: A matrix aligned with the pixels of the RGB image, providing spatial distance information for every point in the scene;

[0090] Timestamp and pose assistance information: The precise timestamp of the acquisition time, and the estimated pose of the agricultural machinery at that time provided by the agricultural machinery integrated navigation system (used for preliminary alignment of multi-frame data).

[0091] The data packet is streamed in real time to the core computing unit (the host equipped with the Ascend processor) via a high-speed serial bus at a stable frame rate (≥10 Hz).

[0092] Step 2 specifically involves: performing high-efficiency and high-precision real-time interpretation of the synchronous data stream input in Step 1, and outputting in parallel a structured understanding of dynamic obstacles and static crop rows in the environment. This step relies on the powerful heterogeneous computing architecture and dedicated neural network processing unit (NPU) of the Ascend processor to engineer and real-time the complex deep learning model inference process. The specific execution flow of the parallel perception and scene understanding is as follows:

[0093] Hardware platform and computing architecture:

[0094] This step runs entirely on the in-vehicle Ascend AI computing unit. The core advantage of this platform lies in its "CPU + NPU" heterogeneous computing architecture. The CPU is responsible for logic scheduling, data preprocessing, and post-processing, while the NPU (Neural Processing Unit), designed specifically for deep learning, is responsible for carrying out the high-concurrency, low-power inference tasks of all deep learning models in this step. Through the scheduling of the Ascend AI software stack, the RGB image and depth map data streams from step 1 are efficiently allocated to different computing units, realizing pipeline parallelism of tasks.

[0095] Deployment and inference of dual-path parallel deep learning models:

[0096] On the Ascend NPU, two lightweight deep learning models optimized by quantization and pruning were deployed. They share the same spatiotemporal data source input from step 1, but are completely decoupled in terms of network structure and function, achieving task-level parallelism.

[0097] Subtask 1: Dynamic Depth Perception of Obstacles

[0098] This task is accomplished by an improved, anchor-free, single-stage object detection network. Its input is the RGB image frame provided in step 1. The network outputs a feature map where each activation point predicts a heatmap of the center point of a potential obstacle, its semantic category (pedestrian, vehicle, livestock, field ridge, ditch), and the size offset of its 2D bounding box. Subsequently, the system combines the depth map D(x, y) generated in step 1, aligned with the RGB image pixels, and uses the following formula to convert the detected 2D pixel center points (u, v) into 3D spatial coordinates (X, y) in the camera coordinate system. c , Y c Z c ):

[0099] Pixel coordinates to camera coordinate system transformation:

[0100] X c = (u – c x ) * Z c / f x

[0101] Yc = (v – c y ) * Z c / f y

[0102] Z c = D(u, v)

[0103] In the formula: (u, v) represents the pixel coordinates of the detected obstacle center point in the image, in pixels; (c x , c y (f) represents the principal point coordinates of the camera, in pixels, and is one of the camera's intrinsic parameters, indicating the intersection of the optical axis and the imaging plane; x , f y () represents the camera's focal length in the x and y directions, measured in pixels, and is one of the camera's intrinsic parameters; (X) c , Y c Z c Z represents the three-dimensional coordinates of the obstacle in the camera coordinate system, in meters (m), where Z c It is directly taken from the value of the depth map D at position (u, v); D(u, v) represents the depth value of the depth map at pixel coordinates (u, v), in meters (m);

[0104] Furthermore, by associating the three-dimensional positions (X) of obstacles of the same category in consecutive frames... c , Y c Z c Estimate its speed using the following formula:

[0105] 3D motion estimation based on inter-frame difference:

[0106] V≈ (P t - P {t-Δt} ) / Δt

[0107] In the formula: V represents the three-dimensional velocity vector of the obstacle in the camera coordinate system, with units of meters per second (m / s); P t This represents the three-dimensional position coordinates (X, t) of the obstacle at the current time t. c, Y c Z c ) t ;P {t-Δt} This represents the three-dimensional position coordinates (X, Δt) of the obstacle in the previous processing cycle (before time Δt). c, Y c Z c){t-Δt} Δt represents the processing cycle, which is the time interval between two consecutive frames, in seconds (s). Its reciprocal is the system frame rate.

[0108] Subtask 2: Real-time identification and geometric modeling of crop rows

[0109] This task is implemented by a lightweight semantic segmentation network with an encoder-decoder structure, whose input is also the RGB image frame provided in step 1. The network performs pixel-level classification of the image and outputs a crop row region segmentation mask M(x, y), where pixels belonging to crop rows are marked as 1 and the background as 0. Subsequently, morphological operations are applied to the mask to remove noise, and Hough transform is used to fit the connected crop row pixel clusters into a series of straight lines. These equations together constitute the geometric model L of the crop row boundary line in the current field of view. i , can be expressed as ρ i = x * cosθ i + y * sinθ i (Hough space straight lines) provide quantifiable operational guide lines for agricultural machinery; where:

[0110] ρ i : Represents the normal distance from the i-th detected line to the origin of the image plane coordinate system. Its unit is usually pixels. This is a scalar value that directly reflects the position of the line in the image; x: Represents the x-coordinate of a point in the image plane coordinate system, in pixels. It indicates the horizontal position of the point in the two-dimensional space of the image; θ i : Represents the angle between the normal to the i-th detected line (the perpendicular line from the origin to the line) and the positive x-axis. This angle parameter defines the direction of the line, and in agricultural machinery scenarios, it directly reflects the direction of the crop rows; y: Represents the ordinate of a point in the image plane coordinate system, in pixels, which indicates the vertical position of the point in the two-dimensional space of the image; Subscript i: Represents the index, used to distinguish multiple different lines (i.e., multiple different crop row boundaries) identified simultaneously in the same image frame.

[0111] Output and encapsulation of structured perception results:

[0112] The final output of this step is a scene understanding data structure that encapsulates the results of parallel processing and serves as direct input to steps 3 (path determination) and 5 (regression decision). This data structure mainly includes:

[0113] Obstacle list: Each detected obstacle is treated as an object, containing its semantic category and three-dimensional spatial coordinates (X, Y, X). c , Y c Z c ), the estimated three-dimensional velocity vector V, and bounding box information;

[0114] Crop row geometry model set: contains mathematical model parameters for all crop row boundaries identified in the current image;

[0115] Associated timestamps and metadata: timestamps aligned with the original data frame, and confidence scores generated during processing.

[0116] Step 3 involves the following steps: Real-time and comprehensive analysis of the structured scenario understanding output from Step 2; based on rigorous mathematical models and rule logic, determining whether the agricultural machinery's current planned operating path faces risks, the level of those risks, and distinguishing the attributes of the risk sources to provide precise decision-making instructions for subsequent differentiated intelligent planning; the specific execution flow for the intelligent judgment of the path obstruction state is as follows:

[0117] Spatiotemporal alignment and unified representation of multi-source information:

[0118] First, the system unifies all the information output in step 2 into a vehicle coordinate system with the rear axle center of the agricultural machinery as the origin, forming a consistent spatiotemporal context; including:

[0119] Obstacle coordinate transformation: Utilizing the current pose of the agricultural machinery, the three-dimensional coordinates and velocity of the obstacle in the camera coordinate system are transformed into coordinates P in the vehicle coordinate system using a coordinate transformation matrix. o = (X o , Y o ) and velocity V o = ( , Then, a risk assessment based on agricultural machinery kinematics is conducted;

[0120] Crop row model projection: The crop row geometric model identified in step 2 based on image coordinates, combined with the relative installation position of the camera and the vehicle body and the extrinsic parameter matrix, is projected onto the ground plane in the vehicle coordinate system to form a global operation path reference line L. g ;

[0121] Agricultural machinery status integration: Real-time acquisition of the agricultural machinery's own status, including but not limited to: current forward speed v h , heading angle ψ, minimum turning radius R min and the width W of the work tools o ;

[0122] Path obstruction determination based on dynamic collision risk:

[0123] The system uses the current speed V of the agricultural machinery. h Using the pre-set work path as the baseline scenario, a collision risk assessment is performed on each tracked obstacle, with the core assessment indicator being the estimated collision time. For dynamic obstacles, the calculation formula is as follows:

[0124] Calculation of predicted collision time based on relative motion:

[0125] TTC = - (r x * s x + r y * s y ) / ( + )

[0126] Constraints: (r x * s x + r y * s y ) < 0

[0127] In the formula: TTC represents the estimated collision time in seconds. It indicates the shortest estimated time required for the agricultural machinery to collide with the obstacle under the current relative motion state; the smaller the TTC value, the more imminent the collision risk; r x This represents the position of the obstacle relative to the agricultural machinery along the X-axis (the longitudinal direction of the agricultural machinery's movement) in the vehicle coordinate system, expressed in meters; r y This represents the position of the obstacle relative to the agricultural machinery along the Y-axis (lateral direction) in the vehicle's coordinate system, expressed in meters (s). x Represents the relative velocity of the obstacle with respect to the agricultural machinery along the X-axis, measured in meters per second (m / s). y This represents the relative velocity of the obstacle to the agricultural machinery along the Y-axis, measured in meters per second. and These are independent measures of the intensity of the longitudinal and lateral relative motion components, respectively. + This constitutes a scalar measure of the total intensity of relative motion (i.e., the square of the velocity modulus).

[0128] For a static obstacle, the relative velocity is zero, and TTC is simplified to the time it takes for the agricultural machinery to travel along the path at the current speed to the location of the obstacle;

[0129] The system presets a dynamic security threshold (TTC). th When the TTC calculated value of any obstacle is greater than zero and less than the TTC th If the current path is "immediately blocked", obstacle avoidance must be initiated. At the same time, the TTC value also quantifies the urgency of the blockage, which is used for priority scheduling of subsequent planning modules.

[0130] Obstacle semantic-motion attribute fusion and classification labeling:

[0131] To support the differentiated path planning strategy in step 4, this step performs a fine-grained classification of obstacles that trigger the "obstructed" judgment:

[0132] Dynamic temporary obstacles: those with semantic categories of "pedestrians", "livestock" or "vehicles" and whose estimated speed amplitude is greater than the minimum threshold, are expected to change location. The system marks their handling strategy as "local temporary detour".

[0133] Static permanent obstacles: semantically categorized as "field ridges," "ditches," and "piles of stones," their movement speed is close to zero; these obstacles have fixed positions, and the system marks their processing strategy as "global replanning and recording"; for obstacles with dimensions, such as ditches, it is also necessary to estimate an "unexplored influence area polygon" A based on its 3D point cloud. avoid ;

[0134] Unknown obstacles: When the confidence level of the obstacle category is low and the motion state is difficult to estimate, the system adopts the most conservative strategy, marking it as "warning and preparing to stop" while continuously observing;

[0135] Rapid analysis and assessment of path feasibility:

[0136] In addition to collision risk assessment based on TTC, the system also performs rapid path feasibility analysis based on space; in the crop row model L g Under the constraints, the system determines whether there exists a smooth local path that satisfies the kinematics of agricultural machinery and can avoid all obstacles on the path that have negative TTC values ​​(i.e., no immediate collision risk). If no such path exists, it is marked as "path infeasible, requires pre-planning" even if the current TTC has not exceeded the threshold.

[0137] Finally, this step outputs a structured decision package, which includes:

[0138] Main status: "Unimpeded", "Immediately Blocked", "Path Not Feasible";

[0139] List of trigger obstacles: ID, category, TTC value, and handling strategy flag (local detour, global replanning) for each obstacle;

[0140] Key parameters: For the "immediately blocked" state, provide the TTC value, location, and classification of the most pressing obstacle;

[0141] Context: Current target crop row reference line L g The equation.

[0142] Step 4 is as follows: After Step 3 determines that the path is blocked, a dynamically feasible path that can simultaneously meet the dual requirements of "safe obstacle avoidance" and "operational continuity" is generated. This step transforms the decision instructions of Step 3 into specific geometric paths. Its innovation lies in integrating "regression alignment" as an optimization objective of equal importance to "obstacle avoidance" for integrated solution. The specific execution flow of the dynamic generation of the dual-objective optimized obstacle avoidance path is as follows:

[0143] Problem Modeling and Optimization Goal Definition

[0144] The system is based on the structured decision package output in step 3 (which includes information on trigger obstacles and the target crop row reference line L). g (And processing strategy label), formalizing the path planning problem as a constrained optimization problem; the state space of this problem consists of the position (x, y) and heading φ of the agricultural machinery in the vehicle coordinate system; the optimization objective function J is defined as a weighted sum of two terms to simultaneously minimize obstacle risk and operation deviation cost:

[0145] The total cost function for bi-objective path optimization is:

[0146] J = w a * C a + w r * C r

[0147] In the formula: J represents the total cost of the planned path, which is a dimensionless scalar value; the planner achieves bi-objective optimization by searching for the path that minimizes J; w a The weight coefficient representing the obstacle avoidance cost is a real constant greater than zero; the larger the weight, the farther the planned path is from all obstacles; C a The obstacle avoidance cost term is a dimensionless scalar value; it is calculated by summing the repulsive potential field function values ​​from all sampling points along the planned path to all obstacles (from the trigger obstacle list in step 3); w r The weighting coefficient representing the regression alignment cost term is a real constant greater than zero; this coefficient determines the system's emphasis on operational continuity and accuracy; C r The regression alignment cost term is a dimensionless scalar value; its core function is to measure the deviation between the final state of the planned path and the ideal regression state.

[0148] Path solving based on an improved hybrid A* algorithm

[0149] To efficiently solve the above bi-objective optimization problem, this step adopts an improved hybrid A* search algorithm. This algorithm has made key improvements on the basis of the traditional hybrid A* algorithm, so that the heuristic function h(n) in its cost function g(n) + h(n) not only includes the cost to the geometric endpoint, but also incorporates the estimation of the alignment cost.

[0150] State space discretization: The algorithm discretizes the continuous state space (x, y, φ) of the agricultural machine to form search nodes; each node represents a possible pose of the agricultural machine.

[0151] Motion primitive expansion: Starting from the current node, the algorithm expands it using a set of short-distance motion primitives that conform to the kinematic model of agricultural machinery to generate successor nodes;

[0152] Dual-objective heuristic guidance: The algorithm designs a composite heuristic function h(n) = h goal (n)+γ*h align (n); where h goal (n) is the Euclidean distance from node n to the geometric target point; h align (n) represents the position from node n to the desired regression line L. g The alignment cost estimate; γ is the weighting coefficient of the alignment heuristic;

[0153] Collision detection and constraint handling: During node expansion, strict collision detection is performed on the path segments generated by each motion primitive; simultaneously, it is checked whether the path meets the minimum turning radius R of the agricultural machinery. min Kinematic constraints; for fixed obstacles marked "global replanning", the corresponding unworked region polygon A_avoid is considered an impassable region;

[0154] Path extraction and smoothing: When the algorithm finds a node that meets the termination condition, it generates an initial path by backtracking, which is composed of discrete nodes. Since the path may contain jagged edges caused by discrete turns, the gradient descent method is then used to smooth the path. Under the premise of ensuring no collision and kinematic feasibility, a continuous and smooth path that can be stably tracked by the controller is obtained.

[0155] Unlike the sequential planning common in existing technologies (planning a pure obstacle avoidance path first, and then adding a regression search at the end of the path), this invention integrates the two objectives of "safe obstacle avoidance" (Ca) and "job regression" (Cr) into a unified optimization problem for simultaneous solution through the aforementioned objective function J. This "integrated optimization" design allows the obstacle avoidance risk and regression cost to be evaluated simultaneously at each extended node in the path search process, thereby fundamentally avoiding the generation of "dead-end paths" that, while bypassing obstacles, deviate significantly from crop rows, leading to difficult or even impossible regression. The improved hybrid A* algorithm incorporates the alignment heuristic h_align(n), which embodies this idea and ensures the dual feasibility of the final path in terms of both geometric endpoint and job alignment state from the search guidance level.

[0156] Structured output of planning results:

[0157] The final output of this step is a structured path planning package, which includes:

[0158] Path planning: Smoothed sequence of path points planned= {p0, p1, ..., p n}, where p represents a pose point on the path, p0 represents the start point of the path, p1 represents the end point of the path, and p0, p1, ..., p n Each point p represents a midpoint of the path. i Includes coordinates (x) i , y i ) and desired heading φ i ;

[0159] Related strategy information: Record the processing strategy markers used in step 3 as the basis for this planning;

[0160] Global task update instruction: When performing global replanning to deal with fixed obstacles, output a task block to be done (Task). block This task block is associated with the previously marked untouched influence area polygon A. avoid This is used in step 7 to update the global job map.

[0161] Step 5 involves processing the obstacle avoidance path generated in step 4. planned Intelligent optimization is performed at the end of the process, determining the "optimal" operation regression point from the "accessible" geometric paths and calculating the control parameters required to achieve accurate and smooth alignment. This step introduces a novel regression decision model, elevating the regression process from a rule-based geometric calculation to a multi-factor dynamic optimization decision-making process that comprehensively considers visual quality, agronomic requirements, agricultural machinery status, and energy consumption. The specific execution flow of the intelligent regression point decision and parameter calculation is as follows:

[0162] Decision space definition and model input construction:

[0163] The system first outputs the path in step 4. planned The end of the term defines a "candidate regression window"; the path points {p} within this window {n-m} , ..., p n The region and its neighboring area constitute a continuous decision space; the input to this step is a multivariate tensor I, which is composed of the following four parts of information:

[0164] Visual feature tensor F v The multi-level feature maps output by the crop row segmentation network in step 2 in the corresponding image region of the candidate window are concatenated and dimensionality reduced to form a feature tensor that encodes the visual saliency, edge sharpness and noise distribution of the crop row.

[0165] Geometric constraint tensor F g The crop row geometric model L established in step two gProjecting onto the ground coordinate system of the candidate window generates a coordinate system representing the point's path to the ideal regression line L. g The lateral distance field and the directional field;

[0166] Agricultural machinery state sequence S h This includes the agricultural machinery's speed, heading angle, yaw rate at the starting point of the candidate window, and the estimated kinematic costs to each point on the path;

[0167] Path shape encoding E p : Path within the candidate window planned Encode it to represent its curvature changes and endpoint trends;

[0168] Intelligent decision-making based on regression decision-making models:

[0169] The regression decision model is the core innovative algorithm of this step; it is a lightweight deep neural network deployed on the Ascend NPU and adopts an "encoder-spatiotemporal attention-multi-objective evaluator" structure.

[0170] Encoder: Performs fusion encoding on the input tensor I to extract a unified feature representation H;

[0171] Spatiotemporal attention module: This module is crucial to the model; it contains two parallel attention heads:

[0172] Spatial attention head: applies attention weights to feature H, focusing it on image feature F. v The pixel region with the clearest and most continuous visual cues of the crop row, while suppressing interference areas caused by shadows, weeds, and lodged crops;

[0173] Sequence attention head: for the agricultural machinery state sequence S h and path encoding E p The analysis focuses on the path range with the smoothest motion, lowest energy consumption, and the ability to seamlessly connect to subsequent operations.

[0174] Multi-objective evaluator: This network layer takes attention-weighted features as input and outputs three evaluation scalars, which together determine the quality of candidate regression points.

[0175] Visual-Geometric Alignment Score align Assess the visual and geometric alignment of the point with the crop row;

[0176] Motion smoothness score S smooth : Evaluate the control effort and trajectory curvature required to transition from the current path to this point and resume straight-line travel;

[0177] Work efficiency score S efficiency : Evaluate the contribution of selecting this point to the overall operation progress and coverage;

[0178] For each potential regression point p within the candidate window candidate The model outputs a comprehensive value V(p), which is calculated using the following formula:

[0179] Regression point comprehensive value function:

[0180] V(p) =σ(w align ⋅S align (p) + w smooth ⋅S smooth (p) + w eff ⋅S efficiency (p)

[0181] In the formula: V(p) represents the comprehensive value score of candidate regression point p; σ(⋅) represents the Sigmoid activation function, used to map the weighted sum to the (0,1) interval; w align The weighting coefficient representing the alignment score item is a preset positive real number, reflecting the importance of agronomic alignment accuracy; S align (p) represents the model's visual-geometric alignment score for point p; w smooth The weighting coefficient for the smoothness score is a positive real number, reflecting the importance of motion stability; S smooth (p) represents the model's output score for the smoothness of motion at point p; w eff The weighting coefficient for the efficiency score item is a positive real number, reflecting the importance of operational efficiency; S efficiency (p) represents the model's operational efficiency score for point p. The three score outputs mentioned above need to be normalized before calculation.

[0182] The model selects the point with the highest V(p) as the optimal regression point p. return And simultaneously output the expected heading φ corresponding to that point. return ;

[0183] Adaptive solution of precise control parameters:

[0184] Obtain the optimal regression point p return =(x return ,y return ,φ return (where: x) return This represents the coordinates of the optimal regression point along the X-axis in the reference coordinate system, measured in meters (m); y return After representing the coordinates of the optimal regression point along the Y-axis in the reference coordinate system, the system further calculates the precise parameters for the path tracking controller in step 6:

[0185] Final lateral deviation setting The constant offset d between the operating tool and the center line of the crop row, preset according to agronomic requirements. set and the regression point p return To the ideal line L g lateral distance d L Calculate the final lateral deviation control target:

[0186] Adaptive lateral deviation calculation:

[0187] =d set +α c (d L )

[0188] In the formula: This represents the final lateral deviation that the controller needs to track, in meters; d set α represents a fixed lateral offset preset according to agronomic requirements, in meters. c This represents an adaptive compensation coefficient, dynamically adjusted based on the confidence level of the regression decision model, with a value range of [0, 1]; d L Representing the regression point p return Reference line L for ideal crop row g The vertical distance, in meters;

[0189] Heading angle correction Δφ and approach speed recommendation v approach :

[0190] Δφ = φ return -φ {n-1}

[0191] v approach = f(S smooth (p return ), return )

[0192] Where: φ return φ represents the expected heading angle of the optimal regression point. {n-1} It is path point p {n-1} The heading, Δφ is the heading correction, v approach It is a score S based on the path curvature and model smoothness at the regression point. smooth Dynamically suggested velocity values ​​are used to achieve smooth docking; f(·) represents a defined function mapping relationship; S smooth (p return ) represents the optimal regression point p return The motion smoothness score at the point, return This represents the optimal regression point p. return The curvature of the planned path;

[0193] The regression decision model in this step is not a simple geometric calculator, but rather acts as an "arbitrator." Building upon the path provided by the integrated planning in step 4, which has already initially considered both objectives, this model utilizes a spatiotemporal attention mechanism to focus on the most reliable crop row features in complex visual scenes, and comprehensively evaluates motion smoothness and operational efficiency. Its "arbitration" is reflected in the fact that when multiple possible regression points exist on the terminal path generated by the integrated planning, the model does not choose the closest one, but rather, based on multi-objective scoring, adjudicates a point that achieves the optimal balance among visual alignment reliability, motion control stability, and overall operational continuity. This complements the global optimization in step 4: step 4 ensures the path "can return," while step 5 decides "how best to return," together solving the core challenge of accurate, smooth, and efficient regression after obstacle avoidance.

[0194] Decision result output:

[0195] This step outputs a regression decision and control parameter package, which includes:

[0196] Optimal regression point: p return = (x return , y return , φ return )

[0197] Precise control parameters: Final lateral deviation setpoint , heading angle correction Δφ, recommended approach speed v approach ;

[0198] Decision confidence: The overall value score V(p) output by the model. return The scores of the data and their sub-items are used for confidence-weighted control in step 6.

[0199] The specific process of step 6 is as follows: The obstacle avoidance path generated in step 4 is... planned The regression point p output in step 5 return The precise control parameters are converted into low-level control commands that can directly drive the actuators (steering and drive systems) of the agricultural machinery chassis. This step uses a specially designed hierarchical model predictive controller (H-MPC) to achieve high-precision and robust tracking of complex paths by the agricultural machinery, and ensures smooth and accurate alignment with crop rows at the end of the path, thus completing a closed loop from perception and decision-making to physical execution. The specific execution process of the agricultural machinery motion control and high-precision path tracking is as follows:

[0200] Control task integration and hierarchical strategy formulation

[0201] This step receives two levels of input tasks: one is the global obstacle avoidance geometry path described as a sequence of spatial points from step 4. planned); secondly, the precise regression pose and dynamic control parameters (p) from step 5, used to define the job recovery point. return , , Δφ, v approach The primary task of the control system is to integrate these two into a unified, time-continuous reference trajectory (χ). ref (t));

[0202] To achieve this goal, the system adopts a hierarchical control architecture:

[0203] Upper layer: Task fusion and reference trajectory generator; this module combines static path points with dynamic regression parameters; its core algorithm is based on time-varying parametric spline curve interpolation technology to generate a path from the current position of the agricultural machinery to the regression point p. return A continuous trajectory; this trajectory intersects with the Path in most sections. planned Overlapping, but dynamically adjusted at the end: the final lateral position depends on... Correction, terminal heading smoothly transitions to φ return And the velocity curve near the regression point is affected by v approach Guidance; This process ensures that the generated reference trajectory meets obstacle avoidance requirements geometrically and agronomic alignment requirements in the terminal state;

[0204] Lower layer: Adaptive Model Predictive Tracking Controller (AMPCC), which is the core actuator in this step. This controller solves a rolling time-domain optimization problem within each extremely short control cycle. Its goal is to calculate the value that most accurately tracks the upper-level reference trajectory χ, considering the kinematic constraints of the agricultural machinery, the physical limits of the actuator, and the real-time safety constraints from step three. ref The optimal steering and speed commands for (t);

[0205] The core operating principle of the adaptive model predictive tracking controller:

[0206] The optimization problem of AMPCC is built around a simplified kinematic model of the agricultural machinery; this model describes the mathematical relationships between the machinery's position, heading and steering angle, and speed; in each control cycle, the controller executes the following closed-loop process:

[0207] State prediction: Based on the measured state (position, heading, speed) and kinematic model of the agricultural machinery at the current moment, the controller predicts a series of trajectories that the agricultural machinery may travel under different control commands in the next few seconds;

[0208] Multi-objective cost evaluation: The controller constructs a multi-objective cost function to evaluate the merits of each predicted trajectory; this function mainly consists of three parts:

[0209] Path tracking error cost: Measured by the ratio of predicted trajectory to reference trajectory (χ²). ref The degree of deviation of (t) ensures that it follows the general direction;

[0210] Terminal alignment accuracy cost: Enhancing the prediction trajectory endpoint and regression point p return The degree of matching between position and heading;

[0211] Smoothness of control versus energy cost: penalizing sharp, frequent steering and speed changes;

[0212] Dynamic constraints are applied online: During the optimization process, the controller strictly applies two types of constraints: one is hard constraints, including the range of steering angles and the physical limits of the actuators at maximum speed; the other is soft constraints, mainly referring to dynamic safety distance constraints. This constraint transforms the obstacle position and velocity information provided in real time in step three into the minimum distance requirement that each point of the predicted trajectory must maintain from all obstacles.

[0213] Optimal command solution and output: Solve the above constrained optimization problem to obtain the optimal control command sequence in the future time domain, but only output the optimal control quantity at the first moment - that is, the instantaneous target steering angle and instantaneous target speed - to the steering system and drive system of the agricultural machinery;

[0214] Key Formula: Terminal Alignment Enhancement Term

[0215] To highlight the core invention of "regression alignment," a key terminal reinforcement term is designed into the cost function of AMPCC; this design aims to transform the intelligent decision result of step five into a strong guiding signal within the controller; its core idea is as follows:

[0216] At the end of the prediction time domain, the controller calculates a terminal cost; this cost is not simply a calculation of the deviation from a fixed point, but is defined by the following formula:

[0217] Terminal alignment cost:

[0218] J terminal = w y * (y N - y target ) 2 + w φ * (φ N - φ return ) 2

[0219] In the formula: J terminal This represents the terminal alignment cost; the smaller the value, the closer the predicted terminal state matches the expected regression state. N and φ NThese represent the predicted lateral position and heading angle of the agricultural machinery at the end of the prediction time domain, respectively; y target The desired lateral position of the terminal is calculated in step 5. (Final lateral deviation setpoint) determined in conjunction with the crop row model; φ return w represents the expected heading angle of the optimal regression point determined in step 5. y and w φ These are weighting coefficients, which are designed to adaptively increase as the predicted point approaches the terminal; this means that when the controller plans the trajectory, the closer it is to the endpoint, the more it will prioritize accurate alignment, thus naturally achieving a smooth switch in behavior from "path tracking as the main focus" to "terminal alignment as the main focus".

[0220] Step 7 involves creating and dynamically maintaining a Global Semantic Operation Map (GSTM) that comprehensively and continuously reflects the spatiotemporal evolution and task progress of field operations. This map, through incremental learning, integrates real-time information from perception, decision-making, and control processes. It not only accurately records "what has been done," but more importantly, intelligently predicts and manages "what needs to be done next," especially legacy tasks arising from obstacle avoidance. The specific execution flow for the incremental construction and task management of the global operation map is as follows:

[0221] Architecture and initialization of the global semantic job map:

[0222] The system constructs a multi-layered Global Semantic Operation Map (GSTM) as a digital twin of the entire operation task. This map is initialized at the start of the operation task based on prior information about farmland boundaries and pre-defined operation rows. GSTM is a composite data structure containing multiple semantic layers.

[0223] Work progress layer: This is the core layer, which records the work status of each piece of ground at high resolution. Its status is divided into four categories: "Not in operation" (waiting to be executed), "In progress" (current work line), "Completed" (work quality meets the standards), and the key "Areas temporarily skipped due to detour of fixed obstacles" (areas temporarily skipped due to detour of fixed obstacles).

[0224] Agronomic structure layer: Persistently stores the global crop row network model formed by repeated sensing and optimization in step two; this model not only includes the precise geometric center line of each crop row, but may also include attributes such as seedling condition and ridge height, forming the baseline framework for operation path planning;

[0225] Environmental memory layer: Persistently records the precise outlines, categories, and first discovery times of all static permanent obstacles identified and classified in step three;

[0226] Real-time incremental map updates based on actual work trajectories:

[0227] This step forms a tightly coupled closed loop with step 6 (motion control); as the agricultural machinery actually moves and operates under the control of step 6, the system uses high-precision positioning data to display the actual movement trajectory of the agricultural machinery and the physical coverage of the operating tools on the GSTM operation progress layer in real time.

[0228] Its working principle is as follows: The system continuously tracks the trajectory of the work tool reference point and calculates the ground area swept by the trajectory based on the known work width; then, in the GSTM work progress layer, the status of these swept grid cells is updated from "not worked" to "completed".

[0229] Intelligent detection and labeling of areas that have been bypassed but need to be filled:

[0230] Once the system determines in step 3 that it has encountered a static permanent obstacle and initiates a global replanning detour in step 4, this step immediately begins a crucial analysis process: the system compares and analyzes the "originally planned ideal operating path" (based on the crop row centerline in the agronomic structure layer) with the "actual obstacle avoidance detour path" (from the Path in step 4). planned By calculating the spatial difference between these two paths, the system can automatically identify and delineate multiple irregular closed regions.

[0231] The management logic that bypasses the pending replenishment state is as follows: The system dynamically decides when to replenish tasks based on the global operation progress, energy status, and agronomic requirements. The typical strategy is to schedule and execute the replenishment task after the current operation is completed, or when an idle time window appears in the operation queue.

[0232] These areas are immediately marked as "bypassed and awaiting completion". The system will create a structured "task unit to be completed" for each such area. This unit includes its geographical boundaries, the crop row to which it belongs, the ID of the obstacle bypassed, and the best time to complete the task based on agronomic requirements.

[0233] Dynamic scheduling and autonomous replanning of the global task queue:

[0234] GSTM is a dynamic task scheduler. The initial queue consists of a basic sequence of jobs line by line. When new "job units to be filled" are generated, they are automatically inserted into the appropriate position in this global queue.

[0235] Based on the results of this dynamic scheduling, this step will issue new, higher-level task instructions to step 4 (path planning module). Step 4 will then carry out a new round of path planning with the goal of completing the task, forming a complete high-level control loop from task discovery (obstacle avoidance) - task recording (labeling) - task scheduling - task execution (replanning).

[0236] This invention further includes step 8, which involves: deeply integrating the dedicated computing architecture of the Ascend series AI processors, the full-stack AI software toolchain, and the heterogeneous scheduling framework for real-time robot systems to perform extreme performance optimization and collaborative scheduling on all computationally intensive modules in steps 1 to 7, thereby constructing a millisecond-level real-time autonomous operation closed loop from perception, understanding, decision-making to control; the specific implementation of the software and hardware collaborative optimization and real-time closed loop based on the Ascend full-stack AI is as follows:

[0237] Hardware Foundation: Vehicle Deployment of the Ascend Heterogeneous Computing Platform

[0238] The core computing unit of the system adopts an in-vehicle computing platform that integrates the Ascend AI processor. This platform provides a heterogeneous computing architecture of "CPU + AI Core (NPU) + task scheduler". Among them, the CPU is responsible for complex logic control, task scheduling and traditional algorithms. The AI ​​Core (NPU) designed for deep learning uses its efficient tensor computing core and on-chip memory system to specifically carry out the forward inference tasks of the deep learning model in step 2 (obstacle detection, crop row segmentation) and step 5 (regression decision model).

[0239] Software Stack: Deep Optimization of the Full-Stack AI Toolchain

[0240] The system is developed and deployed using the Ascend full-stack AI software stack, achieving vertical integration of "application-framework-chip":

[0241] AI Framework Layer (MindSpore): Uses the Ascend native AI framework MindSpore to train and develop the deep learning models in steps 2 and 5;

[0242] Heterogeneous Computing Architecture Layer (CANN): As the heterogeneous computing engine of the Ascend processor, CANN undertakes the following key optimizations:

[0243] Model conversion and graph optimization: Using the ATC tool, the trained MindSpore model is converted into a high-efficiency offline model designed specifically for Ascend NPU; during this process, deep optimizations such as operator fusion, constant folding, and data layout transformation are performed;

[0244] High-performance operator library: Customized operators for nonmaximum suppression of the object detection network in step 2 and attention mechanism in step 5. The TBE provided by CANN is used to call the highly optimized basic operator library to achieve chip-level performance optimization;

[0245] Application Enablement Layer: Provides rich middleware and APIs to facilitate efficient invocation of the NPU for inference by upper-layer applications (step management program) and to manage the data flow between the CPU and NPU;

[0246] Deployment and performance optimization of core algorithm models:

[0247] Targeted deployment and optimization were carried out on the key algorithm modules of this invention:

[0248] Step 2: Deployment of the parallel perception model: Deploy the obstacle detection network and the crop row segmentation network as two independent inference tasks to be executed in parallel on the NPU; Utilize CANN's pipelined parallel technology to make the execution time of the two models overlap, and after the RGB image data is preprocessed by the CPU, it is transferred to the NPU memory in one go through the high-speed bus for the two models to share and use, and complete the complete scene parsing of a single frame image in less than 100 milliseconds;

[0249] Step 5: Lightweighting and quantization of the regression decision model: During the model training phase, network pruning and knowledge distillation techniques are used to compress the model size; during the deployment phase, INT8 quantization technology is used to convert the model weights and activation values ​​from FP32 to INT8 format; this process is completed using CANN's quantization tools and calibration dataset, improving the model inference speed by 2-3 times with minimal accuracy loss (<1%), and fulfilling the real-time requirement of intelligent decision-making for regression points within the path planning interval;

[0250] Construction of system-level real-time closed-loop pipeline

[0251] To ensure end-to-end real-time performance from sensing to control, the system designs a time-triggered heterogeneous scheduling pipeline. Its core idea is to modularize the tasks of different steps and allocate them to different computing units for parallel execution according to their computational characteristics and timing requirements, thus forming a pipeline.

[0252] End-to-end system delay model:

[0253] T total = T acq + max(T perception , T planning ) + T control + T overhead

[0254] In the formula: T total T represents the total end-to-end system latency from the acquisition of one image frame to the output of the corresponding control command, in seconds; acq Represents data acquisition and transmission latency, including camera exposure time and the time it takes for images to be transferred to host memory; T perception This represents the latency of the perception and understanding pipeline; including the model inference time on the NPU in step 2, and the lightweight fusion judgment time in step 3; due to the efficiency of the NPU, T perception Compressed to an extremely low level; Tplanning This represents the delay in the planning and decision-making pipeline; this includes the time spent on path planning in step 4 and regression decision-making in step 5; T control This represents the motion control solution delay, i.e., the solution time of the lower-level MPC; To verhead This represents system overhead, including time spent on task scheduling, memory copying, and synchronization waiting.

[0255] The system uses an asynchronous pipeline scheduler to enable T acq T perception T planning T control The process overlaps in time, resulting in a significantly lower actual latency than serial summation. This keeps Ttotal stably within 200-300 milliseconds, meeting the rapid response requirements of high-speed agricultural machinery to dynamic obstacles.

[0256] This case study uses an unmanned corn harvester as an example to illustrate the specific implementation process of the present invention. The agricultural machine needs to operate continuously in structured corn rows (row spacing 75 cm). The field environment includes static obstacles (power poles, unharvested field edges) and dynamic obstacles (other agricultural vehicles and workers traveling in the field).

[0257] First, based on steps one and two, the system performs real-time scene perception and understanding. The stereo binocular camera mounted on the front end of the agricultural machinery synchronously acquires RGB images and raw parallax data at a frequency of 15Hz. The specific implementation process on the Ascend processor is as follows: During the development phase, we trained two lightweight models using the MindSpore framework. The obstacle detection network uses the YOLO-SE model, an improvement on the YOLOv5s architecture, trained on COCO and a self-built farmland obstacle dataset, enabling it to recognize "pedestrians," "vehicles," "pole-like objects," and "ditches." The crop row segmentation network uses DeepLabV3+MobileNetV2 as the backbone network, trained on a cornfield image segmentation dataset. After training, the Ascend Model Conversion Tool (ATC) is used to convert the weight files of the two models into offline models in .om format that can run efficiently on the Ascend 310 AI processor. During deployment, the system directly sends the RGB image tensors from step one into the NPU memory through the runtime interface of Ascend CANN (heterogeneous computing architecture). The CANN scheduler loads two .om models into the NPU cores, enabling true hardware-level parallel inference: one NPU core executes YOLO-SE, outputting bounding boxes with categories; the other core simultaneously executes DeepLabV3+, outputting pixel-level segmentation masks. Simultaneously, the CPU calculates point clouds from the depth map and combines them with the YOLO-SE detection boxes, using coordinate back-projection formulas to calculate the 3D coordinates (X, Y, X) of each obstacle. c , Y c Zc The entire process is completed within 65 milliseconds per frame, ensuring the real-time nature of the perception.

[0258] Then, based on steps three and four, the system makes decisions and plans. Assuming the agricultural machinery is currently operating along the 5th row, after integrating the perception results in step three, it is determined that there is a static "pole-like object" spanning the path 15 meters ahead (TTC calculates it as urgent), and simultaneously, there is a slow-moving "vehicle" to the left front. The system marks the pole as requiring "global replanning" and the moving vehicle as requiring "local detour." The specific implementation of the path planning algorithm in step four is as follows: After receiving the "global replanning" instruction and the obstacle map, the planning module starts the improved hybrid A* search algorithm. The cost function of this algorithm is C... total Calculated on the CPU. The obstacle cost C is included. a Based on the repulsive potential field of all obstacles, the regression cost C r The target point for calculation is a preset regression point located 20 meters behind the obstacle in the original crop row. Considering the turning radius constraint of the agricultural machinery, the algorithm searches in the state space, ultimately generating a preliminary path that first veers right out of the crop row, around the utility pole, then circles back from the left, and finally points to the regression point. The collaborative optimization on the Ascend platform is reflected in the fact that while the CPU is performing path search, the NPU is not idle but is processing the perception task of the next frame in parallel. This heterogeneous task pipeline greatly improves overall efficiency. The preliminary path is then smoothed by a B-spline curve fitter on the CPU to generate the final executable path. planned It outputs the coordinates of its endpoint region. The total time from triggering planning to generating a smooth path is 120 milliseconds.

[0259] Building upon this, step five, intelligent regression decision-making, is initiated. The image features of the terminal region of the planned path, the geometric model of the crop rows, and the status of the agricultural machinery are encapsulated into tensors and input into the "Regression Decision Model Based on Spatiotemporal Attention Mechanism and Multi-Objective Reinforcement Learning" deployed on the Ascend NPU. This is converted to INT8 precision .om format using an ATC tool. The network, through its multi-head attention module, automatically focuses on the region with the clearest visual texture of the crop rows after bypassing the utility pole, suppressing interference from adjacent weeds, and outputs an optimal, precise regression point p. return And its heading. Then, based on the deviation of this point from the ideal crop line, the system adaptively calculates the lateral control parameters. =0.05 meters (meaning that the agricultural machinery is required to maintain a 5-centimeter offset from the centerline of the crop row) and heading correction.

[0260] Subsequently, steps six and seven are executed and updated. The Model Prediction Controller (MPC) in step six runs on the CPU, and it operates on a path... planned As a reference trajectory, and using p as the output of step fivereturn and For the strongly constrained terminal objective, the optimal steering and speed commands are determined to control the agricultural machinery to precisely execute obstacle avoidance and return maneuvers. The agricultural machinery actually avoids the utility pole and smoothly and accurately re-embeds itself in the fifth row of crop ridges 8 seconds later, with a lateral error of less than 8 centimeters and a heading error of less than 3 degrees. Simultaneously, the global operation map module in step seven automatically calculates and marks a rectangular area that has been "avoided but not yet operated" based on the fixed position of the utility pole and the actual detour trajectory of the agricultural machinery, and adds it to the supplementary operation task queue.

[0261] In subsequent operations, the value of the closed-loop system of this invention is fully realized. After the agricultural machinery completes the harvesting of the main areas, the task manager extracts the remaining supplementary tasks from the queue. Based on this, the planner in step four generates a dedicated path to the area, and the agricultural machinery automatically travels to and completes the harvesting of the missed area, ensuring 100% operation coverage. Supported by the full-stack AI capabilities of the Ascend processor, the entire system maintains an end-to-end average processing latency of less than 220 milliseconds, successfully achieving continuous, safe, and high-quality operation in dynamic and complex environments.

[0262] Through the comprehensive application of this invention, this case has achieved remarkable results: compared with traditional unmanned agricultural machinery operating along preset paths, operational efficiency has increased by approximately 25% due to reduced ineffective idling and repetitive work, crop row alignment accuracy has improved by over 70%, and field losses caused by missed operations have been reduced to near zero. Simultaneously, the system's response success rate to dynamic obstacles reached 99.2%, fully demonstrating the immense value of this invention in enhancing the autonomy, operational quality, and overall operational efficiency of unmanned agricultural machinery.

[0263] As can be seen from the above embodiments, the method and system for dynamic obstacle avoidance and operation path recovery of unmanned agricultural machinery based on depth vision provided by the present invention has the advantages of rich perception dimensions, intelligent scene understanding, integrated decision-making and planning, and efficient hardware and software collaboration. It can be widely applied to various autonomous field operation scenarios such as sowing, plant protection, and harvesting.

[0264] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0265] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for dynamic obstacle avoidance and operation path recovery of unmanned agricultural machinery based on depth vision, characterized in that, Includes the following steps: Step S1: Synchronously acquire RGB images and depth information of the field scene using a stereo vision system to construct a synchronous data stream containing texture and three-dimensional structure; Step S2: Based on the heterogeneous computing platform, the synchronous data stream is processed in parallel to realize the deep dynamic perception of multiple types of obstacles and the real-time identification and geometric modeling of crop row boundaries, and output a structured scene cognition containing three-dimensional information of obstacles and geometric model of crop rows. Step S3: Integrate the structured scene cognition with the agricultural machinery's own status to make intelligent judgments on the path obstruction status, and output a judgment decision package that includes the obstruction status, obstacle classification, and target crop row reference line; Step S4: When the path is determined to be blocked, path planning is initiated with 'safe obstacle avoidance' and 'job return' as the collaborative optimization objectives. Based on the judgment decision package, obstacle avoidance detour paths are dynamically generated by solving the integrated optimization problem that integrates the two objectives, so as to achieve integrated protection of safety and job continuity. Step S5: Optimize the end of the obstacle avoidance path. Using an intelligent decision-making model based on an attention mechanism, determine the optimal regression point under the constraint of the dual objectives through arbitration, and calculate the control parameters required to achieve precise alignment with the crop row. Step S6: Based on the obstacle avoidance path and the control parameters, generate low-level control commands to control the steering and drive system of the agricultural machinery to perform motion and path tracking, so that the agricultural machinery can achieve precise alignment with the crop row at the optimal return point; Step S7: During the operation, incrementally build and dynamically maintain a global semantic operation map containing the operation progress layer, agronomic structure layer and environmental memory layer for closed-loop task management. By comparing the planned path with the actual path, automatically identify and manage the 'already bypassed but to be filled' areas caused by obstacle avoidance to ensure full operation coverage.

2. The method according to claim 1, characterized in that, In step S2, the heterogeneous computing platform is a heterogeneous computing platform integrating a central processing unit (CPU) and a dedicated neural network processor (NPU); the parallel processing includes: A first deep learning model and a second deep learning model, after model compression, are run in parallel on the NPU; the first deep learning model is an object detection network for obstacle detection, and the second deep learning model is an encoder-decoder network for semantic segmentation. The first deep learning model is used to detect obstacles based on RGB images, and combines depth information to convert the detection results into three-dimensional spatial coordinates and estimate the motion speed; The second deep learning model is used to perform semantic segmentation of crop rows based on RGB images, and to perform geometric fitting on the segmentation results to obtain a mathematical model of the crop row boundary lines.

3. The method according to claim 1, characterized in that, In step S3, the intelligent determination of the path obstruction state includes: Unify all perceived information into the vehicle coordinate system; The estimated collision time between the computer and the obstacle, based on the predicted collision time model; The estimated collision time is compared with a dynamic safety threshold to determine whether immediate thwarting is possible. Based on the semantic category and motion state of the obstacle, it is classified into dynamic temporary obstacle, static permanent obstacle, or unknown obstacle, and the corresponding handling strategy is marked.

4. The method according to claim 3, characterized in that, In step S4, the bi-objective optimization path planning is achieved by solving a constrained optimization problem. The objective function J is defined as the weighted sum of the obstacle avoidance cost term Ca and the regression alignment cost term Cr: J = wa*Ca + wr*Cr; where wa and wr are the weight coefficients of the two terms, respectively.

5. The method according to claim 4, characterized in that, The optimization problem is solved using an improved hybrid A* algorithm, whose heuristic function h(n) incorporates the cost h to the geometric objective point. goal (n) and the estimated alignment cost h align (n): h(n) = h goal (n)+γ*h align (n), where γ is the weight coefficient of the alignment heuristic.

6. The method according to claim 1, characterized in that, In step S5, the intelligent decision-making model is a regression decision-making model based on spatiotemporal attention mechanism and multi-objective reinforcement learning, which includes an encoder unit for feature encoding, a spatiotemporal attention unit for focusing key information, and a multi-objective evaluation unit for outputting multi-dimensional evaluation scores. The process of determining the optimal regression point includes: Define candidate regression windows at the end of the obstacle avoidance and detour path; Construct a multivariate input tensor that includes visual features, geometric constraints, agricultural machinery status, and path shape; The candidate regression points are evaluated using the regression decision model, and the visual-geometric alignment score, motion smoothness score, and work efficiency score are output. The comprehensive value of each candidate point is calculated based on the scores, and the point with the highest comprehensive value is selected as the optimal regression point.

7. The method according to claim 1, characterized in that, In step S6, a hierarchical model predictive controller is used to achieve path tracking and terminal alignment. The cost function of the controller includes a terminal alignment cost term, which is used to enhance the matching degree between the predicted trajectory endpoint and the optimal regression point in terms of position and heading.

8. The method according to claim 1, characterized in that, In step S7, the global semantic job map is a multi-layered fusion structure, including at least: The work progress layer is used to record the ground status as "not in operation", "in progress", "completed" and "avoided and awaiting rework". The agronomic structure layer is used to persistently store the global crop row network model; An environmental memory layer is used to persistently record information about identified static, permanent obstacles. The method further includes: by comparing the original planned operation path with the actual detour path, automatically identifying and marking the "detoured and to be completed" area, generating the task unit to be completed, and inserting it into the global task queue for dynamic scheduling.

9. A dynamic obstacle avoidance and operation path recovery system for unmanned agricultural machinery based on depth vision, characterized in that, The system for implementing the method of any one of claims 1-8 comprises: The synchronous sensing module is configured to synchronously acquire RGB images and depth information of the field scene through a stereo vision system mounted on agricultural machinery, and construct a synchronous data stream; The scene understanding module is configured to perform parallel processing on the synchronous data stream based on a heterogeneous computing platform and output structured scene cognition. The intelligent judgment module is configured to integrate the structured scene cognition with the agricultural machinery status to judge the path obstruction status and classify obstacles; The path planning module is configured to perform path planning with 'safe obstacle avoidance' and 'job return' as the collaborative optimization objectives when the path is blocked. By solving the integrated optimization problem that combines the two objectives, an obstacle avoidance detour path is generated. The regression decision module is configured to optimize the end of the obstacle avoidance path. It uses an intelligent decision model based on an attention mechanism to arbitrate under the constraints of the two objectives to determine the optimal regression point and solve the control parameters. The motion control module is configured to generate control signals for driving the steering and travel mechanisms of the agricultural machinery based on the obstacle avoidance path and the control parameters, so as to control the agricultural machinery to track the path and achieve precise alignment at the optimal return point; The task management module is configured to incrementally build and maintain a global semantic job map with a multi-layered structure to achieve closed-loop task management and automatically identify and manage 'already bypassed and waiting to be filled' areas; wherein, the scene understanding module, intelligent judgment module, path planning module, regression decision module and motion control module constitute a real-time processing closed loop.

10. The system according to claim 9, characterized in that, The heterogeneous computing platform is an in-vehicle heterogeneous computing unit that integrates a central processing unit (CPU) and a dedicated neural network processor (NPU). The scene understanding module is deployed on the NPU of this platform and includes a parallel-running obstacle depth dynamic perception submodule and a crop row real-time recognition and geometric modeling submodule.

Images