Automatic row control system of harvester

Abstract

The invention relates to the field of mechanical control, in particular to an automatic row control system of a harvester, which comprises an information acquisition module used for acquiring an image of a working area in front of the harvester along the advancing direction of the harvester and acquiring vehicle driving information of the harvester; the information processing module is used for acquiring a target crop row center line from the image of the working area in front of the harvester, and constructing a vehicle track state vector before fusion according to the target crop row center line; according to the vehicle driving information and the vehicle track state vector before fusion, a vehicle track state vector after fusion is obtained, and the vehicle track state vector comprises a transverse deviation after fusion of the harvester relative to the center line of the target crop row and a course deviation after fusion of the harvester relative to the target crop row; generating a steering control instruction according to the fused transverse deviation and the fused course deviation; and the vehicle control module is configured to control the harvester to run according to the steering control instruction. The operation of the harvester is controlled according to the fused multi-source information, and the problem of low control precision of the harvester is solved.

Description

Technical Field

[0001] This application relates to the field of mechanical control, and in particular to an automatic row alignment control system for a harvester. Background Technology

[0002] The ability of a harvester to align rows directly affects the quality and efficiency of crop harvesting. In traditional operations, harvesters are usually operated manually to align rows. However, long-term, high-intensity operations in complex environments during crop maturity can easily lead to problems such as row deviation and missed harvesting. This results in high labor intensity for drivers and a high rate of harvest loss.

[0003] To reduce driver stress, most harvesters currently employ automatic row alignment, controlled by a single vision system, satellite positioning, or a single sensor. However, these control methods are prone to instability in sensing and deviation from row alignment when facing complex field conditions due to the limited information available, resulting in low control precision for the harvester. Summary of the Invention

[0004] This application provides an automatic row alignment control system for harvesters, which at least solves the problem of low control accuracy of harvesters in related technologies.

[0005] In a first aspect, embodiments of this application provide an automatic row alignment control system for a harvester, comprising: The information acquisition module is configured to collect images of the working area in front of the harvester along the direction of travel of the harvester, and to acquire vehicle driving information of the harvester; The information processing module is configured to acquire the center line of the target crop row from an image of the working area in front of the harvester; Based on the centerline of the target crop row, a vehicle trajectory state vector before fusion is constructed; the vehicle trajectory state vector before fusion includes the initial lateral deviation of the harvester relative to the centerline of the target crop row and the initial heading deviation of the harvester relative to the target crop row; Based on the vehicle driving information and the vehicle trajectory state vector before fusion, the vehicle trajectory state vector after fusion is obtained; the vehicle trajectory state vector after fusion includes the fusion lateral deviation of the harvester relative to the center line of the target crop row and the fusion heading deviation of the harvester relative to the target crop row. Based on the fused lateral deviation and the fused heading deviation, a steering control command is generated; The vehicle control module is configured to control the harvester to travel along the center direction of the target crop row according to the steering control command.

[0006] In some embodiments, the information processing module is further configured to: Extract the connected components corresponding to each main crop row from the image of the working area in front of the harvester to form a set of connected components; Each connected component is refined to obtain its corresponding skeleton point set; Select one from the entire set of skeleton points as the target crop row; Through coordinate transformation, the skeleton points corresponding to the target crop row in the image coordinate system are converted into the center line of the target crop row in the vehicle coordinate system.

[0007] In some embodiments, the information processing module is further configured to: The crop row region and the background region are distinguished from the image of the working area in front of the harvester to form a crop row binary segmentation mask; Construct structural elements, defining them to elongate along the direction of the crop rows and narrow along the direction perpendicular to the crop rows; The morphological processing results are obtained by performing directional opening and closing operations on the binary segmentation mask and structuring element of the crop row. Connectivity labeling is performed on the morphological processing results to obtain the connected components corresponding to the crop rows; Set the connected component constraint for the main crop row, and use the connected component corresponding to the crop row that satisfies the connected component constraint for the main crop row as the connected component corresponding to the main crop row.

[0008] In some embodiments, the information processing module is further configured to: According to the opening and closing operation calculation formulas, directional opening and closing operations are performed to obtain morphological processing results. The formula for calculating the opening operation is as follows: ; in, The crop row is a binary segmentation mask. For the aforementioned structural element, For corrosion operation, For expansion operation, The result of the opening operation; The formula for calculating the closing operation is as follows: ; in, This refers to the morphological processing result.

[0009] In some embodiments, the information processing module is further configured to: Construct a state equation characterizing the motion state of the harvester and a multi-source measurement equation characterizing the vehicle's driving information. The multi-source measurement equation includes a measurement vector and a measurement matrix. Within the extended Kalman filter framework, based on the vehicle trajectory state vector before fusion and the state equation, the prior state at the current moment and the prior covariance matrix at the current moment are generated. Construct a modal weight matrix, and generate a Kalman gain matrix based on the modal weight matrix, the measurement vector, and the prior covariance matrix at the current time. The fused vehicle trajectory state vector is calculated based on the prior state at the current moment, the Kalman gain matrix, the measurement vector, and the measurement matrix.

[0010] In some embodiments, the information processing module is further configured to: The fused vehicle trajectory state vector is calculated according to the calculation formula of the fused vehicle trajectory state vector. The formula for calculating the fused vehicle trajectory state vector is as follows: ; in, This refers to the fused vehicle trajectory state vector. This refers to the prior state at the current moment. Let Kalman gain matrix be the given value. For the measurement vector, The measurement matrix is ​​described above.

[0011] In some embodiments, the information processing module is further configured to: The prior state at the current moment is obtained by iterative calculation according to the calculation formula of the prior state at the current moment. The formula for calculating the prior state at the current moment is: ; in, Here is the state transition matrix. To control the input matrix, To control the input vector, This represents the posterior state of the previous time step; In the iterative calculation process, the initial value of the posterior state at the previous moment is the vehicle trajectory state vector before fusion.

[0012] In some embodiments, the information processing module is further configured to: Before calculating the merged vehicle track state vector, the pre-merge vehicle track state vector is set. The initial lateral deviation of the harvester relative to the centerline of the target crop row is set as the first component of the pre-merge vehicle track state vector, and the initial heading deviation of the harvester relative to the target crop row is set as the second component of the pre-merge vehicle track state vector. After the fusion of the vehicle track state vector is calculated, the first component of the fusion of the vehicle track state vector is extracted as the lateral deviation of the harvester relative to the center line of the target crop row. The second component of the fused vehicle trajectory state vector is extracted as the heading deviation of the harvester relative to the target crop row.

[0013] In some embodiments, the information processing module is further configured to: The formula for calculating the Kalman gain matrix is ​​set as follows: ; in, Let Kalman gain matrix be the given value. Let be the prior covariance matrix at the current moment. For the measurement matrix, To measure the noise covariance matrix; Let be the modal weight matrix.

[0014] In some embodiments, the information processing module is further configured to: In the vehicle coordinate system, the distance from the center point of the harvester to the center line of the target crop row is calculated to obtain the initial lateral deviation; In the vehicle coordinate system, the difference between the vehicle's heading angle and the direction angle of the centerline of the target crop row is calculated as the initial heading deviation.

[0015] Compared to related technologies, the automatic alignment control system for harvesters provided in this application solves the problem of low control precision of harvesters by integrating multi-source information and conforming to actual operating scenarios. It achieves coordinated control of lateral deviation and heading deviation, thereby improving the control precision and stability of the harvester.

[0016] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a structural block diagram of an automatic row alignment control system for a harvester according to an embodiment of this application; Figure 2 This is a structural block diagram of a harvester using an automatic row alignment control system according to an embodiment of this application; Figure 3This is a structural block diagram of the information processing module in an automatic row alignment control system for a harvester according to an embodiment of this application.

[0018] In the diagram: 101, Information Acquisition Module; 102, Information Processing Module; 103, Vehicle Control Module; 201. Forward-facing binocular camera; 202. Positioning unit; 203. Left side lateral distance sensor; 204. Wheel speed sensor; 205. Inertial measurement unit; 206. Right side lateral distance sensor; 301. Line visual perception unit; 302. Line structure and centerline extraction unit; 303. Multi-source line track fusion and deviation calculation unit; 304. Automatic line control command generation unit. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0020] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0021] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0022] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0023] When harvesting row-sown crops such as corn and wheat, the harvester's ability to align with the rows greatly affects the quality and efficiency of the harvest. In traditional operations, the driver visually observes the crop rows and row spacing, and manually adjusts the harvester's trajectory by manipulating the steering wheel to ensure it travels along the center of the crop rows.

[0024] The central direction of a crop row refers to the direction of the central line formed along the overall extension trend of the crop row.

[0025] The row centerline is a virtual straight line or curve located at the geometric center between the two edges of the crop planting strip and extending along the crop growth direction. In actual operation, this line is usually generated by fitting crop feature points collected by sensors through an algorithm, representing the ideal travel path that the harvester should follow.

[0026] If the harvester deviates from the row centerline, the header, reel, or cutter will not be aligned with the crop, resulting in uneven cutting, missed cutting, or double cutting. It will also cause crop on one side of the harvester to be crushed, scratched, or knocked down. Especially in densely planted crops such as corn and wheat, even a deviation of only a few centimeters can cause significant yield loss. If the harvester is consistently deviated from the row centerline, the header may become overloaded on one side, leading to uneven cutting, poor straw crushing, and increased grain breakage.

[0027] During the crop maturity stage, due to the combined effects of factors such as dry and curled leaves, increased straw residue, exposed bare soil between rows, local lodging, and weed interference, drivers find it difficult to adjust the driving direction in a timely and accurate manner under long-term high-intensity operation conditions. This can easily lead to situations such as deviation from the designated path, missing sections, missed harvests, and crop damage. Not only does this result in excessive labor intensity for drivers, but it also increases the harvest loss rate.

[0028] To reduce driver workload and improve row alignment accuracy, various automatic row alignment methods have been proposed in existing technologies. These include single-vision-based automatic row alignment methods, which acquire crop images using a forward-facing camera and extract crop row directions using image processing techniques such as edge detection, texture analysis, or simple thresholding, or fit an approximate straight line of the crop row using a small number of feature points, and then generate steering control values ​​based on this. Other methods involve introducing deep learning networks to perform crop row target detection or segmentation. However, these methods rely on single-vision information and are sensitive to changes in lighting, occlusion noise, and plant morphology. Especially in the context of mature, withered leaves, straw residues, bare soil, and lodged plants, the forward-facing vision struggles to stably extract the true center structure of the crop row. The crop row area is easily missegmented or broken, row lines are prone to breakage or cross-row adhesion, and the extraction of the row center line is unstable, leading to jitter and failure in automatic row alignment control.

[0029] In addition, there are automatic row alignment methods based on satellite positioning, which mainly utilize the harvester's position and heading information provided by GNSS or RTK, and use a pre-planned driving trajectory or operation route as a reference to perform path tracking control of the harvester. This type of method is effective in regular, neat plots with fixed row spacing. However, due to the lack of direct perception of the geometric structure of the crop rows ahead, when there are situations such as sowing deviation, uneven row spacing, missing plants, replanting, or turning at the edge of the field, the pre-planned driving trajectory often deviates from the center of the actual crop rows. This can easily lead to the harvester following the pre-planned driving trajectory but not along the center of the crop rows, making it difficult to guarantee row alignment accuracy and harvest quality.

[0030] Existing technologies also include row alignment methods based on a single structure or distance sensor. For example, these methods detect the relative position of crop rows using mechanical contact rods, lidar, or ultrasonic ranging, or measure the lateral distance between the header and the crop rows by arranging multiple distance sensors on the header, thereby correcting the header position or making simple directional adjustments to the harvester. These methods are typically sensitive to local positional changes, but their sensing range is limited, making it difficult to reflect the overall geometry of multiple crop rows within a certain distance in a timely manner. Furthermore, some solutions only adjust the lateral position of the header, failing to establish a model of the overall machine's movement and lacking systematic control over the machine's movement. Therefore, their row alignment effectiveness is limited when the crop row structure is complex or the field contains curved rows.

[0031] Existing technologies include multimodal or multisensor fusion control methods, but most focus on longitudinal control issues such as adaptive adjustment of header height, or simply superimpose sensor signals in localized areas. They lack a unified modeling and adaptive fusion mechanism for crop row geometry, implement posture, and lateral row distance, resulting in insufficient utilization of multi-source information. Regarding automatic row control strategies, they generally rely on single deviations or simplified geometric relationships, failing to adequately consider the coordinated control of lateral and heading deviations. This leads to poor adaptability to complex field conditions and low accuracy and smoothness in row control.

[0032] To address the aforementioned issues, this application proposes an automatic row alignment control system for harvesters. This system can stably extract the centerline of crop rows in complex field environments with mature crops, comprehensively utilize multi-source information for row tracking, and, based on this, achieve automatic row alignment of the harvester through joint control of lateral deviation and heading deviation. This further improves alignment accuracy and robustness, and reduces operational risks such as staggered harvesting, missed harvesting, and crop damage.

[0033] like Figure 1 As shown in the figure, this application provides an automatic row alignment control system for a harvester, including: The information acquisition module 101 is configured to collect images of the working area in front of the harvester along the direction of travel of the harvester, and to acquire vehicle driving information of the harvester.

[0034] After the information acquisition module 101 is started, it synchronously collects images of the crop row in front of the harvester and vehicle driving information such as vehicle speed and steering angle, realizing the real-time acquisition of multi-source basic data, providing data support for subsequent deviation analysis, and avoiding the limitations of a single information source.

[0035] Information processing module 102 is configured as follows: Obtain the center line of the target crop row from an image of the working area in front of the harvester; Based on the centerline of the target crop row, construct the vehicle trajectory state vector before fusion; Specifically, the vehicle trajectory state vector before fusion is used to represent the harvester's deviation from the target crop row centerline and the harvester's speed, and can be set as follows: .

[0036] in, This represents the initial lateral deviation of the harvester relative to the centerline of the target crop row. This represents the initial heading deviation of the harvester relative to the centerline of the target crop row. For the longitudinal speed of the vehicle, This represents the vehicle's yaw rate.

[0037] Based on the vehicle driving information and the vehicle trajectory state vector before fusion, the fused vehicle trajectory state vector is obtained. Specifically, the fused vehicle trajectory state vector refers to the lateral deviation, heading deviation, longitudinal speed, and yaw rate of the harvester relative to the centerline of the target crop row at the current moment after fusing the vehicle driving information.

[0038] Specifically, the fused vehicle trajectory state vector can be represented as: .

[0039] From the fused vehicle trajectory state vector, extract the fused lateral deviation of the harvester relative to the centerline of the target crop row and the fused heading deviation of the harvester relative to the target crop row.

[0040] Specifically, when constructing the vehicle track state vector before fusion, the initial lateral deviation of the harvester relative to the centerline of the target crop row is set as the first component of the vehicle track state vector before fusion, and the initial heading deviation of the harvester relative to the centerline of the target crop row is set as the second component of the vehicle track state vector before fusion. The composition of the state vector will not change arbitrarily during the calculation process. Therefore, extracting the first component of the vehicle track state vector after fusion is the lateral deviation of the harvester relative to the centerline of the target crop row after fusion, and extracting the second component of the vehicle track state vector after fusion is the heading deviation of the harvester relative to the target crop row after fusion.

[0041] To facilitate the use of lateral and heading deviations, a selection matrix can be introduced: .

[0042] The lateral deviation of the harvester relative to the centerline of the target crop row after fusion can be expressed as: .

[0043] The combined heading deviation of the harvester relative to the target crop row can be expressed as: .

[0044] The third component of the fused vehicle trajectory state vector is extracted, which is the fused vehicle longitudinal velocity. The fused vehicle longitudinal velocity can be expressed as: .

[0045] The fourth component of the fused vehicle trajectory state vector is extracted, which is the fused vehicle yaw rate. The fused vehicle yaw rate can be expressed as: .

[0046] Based on the fused lateral deviation and fused heading deviation, steering control commands are generated.

[0047] The information processing module 102 extracts the center line of the target crop row from the image, constructs a pre-fusion state vector representing the deviation between the harvester and the center line of the target crop row, and then combines it with vehicle driving information to complete vector fusion. This process corrects the deviation between the harvester and the center line of the target crop row through multi-source information complementarity, effectively reducing the error of single image recognition or single vehicle state detection, making the extracted lateral deviation and heading deviation closer to the actual state, so as to further improve the accuracy of deviation parameter detection.

[0048] The vehicle control module 103 is configured to control the harvester to travel along the center direction of the target crop row according to the steering control command.

[0049] Steering control commands are generated based on deviation parameters and executed by vehicle control module 103, causing the harvester to fine-tune to align with the centerline, forming a closed loop of deviation detection, command generation, and direction correction. This ensures timely response when deviations occur, guarantees tracking stability, and avoids missed or repeated harvesting.

[0050] During continuous operation, the control system of this application constructs a complete closed-loop control framework to ensure that control commands are generated in a timely manner and the driving direction is corrected when deviations occur, and the driving state is always maintained along the track. This realizes the automatic tracking function of the harvester, freeing it from dependence on manual driving, reducing the labor intensity of operators, and avoiding driving deviations caused by fatigue and judgment errors during manual driving, thereby further improving harvesting efficiency.

[0051] The information acquisition module 101 is used to collect images of the working area in front of the harvester and vehicle driving information. The vehicle driving information includes the harvester's position information, attitude information, longitudinal speed, and lateral distance information between the harvester and adjacent crop rows.

[0052] like Figure 2 As shown, the harvester includes a header, cab, main frame, left lateral probe, right lateral probe, canopy front edge support, drive wheels, etc. The cab houses the electrical control box or equipment box.

[0053] The information acquisition module 101 includes a forward-facing binocular camera 201, a positioning unit 202, an inertial measurement unit 205, a wheel speed sensor 204, a left-side lateral distance sensor 203, and a right-side lateral distance sensor 206.

[0054] The forward-facing binocular camera 201 is fixed to the middle of the front end of the bracket extending 200-400mm outward from the front edge of the cab roof. The installation height is about 1.5-2.5m higher than the upper surface of the cutting platform. The optical axis of the forward-facing binocular camera 201 is slightly downward along the direction of vehicle movement.

[0055] The positioning unit 202 includes a positioning antenna and a positioning receiver. The positioning antenna is installed at the rear of the cab roof near the lateral geometric center of the entire machine to obtain a wide overhead field of view. The positioning receiver is installed in the electrical control box or equipment box inside the cab and is connected to the positioning antenna on the roof via a coaxial cable.

[0056] The inertial measurement unit 205 is fixedly installed on the rigid crossbeam in the middle of the main frame located between the engine and the grain bin, and its coordinate axis direction is consistent with the vehicle coordinate axis direction.

[0057] The wheel speed sensor 204 is installed near the axle of the harvester's drive wheel and acquires the wheel speed signal by being arranged in correspondence with the speed measuring gear ring or encoder set on the wheel hub.

[0058] The left-side lateral distance sensor 203 is installed at the front end or below the transverse probe on the left side of the front main frame. The probe is 0.3-0.8m above the ground and the detection direction is towards the adjacent crop row on the left.

[0059] The right-side lateral distance sensor 206 is installed at the front end or below the transverse probe on the right side of the front main frame. The probe is 0.3-0.8m above the ground and the detection direction is towards the adjacent crop row and plant strip on the right.

[0060] The information processing module 102 is installed in the electrical control box at the rear or side of the driver's cab. It is electrically connected to the forward binocular camera 201, positioning unit 202, inertial measurement unit 205, wheel speed sensor 204, left-side lateral distance sensor 203 and right-side lateral distance sensor 206 via wiring harnesses, and is connected to the vehicle control module 103 via bus.

[0061] The vehicle control module 103 is installed in the original steering wheel position in the cab or near the front axle steering mechanism, depending on the implementation. For example, it may be near the electric steering wheel or the front axle steering device, where the steering device is mechanically and hydraulically connected to the steering mechanism.

[0062] like Figure 3 As shown, the information processing module 102 includes a line visual perception unit 301, a line structure and centerline extraction unit 302, a multi-source line track fusion and deviation calculation unit 303, and an automatic line control command generation unit 304.

[0063] The automatic alignment control command generation unit 304 is connected to the multi-source track fusion and deviation calculation unit 303 and the vehicle control module 103 respectively. The automatic alignment control command generation unit 304 is used to construct the alignment control quantity based on the lateral deviation and heading deviation. It uses a fuzzy-PID composite control or model predictive control algorithm that combines lateral deviation and heading deviation to calculate the steering control command and send the steering control command to the vehicle control module 103. The vehicle control module 103 controls the harvester to adjust the front wheel angle so that the harvester can automatically align with the center of the target crop row.

[0064] The row vision perception unit 301 is used to process images of the working area in front of the harvester and output a binary segmentation mask for crop rows.

[0065] The row structure and centerline extraction unit 302 is connected to the row visual perception unit 301. The row structure and centerline extraction unit 302 is used to perform directional morphological opening and closing operations, connected component analysis and skeleton refinement according to the binary segmentation mask to obtain the skeleton line of the connected component of the main crop row. The Hough transform or robust straight line and broken line fitting algorithm is used to construct the crop row centerline model and transform it to the vehicle coordinate system.

[0066] In some embodiments, the row visual perception unit 301 in the information processing module 102 is further configured as follows: Extract the connected components corresponding to each main crop row from the image of the working area in front of the harvester to form a set of connected components.

[0067] Each connected component is refined to obtain its corresponding skeleton point set.

[0068] Specifically, for each main crop row, the connected component... A thinning algorithm is used to iteratively remove edge pixels, resulting in a skeleton set that is one pixel wide. .

[0069] Here, thin() is the skeleton thinning operator, used to thin a blocky connected region into a single-pixel wide skeleton. For the first The set of skeletal points for each crop row preserves the overall direction and connectivity of the row.

[0070] Select one from the entire set of skeleton points as the target crop row.

[0071] Through coordinate transformation, the skeleton points corresponding to the target crop row in the image coordinate system are converted into the center line of the target crop row in the vehicle coordinate system.

[0072] For skeleton line point set Perform Hough transform line fitting and express the line in polar coordinates: .

[0073] By analyzing all skeleton points Cumulative voting, selecting the parameter pair with the highest vote value. As the first The optimal polar coordinate parameters for the center line of the row.

[0074] Through coordinate transformation, the line is transformed from polar coordinates to the slope-intercept form in the image coordinate system. The slope-intercept expression can be set as follows: .

[0075] in, These are the pixel coordinates of the skeleton points in the image coordinate system. For the first The normal distance of a straight line in polar coordinates For the first The angular parameters of a straight line in polar coordinates. Let be the set of parameters with the largest voting value in the Hough space. For the first The slope of the center line of each crop row in the image coordinate system. For the first The intercept of the center line of each crop row in the image coordinate system. No. The formula for calculating the slope of the center line of a crop row in the image coordinate system is: .

[0076] No. The formula for calculating the intercept of the centerline of a crop row in the image coordinate system is: .

[0077] Combined with binocular depth maps With camera calibration parameters, certain feature points on the skeleton line can be transformed from the image coordinate system to the camera coordinate system, and then to the vehicle coordinate system.

[0078] The camera calibration parameters include the equivalent focal length and depth values ​​of the forward-facing binocular camera 201.

[0079] Specifically, set skeleton points corresponding depth In the camera coordinate system, coordinates This can be further expressed as: , .

[0080] The three-dimensional coordinates of the skeleton points in the camera coordinate system can be represented as: .

[0081] 3D coordinates of skeleton points in the vehicle coordinate system Represented as: .

[0082] in, These are the pixel coordinates of the skeleton points on the image plane. Let these be the coordinates of the camera's principal point. The equivalent focal length of the forward-facing binocular camera 201 in the horizontal direction. The equivalent focal length of the forward-facing binocular camera 201 in the vertical direction. The depth value in the camera coordinate system can be obtained by converting parallax or depth maps. Let be the rotation matrix from the camera coordinate system to the vehicle coordinate system. This is the translation vector of the origin of the camera coordinate system in the vehicle coordinate system.

[0083] and The camera mounting parameters are obtained through a one-time extrinsic parameter calibration calculation. The specific calculation method includes: arranging a calibration plate or several known 3D feature points in the vehicle coordinate system; using the camera's intrinsic parameters and the pixel coordinates of the calibration plate feature points in multiple frames of images; and employing a nonlinear least squares optimization method to solve for the rigid body pose of the camera coordinate system relative to the vehicle coordinate system, thereby obtaining... and .

[0084] During system operation, and The calibration results, used as a fixed reference for the rigid mounting relationship between the camera and the vehicle, do not change over time.

[0085] Camera intrinsic parameters are obtained through conventional camera calibration. For the skeleton points corresponding to the crop rows selected by voting and transformed to the vehicle coordinate system, perform least-squares line fitting again to obtain the expression of the target crop row centerline in the vehicle coordinate system: .

[0086] in, Along the direction of the vehicle's movement, It refers to the horizontal direction.

[0087] Understandably, to select target crop rows more accurately, the selection method can be set as follows: connected components of each main crop row The skeleton point set was obtained after refinement. ,right After performing Hough transform or least squares fitting, multiple crop row centerlines are obtained. Then, considering the harvester's cutting width and the machine's geometry, a preliminary target crop row centerline is selected from these centerlines. For example, the row closest to the vehicle's longitudinal symmetry plane can be chosen, or the geometric midline of two adjacent rows can be used as the preliminary target crop row centerline. Finally, least squares fitting is performed again on the set of skeleton points corresponding to the selected preliminary target row in the vehicle coordinate system to obtain the final target crop row centerline used to control the harvester, and the expression of the target crop row centerline in the vehicle coordinate plane is obtained. .

[0088] First, the image of the working area in front of the harvester is processed to extract connected components corresponding to multiple crop rows, eliminating small connected components such as weeds, effectively removing background noise, reducing the influence of interference factors, and improving the accuracy and precision of crop row area recognition. The connected components are then refined into skeletons to obtain the corresponding skeleton point set. Redundant pixels are removed while retaining the central contour, making the central features of the crop rows more prominent and accurately locating the central contour of the crop rows. Following the selection method, one crop row is chosen as the target crop row, achieving accurate target selection in multi-crop scenarios. The image coordinate system skeleton points are converted to vehicle coordinate system points through coordinate transformation, and then linear fitting is used to obtain the centerline of the target crop row. This centerline directly reflects the relative position of the harvester body and the crop row, providing accurate basic data for initial deviation calculation.

[0089] In some embodiments, the row visual perception unit 301 in the information processing module 102 is further configured as follows: The crop row region and the background region are distinguished from the image of the working area in front of the harvester to form a binary segmentation mask for the crop row.

[0090] Specifically, the forward-facing binocular camera 201 acquires images of the working area in front of the harvester at a fixed frame rate, including left and right view images. The camera's intrinsic and extrinsic parameters and disparity-depth mapping relationship are obtained through binocular calibration. The left view image is input into a deep learning network that integrates row-level detection and segmentation functions. The network outputs a probability map of each pixel in the image belonging to a crop row, and then a binary segmentation mask is obtained based on a threshold.

[0091] Furthermore, the probability graph can be set as follows: .

[0092] Furthermore, the binary segmentation mask can be set as follows: .

[0093] in, The original image for the left eye. This is a row-level detection and segmentation network built on the YOLOv11 and Mask R-CNN architecture. For network parameters, For pixels The probability of belonging to a crop row. This is a binary segmentation mask for crop rows, where 1 represents the crop row region and 0 represents the background region. This is the segmentation probability threshold.

[0094] YOLOv11 is a real-time object detection model that is widely used for object detection, instance segmentation, image classification, pose estimation, and bounding box detection.

[0095] Mask R-CNN is a deep learning architecture used for pixel segmentation.

[0096] Clearly distinguishing between crop row areas and background areas provides a data foundation for subsequent morphological processing and further refinement of crop rows, thereby improving the accuracy of crop row identification.

[0097] Furthermore, by calculating the disparity between two cameras, the corresponding depth map can be obtained simultaneously. This provides auxiliary information for subsequent geometric transformations.

[0098] Construct structural elements, defining them to elongate along the direction of the crop rows and narrow along the direction perpendicular to the crop rows.

[0099] The morphological processing results are obtained by performing directional opening and closing operations on the binary segmentation mask and structuring element of the crop row.

[0100] Connectivity labeling is performed on the morphological processing results to obtain the connected components corresponding to the crop rows.

[0101] Set the connectivity constraints for the main crop row. The connectivity of the crop row that satisfies the connectivity constraints for the main crop row is used as the connectivity of the main crop row.

[0102] By setting the connectivity constraints of the main crop rows, connectivity regions with small areas, insufficient vertical length, and shapes that do not conform to the long and thin strip pattern are filtered out, while the connectivity regions of the main crop rows are retained, thus improving the accuracy of the main crop row screening.

[0103] Connectivity labeling is performed on the morphological processing results to obtain connected components. .

[0104] For the Connected components, according to , , Calculations are performed, and the results are then filtered.

[0105] Since the connected component of the main crop row needs to satisfy certain constraints, the constraints of the connected component of the main crop row are set as follows: , , .

[0106] in, For the first Connected components, The area of ​​the connected component is the number of pixels. Let be the projection length of the connected component along the vertical direction. Let the width and height of the bounding rectangle of the connected component be given. This is an area threshold used to filter connected components with low noise. This is a vertical length threshold used to ensure that connected components have sufficient row-wise extension. and The upper and lower limits of the aspect ratio of the circumscribed rectangle are used to define the shape of the long and thin strip.

[0107] Several conditions satisfying the connected component constraint of the main crop row The connected components corresponding to the rows selected as the principal crop are denoted as the set of connected components. .

[0108] During the maturity stage, crops commonly suffer from problems such as withered and curled leaves, messy straw or stem residues, exposed bare soil between rows, and localized lodging. These conditions generate a lot of noise, affecting the identification of crop rows in images. To solve these problems, in some embodiments, the row visual perception unit 301 in the information processing module 102 is further configured as follows: According to the opening and closing operation calculation formulas, directional opening and closing operations are performed to obtain morphological processing results. The formula for calculating the opening operation is as follows: ; in, Use a binary segmentation mask for crop rows. As a structural element, For corrosion operation, For expansion operation, The result of the opening operation; The formula for calculating the closing operation is as follows: ; in, This is the result of morphological processing.

[0109] To suppress noise from withered leaves, straw residues, and bare soil in mature crop scenes, directional morphological structural elements that are elongated along the row direction and narrowed perpendicular to the row direction are used to modify the binary segmentation mask. Directional opening and closing operations are performed, and directional morphological filtering is applied to remove noise spots with small area and irregular shape, and to fill small gaps caused by missing plants, making the crop rows smoother and enhancing the connectivity and noise resistance of the crop rows.

[0110] Specifically, setting structural elements It elongates along the direction of the crop rows and narrows in the direction perpendicular to the rows.

[0111] Define the erosion operation as: .

[0112] Define the expansion operation as: .

[0113] First, perform directional erosion on the binary segmentation mask, then perform directional dilation to obtain the opening operation result. Then, a closing operation is performed again to obtain the final morphological processing result. .

[0114] The result of the opening operation can be represented as: .

[0115] The final morphological processing result can be expressed as: .

[0116] in, The result of the opening operation is mainly used to remove small noise. The result of morphological processing is the final binary graph after opening and closing combination operations, which is used for connected component analysis.

[0117] In some embodiments, the automatic line control instruction generation unit 304 in the information processing module 102 is further configured to: The initial control output is calculated using the PID control algorithm based on the formula for calculating the initial control output.

[0118] Specifically, the formula for initially controlling the output is set as follows: .

[0119] in, To initially control the output, This is the lateral deviation proportional gain. This is the proportional gain for the heading deviation. For integral gain, For differential gain, For the lateral deviation after fusion, To account for the heading deviation after merging, To control the cycle, For the lateral deviation integral, For the current moment, Let be any discrete time in the range 0-k.

[0120] By setting fuzzy parameter increments, the initial control output is adjusted, and finally, steering control commands are generated.

[0121] Specifically, the fuzzy parameter increment includes the lateral deviation proportional parameter increment. Increment of heading deviation proportional parameter Increment of integral parameters Increment of differential parameter .

[0122] Steering control commands can be expressed as: .

[0123] in, This is the adjusted lateral deviation proportional gain. This is the adjusted heading deviation proportional gain. The adjusted integral gain. The adjusted differential gain, This is a steering control command used to control the steering of the harvester's rear wheels in order to adjust the steering angle of the harvester's front wheels.

[0124] The adjusted lateral deviation proportional gain is the sum of the lateral deviation proportional parameter increment and the lateral deviation proportional gain.

[0125] The adjusted heading deviation proportional gain is the sum of the heading deviation proportional parameter increment and the heading deviation proportional gain.

[0126] The adjusted integral gain is the sum of the integral parameter increment and the integral gain.

[0127] The adjusted differential gain is the sum of the differential parameter increment and the differential gain.

[0128] The fuzzy parameter increment adjustment rule mainly takes into account the lateral deviation. Heading deviation Horizontal deviation change rate Rate of change of heading deviation In addition, factors such as vehicle longitudinal speed and structural complexity.

[0129] First, establish the error-related adjustment rules.

[0130] For example, when and Larger and and A larger value indicates a large and rapidly changing deviation, requiring quick correction. and Take a positive value and make it appropriately larger to increase the proportional gain. Increase appropriately to suppress overshoot. Generally, the value should be reduced or kept small to avoid excessive integration under large errors, which could lead to oscillations.

[0131] when and Smaller but with persistent bias. and When the value is close to 0, it indicates that the line is close to the correct alignment but there is a steady-state error. In this case, appropriately increasing the value... Enhance the integral process to eliminate residual bias. and Only minor adjustments were made.

[0132] when and and and When all values ​​are relatively small, it indicates that the alignment has been basically stabilized. At this point, the increments of each fuzzy parameter approach 0, causing the lateral deviation proportional gain, heading deviation proportional gain, integral gain, and derivative gain to return to or approach the reference values, maintaining smooth steering output.

[0133] Secondly, considering the vehicle's longitudinal speed and structural complexity, at high vehicle speeds, under the same error conditions, to avoid excessive control causing serpentine behavior at high speeds, the following measures should be taken to reduce... and The range should be appropriately increased. To increase damping and reduce To maintain stable control.

[0134] When the row structure is highly complex, such as in cases of significant row curvature, discontinuous skeleton, or low segmentation confidence, the system should possess a certain degree of correction capability under complex row shapes, while ensuring no oscillations, and appropriately improve [its performance]. and At the same time, and Set a reasonable range to prevent jitter caused by excessive integration or differentiation in complex scenarios.

[0135] Multiple gain parameters can be independently adjusted through their corresponding fuzzy parameter increments, further enabling precise optimization of deviation characteristics under different operating scenarios.

[0136] Compared to traditional fixed-parameter PID control, the introduction of fuzzy parameter increments in this application enables dynamic adaptive adjustment of the gain parameter, improving the robustness of the control algorithm and enabling it to cope with deviation changes under different operating environments. The combination of PID and fuzzy adjustment makes steering control more precise and smooth, reducing overshoot and oscillation problems caused by fixed parameters, avoiding severe shaking of the harvester during turning, protecting the machine structure and improving operational safety.

[0137] In addition, the setting of fuzzy parameter increments is flexible and can be adapted and optimized according to different crop types and field conditions, further broadening the applicable scenarios of the system.

[0138] The correspondence and adjustment logic between fuzzy parameter increments and gain parameters make the fuzzy adjustment process quantifiable and traceable, avoiding the influence of subjectivity in fuzzy inference on the control effect. The adjustment logic, based on gain parameters and further refined through fuzzy parameter increments, ensures that the adjusted gain parameters do not deviate from a reasonable range, balancing control flexibility and stability. Compared to directly resetting parameters, incremental adjustment better inherits the adaptability of gain parameters, reducing post-adjustment fluctuations. The calculation formula for steering control commands remains consistent with the initial control output, ensuring the continuity of control logic, reducing system computational complexity, improving command generation efficiency, and meeting the real-time control requirements of the harvester.

[0139] The multi-source track fusion and deviation calculation unit 303 is connected to the row structure and centerline extraction unit 302 and the information acquisition module 101, respectively. The multi-source track fusion and deviation calculation unit 303 is used to perform credibility assessment and adaptive weighted fusion of vehicle driving information under a unified time and coordinate reference. Based on the extended Kalman filter or weighted least squares method, it calculates the track state of the harvester relative to the target crop row and finally calculates the lateral deviation of the harvester relative to the centerline of the target crop row and the heading deviation of the harvester relative to the target crop row.

[0140] In some embodiments, the multi-source track fusion and deviation calculation unit 303 in the information processing module 102 is further configured as follows: A state equation characterizing the harvester's motion state and a multi-source measurement equation characterizing the vehicle's driving information are constructed. The multi-source measurement equation includes a measurement vector and a measurement matrix.

[0141] Within the extended Kalman filter framework, the prior state and prior covariance matrix at the current moment are generated based on the fusion of the vehicle trajectory state vector and state equation before the current moment.

[0142] The prior state at the current moment is used to characterize the state estimate predicted solely based on the posterior state vector of the previous moment and the harvester's dynamic model, before considering the use of current moment measurement information.

[0143] The prior covariance matrix is ​​used to characterize the error distribution characteristics of the prior state at the current moment, reflecting the degree of influence of factors such as motion model accuracy, process noise, and historical state errors on the uncertainty of the current predicted state.

[0144] Construct the modal weight matrix, and generate the Kalman gain matrix based on the modal weight matrix, the measurement vector, and the prior covariance matrix at the current time.

[0145] Based on the prior state at the current moment, the Kalman gain matrix, the measurement vector, and the measurement matrix, the fused vehicle trajectory state vector is calculated.

[0146] By constructing state equations to describe the motion state of the harvester, and by constructing multi-source measurement equations to associate measured data such as vehicle speed and steering angle with state vectors, mathematical modeling of the harvester's motion law and measured data is realized, providing a theoretical framework for fusion and laying the foundation for deep fusion of multi-source information.

[0147] Within the extended Kalman filter framework, the prior state and prior covariance matrix at the current moment are predicted based on the fusion of the pre-state vector and state equation. This prediction process uses motion laws to predict state changes, providing a benchmark for subsequent corrections and further ensuring the continuity of state estimation.

[0148] By constructing a modal weight matrix, weights can be dynamically allocated based on the reliability of the measurement data, thereby improving the reliability of the fusion results.

[0149] By correcting the prior state using Kalman gain, the logic control of prediction followed by correction is completed, ensuring that the state estimation responds to changes in the fuselage in real time, providing accurate data for deviation extraction, and also ensuring the accuracy of subsequent control.

[0150] In some embodiments, the state equation characterizing the harvester's motion state can be expressed as: .

[0151] in, The state transition matrix is ​​determined based on the vehicle kinematics model and the sampling period. To control the input matrix, To control the input vector, This is process noise.

[0152] The state transition matrix can be derived and discretized based on the vehicle trajectory state vector and the planar kinematic model of the harvester. The selection method includes the following steps.

[0153] First, based on the continuous-time kinematic model, the state evolution relationship is determined, and the vehicle trajectory state vector is defined as: .

[0154] in, This refers to the lateral deviation of the vehicle's centerline relative to the centerline of the target crop row. For heading deviation, For the longitudinal speed of the vehicle, Let be the vehicle's yaw rate. Combining commonly used planar vehicle kinematics, under the assumptions of small angles and small sampling periods, we can obtain an approximate form of a set of continuous-time state equations, for example: ; ; ; It can be seen that the lateral deviation changes with the product of the vehicle's longitudinal speed and heading deviation. The heading deviation is obtained by integrating the vehicle's yaw rate. The vehicle's longitudinal speed and yaw rate are approximately constant within one sampling period.

[0155] Next, the state equations are discretized according to the sampling period and extracted. .

[0156] Specifically, let the system sampling period be... The above continuous-time state equation is applied at adjacent sampling times. By performing a discrete approximation between them, we can obtain the following: .

[0157] Write it in matrix form: The state transition matrix can be read directly. Each element in the middle.

[0158] For example, in a simplified implementation, Option: .

[0159] The first row reflects the change of lateral deviation with the vehicle's longitudinal speed and heading deviation; the second row reflects the evolution of heading deviation by the integral of yaw rate; and the third and fourth rows indicate that the speed and yaw rate remain approximately constant within a sampling period, or are only affected by process noise.

[0160] It should be noted that in actual engineering projects, different machine models, sampling cycles, and vehicle speed ranges will result in varying performance. The specific values ​​can be derived from the above discrete kinematic relationships and vehicle geometric parameters, and appropriately tuned using experimental data. During calculation, the values ​​can be... It can be regarded as an approximately constant matrix, or a time-varying matrix that changes slowly with vehicle speed, etc.

[0161] Control input matrix The elements include control quantities such as steering angle, which are steering control commands used to control the steering angle of the front wheels.

[0162] At the same time, in order to make the state equations have a certain degree of universality, The control inputs included can be adjusted according to the structure and type of the harvester. For example, for harvesters using front and rear wheel coordinated steering or articulated steering, the rear wheel steering angle command and / or articulated steering angle command can be used as... Components, by Establish its influence relationship on track status; for harvesters with tracked or differential steering chassis, the speed difference command of the left and right drive wheels or left and right tracks can be used as one of the control inputs to reflect the influence of differential steering on the vehicle's yaw rate and track status; the pump displacement adjustment command of the hydrostatic drive system, the drive torque of the engine and motor, or the target driving speed command can also be used as control inputs to characterize the effect of longitudinal speed adjustment on the vehicle's longitudinal speed and track evolution.

[0163] Furthermore, depending on the chassis structure, rear wheel steering angle commands and / or articulated steering angle commands can be incorporated into the control input vector. As the corresponding component.

[0164] For a specific machine model, one or more steering command components are used. For example, when the harvester chassis has a rear-wheel steering mechanism, the control input vector is the rear-wheel steering angle command. When the harvester chassis has an articulated steering mechanism, the control input vector is the articulated steering angle command. When the harvester chassis has both a rear-wheel steering mechanism and an articulated steering mechanism, the control input vector is both the rear-wheel steering angle command and the articulated steering angle command.

[0165] Control input vector It is the control command issued to the vehicle control module at a certain moment, obtained directly from the automatic travel control command generation unit 304 or from the vehicle's upper-level control system and driver operation signals acquired through an interface. Control input vector The components are input quantities, including control commands directly related to vehicle steering and driving status, such as front wheel steering angle commands, longitudinal driving speed commands, and differential or articulated steering control quantities.

[0166] Differential or articulated steering control parameters include commands for the speed difference between the left and right drive wheels and commands for the articulated steering angle.

[0167] Understandably, controlling the input vector These are the actual control commands, while the control input matrix... In the state equation, the control input vector is... Mapped to changes in track status, this describes the extent to which a control command affects state variables such as lateral deviation, heading deviation, vehicle longitudinal speed, and vehicle yaw rate.

[0168] For example, the effect of front wheel steering angle command on heading deviation and lateral deviation will be reflected in... The corresponding row and column elements, while the control quantities related to longitudinal velocity are reflected in It acts on the row of the velocity component in the state vector.

[0169] Therefore, in the state equation, It reflects the control input vector The control commands represented also reflect the control input matrix. This characterizes the intensity and direction of the influence of the control command on each state component.

[0170] Process noise is an equivalent random term in the state-space model used to characterize the incompleteness of the vehicle motion model and external disturbances. It includes: deviations between the vehicle kinematics and dynamics model and the actual motion, such as the inconsistency between the actual lateral offset and heading change caused by tire / track slippage, uneven ground, and slope changes and the ideal model; differences between the actual steering response and the command caused by nonlinear factors such as dead zone, hysteresis, and backlash of the steering actuator; changes in the center of mass position and inertial parameters caused by elastic vibration of the vehicle structure and load changes; and the influence of small disturbances not explicitly modeled, such as small crosswinds and local soil hardness changes, on the track.

[0171] In the calculation process, process noise is generally assumed to be a random vector with zero mean.

[0172] The multi-source measurement equation characterizing vehicle driving information can be expressed as: .

[0173] in, For measurement vectors, For the measurement matrix, For measuring noise.

[0174] The measurement matrix is ​​used to map the state vector to the observation space of the positioning unit 202, inertial measurement unit 205, wheel speed sensor 204, left side lateral distance sensor 203, and right side lateral distance sensor 206 in the information acquisition module 101.

[0175] Specific mapping methods include: based on the actual observations and track state quantities of each sensor. The geometric and kinematic relationships between the states are used to formulate the measurement function. The measurement function is then linearized to the first order near the current state estimate, and its partial derivatives with respect to the state vector are calculated to form the measurement matrix. .

[0176] The velocity and yaw rate observed by the inertial measurement unit 205, the wheel speed observed by the wheel speed sensor 204, the lateral distance between the vehicle and the crop row on the left side observed by the left-side lateral distance sensor 203, and the lateral distance between the vehicle and the crop row on the right side observed by the right-side lateral distance sensor 206 are respectively reflected in the above partial derivative relationships. The measurement matrix is ​​finally obtained from the corresponding rows.

[0177] When using the data acquired by various devices in the information acquisition module, time synchronization and coordinate unification processing are required.

[0178] In some embodiments, the multi-source track fusion and deviation calculation unit 303 in the information processing module 102 is further configured as follows: The prior state at the current moment is obtained by iterative calculation according to the calculation formula of the prior state at the current moment. Within the extended Kalman filter framework, and combining the state equation characterizing the harvester's motion state, the formula for calculating the prior state at the current moment can be expressed as: ; in, Here is the state transition matrix. To control the input matrix, To control the input vector, This represents the posterior state of the previous time step; In the iterative calculation process, the initial value of the posterior state of the previous time step is the vehicle trajectory state vector before fusion.

[0179] The prior state at the current moment is the foundation for calculating the fused vehicle trajectory state vector. The calculation method for the prior state at the current moment is clarified, and its initial values ​​for iteration are derived from data provided by image detection, making the iteration closer to the actual state. Utilizing the calculation of the prior state at the current moment enables rapid convergence, further improving the convergence speed and the real-time performance of state estimation.

[0180] In addition, the prior covariance matrix can also be calculated during the calculation process. The formula for calculating the prior covariance matrix is ​​as follows: .

[0181] in, Let be the process noise covariance matrix.

[0182] Noise covariance matrix It can be obtained in the following ways.

[0183] First, based on the lateral deviation, heading deviation, vehicle longitudinal velocity, vehicle yaw rate, and vehicle kinematic / dynamic model, the maximum possible changes in lateral deviation, heading deviation, vehicle longitudinal velocity, and vehicle yaw rate within a single sampling period are estimated, giving an initial diagonal covariance matrix, for example: .

[0184] in Based on historical experience and model analysis, and then by collecting actual driving data, the lateral deviation, heading deviation, vehicle longitudinal velocity, and vehicle yaw rate in the prior state are compared with the lateral deviation, heading deviation, vehicle longitudinal velocity, and vehicle yaw rate in the fused vehicle trajectory state vector. The variance of the prediction error is statistically analyzed. Perform iterative adjustments to bring the prediction error closer to the set value. They are basically the same in magnitude. The final determination will be made. As the process noise covariance matrix for this machine model and operating condition, it can be regarded as a constant or slowly changing parameter during operation.

[0185] In some embodiments, the multi-source track fusion and deviation calculation unit 303 in the information processing module 102 is further configured as follows: The fused vehicle trajectory state vector is calculated according to the formula for calculating the fused vehicle trajectory state vector. The formula for calculating the fused vehicle trajectory state vector is as follows: .

[0186] in, This is the fused vehicle trajectory state vector. This represents the prior state at the current moment. Here is the Kalman gain matrix. For measurement vectors, This is the measurement matrix.

[0187] The calculation logic of the fused state vector was clarified, making the fusion process of the extended Kalman filter quantifiable and reproducible, thus reducing the technical implementation difficulty. Through calculation... The measurement residuals are obtained to reflect the deviation between the prior state and the actual state. The correction weights of the residuals are then dynamically adjusted using the Kalman gain matrix to ensure the accuracy of the correction process. When the reliability of the measurement data is high (i.e., the Kalman gain is large), the residual correction weights are high, and the prior state aligns more closely with the measured data. When the reliability of the measurement data is low (i.e., the Kalman gain is small), the residual correction weights are low, prioritizing the preservation of the reasonableness of the prior state. This approach balances the continuity of the predicted state with the authenticity of the measured data, avoiding error amplification caused by relying solely on either prediction or measured data, and further improving the accuracy and stability of the fused state vector.

[0188] At the start of the iteration process, the vehicle trajectory state vector before fusion at the first moment is substituted into the calculation formula of the prior state at the current moment. Calculations are performed to obtain the prior state at the second time step, based on the vehicle trajectory state vector before fusion at the first time step. The prior state at the second time step is then substituted into the calculation formula for the fused vehicle trajectory state vector to obtain the posterior state at the second time step, which is the fused vehicle trajectory state vector at the second time step. When calculating the fused vehicle trajectory state vector at the third time step, the fused vehicle trajectory state vector at the second time step is again substituted into the calculation formula for the prior state. Then, the calculation is repeated until the fused vehicle trajectory state vector at time k is obtained.

[0189] In some embodiments, the multi-source track fusion and deviation calculation unit 303 in the information processing module 102 is further configured as follows: Before calculating the vehicle trajectory state vector after fusion, the vehicle trajectory state vector before fusion is set. The initial lateral deviation of the harvester relative to the center line of the target crop row is set as the first component of the vehicle trajectory state vector before fusion, and the initial heading deviation of the harvester relative to the target crop row is set as the second component of the vehicle trajectory state vector before fusion. Specifically, the vehicle trajectory state vector before fusion can be set as follows: .

[0190] in, This represents the initial lateral deviation of the harvester relative to the centerline of the target crop row. This represents the initial heading deviation of the harvester relative to the centerline of the target crop row. For the longitudinal speed of the vehicle, This represents the vehicle's yaw rate.

[0191] After calculating the fused vehicle track state vector, the first component of the fused vehicle track state vector is extracted as the lateral deviation of the harvester relative to the center line of the target crop row.

[0192] The second component of the fused vehicle trajectory state vector is extracted as the heading deviation of the harvester relative to the target crop row.

[0193] The component order of the fused state vector was clearly defined, with lateral and heading deviations set as the first two components. Standardized parameter extraction rules were established to avoid extraction chaos caused by ambiguous component definitions, thus standardizing data extraction and further improving system stability. A direct component index extraction method was adopted to extract deviations directly from the fused vehicle trajectory state vector without complex calculations, enabling real-time deviation extraction and meeting the requirements of real-time control.

[0194] In some embodiments, the multi-source track fusion and deviation calculation unit 303 in the information processing module 102 is further configured as follows: In the vehicle coordinate system, the distance from the center point of the harvester to the center line of the target crop row is calculated to obtain the initial lateral deviation; In the vehicle coordinate system, the difference between the vehicle's heading angle and the direction angle of the target crop row centerline is calculated as the initial heading deviation.

[0195] The initial lateral deviation of the harvester relative to the centerline of the target crop row is obtained by calculating the shortest distance from the origin of the vehicle coordinate system to the centerline of the target crop row in the vehicle coordinate system.

[0196] The formula for calculating the initial heading deviation of the harvester relative to the centerline of the target crop row is: .

[0197] in, , This refers to the vehicle's heading angle.

[0198] The equation of the straight line of the target crop row centerline in the vehicle coordinate system get.

[0199] The calculation benchmarks and specific methods for initial deviation parameters are further clarified. Initial lateral and heading deviations are calculated according to the established principles, ensuring that the acquisition logic for these deviations is clear and reproducible, thus avoiding parameter errors caused by inconsistent benchmarks. The precisely defined initial lateral and heading deviations cover the core position and attitude deviations of tracking, providing comprehensive foundational data for the pre-fusion state vectors and laying a reliable foundation for subsequent fusion, ensuring the accuracy of the fused parameters.

[0200] In some embodiments, the multi-source track fusion and deviation calculation unit 303 in the information processing module 102 is further configured as follows: The Kalman gain matrix is ​​calculated using the formula for calculating the Kalman gain matrix. The formula for calculating the Kalman gain matrix is ​​set as follows: .

[0201] in, Here is the Kalman gain matrix. Let be the prior covariance matrix at the current moment. For the measurement matrix, To measure the noise covariance matrix. This is the modal weight matrix.

[0202] Measurement noise covariance matrix It can be obtained in the following ways.

[0203] First, set the measurement vector as follows: .

[0204] Preferably, the Set as a diagonal matrix: .

[0205] Among them, the variance term in the formula The measurement noise intensity corresponding to the measured quantity, specifically, from left to right, represents the visual bias. The velocity corresponding to the velocity observed by the inertial measurement unit 205 Wheel speed corresponding The lateral distance corresponding to the left row The lateral distance corresponding to the right row The initial values ​​of the measured noise intensity were obtained through static calibration and bench or low-speed constant-speed driving tests.

[0206] For example, under conditions where the vehicle is stationary or traveling at a constant speed in a straight line, the time series of visual deviation, speed, wheel speed, left-side lateral distance, and right-side lateral distance observed by the inertial measurement unit 205 are recorded, and the variance of its zero-mean fluctuation is calculated as... The initial estimate is then used. Based on this, combined with field operation data, statistical analysis is performed on the filtered residuals, and the corresponding values ​​of a certain measurement are appropriately amplified or reduced. This makes the residual distribution after filtering consistent with... The settings are a good match.

[0207] The modal weight matrix assigns different weights to the data observed by the positioning unit 202, inertial measurement unit 205, wheel speed sensor 204, left side lateral distance sensor 203, and right side lateral distance sensor 206, and can be adjusted according to visual confidence, RTK solution status, and the validity of the lateral distance signal.

[0208] For example, the higher the visual confidence level or the more effective the signal, the greater its corresponding weight.

[0209] The modal weight matrix mainly uses different numerical weights to adjust the influence of various sensor observations in the measurement equation. By setting different values ​​in the matrix, the different weights assigned to the left and right distance sensors and wheel speed sensors can be reflected.

[0210] Constructing measurement vectors Then, a diagonal modal weight matrix is ​​introduced: .

[0211] Each of them The weight of the corresponding sensor mode is indicated by the value; the larger the value, the higher the reliability of the sensor at the current moment.

[0212] In extended Kalman filtering or weighted least squares, for measurement residuals Weighting, or adjusting the measurement noise covariance Scaling is applied so that modes with larger weights have a greater impact when calculating the Kalman gain or fitting the cost function, while modes with smaller weights have a weaker impact.

[0213] For example, through Different values ​​are assigned to the diagonal elements to reflect the different weights of the left and right distance sensors and wheel speed sensors. Under one operating condition, the modal weight matrix can be set as follows: .

[0214] If a deterioration in the distance sensor signal quality is detected, the distance will be reduced accordingly. and The modal weight matrix can be updated as follows: .

[0215] At this point, the weight of the residuals of the left and right distances in the filtering process automatically decreases, reducing their impact on the lateral deviation state and preventing individual sensor malfunctions from dragging down the overall estimation.

[0216] The calculation logic of the Kalman gain matrix was clarified, making the allocation of correction weights more scientific and interpretable, and avoiding the subjectivity of weight setting. The introduction of a priori covariance and measurement noise covariance matrices enabled dynamic adaptive adjustment of the correction weights, allowing for optimization of weight allocation based on real-time state uncertainty and measurement noise levels, thus improving the reliability of the fusion results. The inclusion of the modal weight matrix made the fusion of multi-source measurement data more flexible, allowing for weight adjustments based on the performance differences of each measurement source under different operating scenarios, further broadening the system's adaptability. Synchronous correction of multi-dimensional state variables was achieved through matrix operations, taking into account the accuracy of multiple key parameters such as lateral deviation and heading deviation, providing comprehensive and accurate basic data for subsequent control command generation.

[0217] In some embodiments, during the process of generating the fused vehicle trajectory state vector, a posterior covariance matrix is ​​also generated. It can be represented as: .

[0218] in, It is an identity matrix.

[0219] The posterior covariance matrix can be used to calculate the Kalman gain matrix during the generation of the prior state at the next time step.

[0220] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0221] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A harvester automatic row control system, characterized by, The information acquisition module is configured to collect images of a working area in front of the harvester along a direction in which the harvester travels, and to acquire vehicle driving information of the harvester. The information processing module is configured to: obtain a target crop row center line from the images of the working area in front of the harvester; construct a fusion front vehicle trajectory state vector based on the target crop row center line, the fusion front vehicle trajectory state vector including an initial lateral deviation of the harvester relative to the target crop row center line and an initial heading deviation of the harvester relative to the target crop row; obtain a fusion rear vehicle trajectory state vector according to the vehicle driving information and the fusion front vehicle trajectory state vector, the fusion rear vehicle trajectory state vector including a fusion rear lateral deviation of the harvester relative to the target crop row center line and a fusion rear heading deviation of the harvester relative to the target crop row; generate a steering control instruction according to the fusion rear lateral deviation and the fusion rear heading deviation; and the vehicle control module is configured to control the harvester to travel along the center of the target crop row according to the steering control instruction. The information processing module is further configured to:

2. An automatic row control system for a harvester as claimed in claim 1, wherein, extract a connected domain corresponding to each main crop row from the images of the working area in front of the harvester to form a connected domain set; refine each connected domain to obtain a corresponding skeleton point set; select one of all the skeleton point sets as a target crop row; convert the skeleton points corresponding to the target crop row in an image coordinate system into a target crop row center line in a vehicle coordinate system through coordinate transformation. The information processing module is further configured to:

3. An automatic row control system for a harvester as claimed in claim 2, wherein, distinguish a crop row region and a background region from the images of the working area in front of the harvester to form a crop row binary segmentation mask; construct a structure element, define the structure element to be elongated along a crop row direction and narrowed along a direction perpendicular to the crop row; perform directional opening and closing operations on the crop row binary segmentation mask and the structure element to obtain a morphological processing result; perform connected domain labeling on the morphological processing result to obtain a connected domain corresponding to a crop row; set a main crop row connected domain constraint condition, and set a connected domain corresponding to a crop row that satisfies the main crop row connected domain constraint condition as a connected domain corresponding to a main crop row. The information processing module is further configured to:

4. An automatic row control system for a harvester as claimed in claim 3, wherein, perform directional opening and closing operations according to opening operation calculation formulas and closing operation calculation formulas to obtain a morphological processing result. The information processing module is further configured to: wherein the OR operation calculation formula is: ; wherein, is the binary segmentation mask for the crop row, is the structuring element, is the erosion operation, is the dilation operation, is the opening operation result; In the formula, the closing operation calculation formula is: ; wherein, is the morphological processing result.

5. An automatic row control system for a harvester as defined in claim 1, wherein, construct a state equation representing a motion state of the harvester and a multi-source measurement equation representing the vehicle driving information, the multi-source measurement equation including a measurement vector and a measurement matrix; generate a prior state at a current time and a prior covariance matrix at the current time based on the fusion front vehicle trajectory state vector and the state equation under an extended Kalman filtering framework; construct a modal weight matrix, and generate a Kalman gain matrix according to the modal weight matrix, the measurement vector, and the prior covariance matrix at the current time. ​ According to the prior state of the current time, the Kalman gain matrix, the measurement vector and the measurement matrix, the fused vehicle track state vector is calculated.

6. An automatic row control system for a harvester as claimed in claim 5, wherein, The information processing module is further configured to: According to the calculation formula of the fused vehicle track state vector, the fused vehicle track state vector is calculated; The calculation formula of the post-fusion vehicle track state vector is: ​ wherein, is the post-fusion vehicle trajectory state vector, is the prior state at the current time instant, is the Kalman gain matrix, is the measurement vector, is the measurement matrix.

7. An automatic row control system for a harvester as claimed in claim 6, wherein, The information processing module is further configured to: According to the calculation formula of the prior state of the current time, the prior state of the current time is calculated iteratively; The calculation formula of the prior state of the current moment is: ; wherein is a state transition matrix, is a control input matrix, is a control input vector, is the a posteriori state of the previous time instant; Wherein, in the iterative calculation process, the iterative initial value of the posterior state of the previous time is the pre-fused vehicle track state vector.

8. An automatic row control system for a harvester as claimed in claim 7, wherein, The information processing module is further configured to: Before the calculation of the fused vehicle track state vector, the pre-fused vehicle track state vector is set, the initial lateral deviation of the harvester relative to the target crop row center line is set as the first component of the pre-fused vehicle track state vector, and the initial heading deviation of the harvester relative to the target crop row is set as the second component of the pre-fused vehicle track state vector; After the calculation of the fused vehicle track state vector, the first component of the fused vehicle track state vector is extracted as the lateral deviation of the harvester relative to the target crop row center line; The second component of the fused vehicle track state vector is extracted as the heading deviation of the harvester relative to the target crop row.

9. An automatic row control system for a harvester as defined in claim 5, wherein, The information processing module is further configured to: According to the calculation formula of the Kalman gain matrix, the Kalman gain matrix is calculated; The calculation formula of the Kalman gain matrix is: ; wherein, is the Kalman gain matrix, is the prior covariance matrix at the current time instant, is the measurement matrix, is the measurement noise covariance matrix; is the modal weight matrix.

10. An automatic row control system for a harvester as defined in claim 1, wherein, The information processing module is further configured to: In the vehicle coordinate system, the distance from the center point of the harvester to the target crop row center line is calculated to obtain the initial lateral deviation; In the vehicle coordinate system, the difference between the vehicle heading angle and the direction angle of the target crop row center line is calculated as the initial heading deviation.

Images