A Global Evaluation Method for Lodging of Field Rice and Wheat Based on Inter-frame Relevance of On-vehicle Dynamic Field of View

Abstract

The present invention discloses a global evaluation method for field rice and wheat lodging based on the correlation between vehicle-mounted dynamic field frames. First, based on the constructed static coordinate model for lodging evaluation and the coordinate relationship chain compensation and correction of the inverse perspective transformation equation, the image is adaptively divided into grids through the perspective and inverse perspective transformation of key points; then, based on the edge segment fitting in two-dimensional space and the main direction extraction of multi-segment in three-dimensional space, the lodging posture of rice and wheat in the grid is evaluated, and the lodging starting point and the main direction of the correction area are extracted based on the KAZE feature; then, the matching point area estimation based on the main motion direction and the matching point search combined with the main and secondary motion directions are used to obtain a set of sparse matching point pairs in the upper and lower frame grids; finally, based on the coordinate positions of the matching point set in the intersection of the upper and lower frame test areas at different times, a coordinate chain of the harvester operation at each time is constructed, and the lodging posture at each time is converted to the same coordinate system based on the coordinate chain, thereby realizing the global evaluation of field rice and wheat lodging on the vehicle.

Description

Technical Field

[0001] The present invention relates to the field of machine vision, and particularly to the perception of the operating state and environmental information of harvesting machinery. A global evaluation method for lodging of field rice and wheat based on the inter-frame correlation of on-vehicle dynamic fields of view is used to achieve precise control of harvesting operations based on stereo vision for rice and wheat combine harvesters. Background Art

[0002] China is a major global producer of rice and wheat. Improving the production and harvesting levels of rice and wheat to achieve increased production and income is crucial for ensuring China's national food security and enhancing the quality, efficiency, and competitiveness of agriculture. With the development of agricultural mechanization and intelligent detection technologies, the production management of rice and wheat in China has gradually become automated, intelligent, and precise, gradually entering a new era of smart agriculture that adapts to China's national conditions and agricultural development. Currently, the technologies for increasing rice and wheat yields cover four aspects: tillage, sowing, management, and harvesting. Among them, since the growth and development status of the crops have been finalized in the harvesting part, maximizing the harvesting yield from a crop group with a fixed yield is an important research direction for the intelligentization of agricultural equipment. Differences in variety selection, field management, light conditions, climate temperature, etc. can easily lead to differences in the morphology of mature rice and wheat. Among them, the lodging phenomenon caused by unscientific management and external environmental impacts is an important factor affecting the harvesting efficiency and loss rate of rice and wheat. Modifying and installing devices such as subsoilers, reel lifters, and reel wheels, and realizing the automatic adjustment of header parameters such as the vertical and horizontal displacement and speed of the reel, the angle of the reel teeth, and the speed of the cutter is a reliable means for the harvester to reduce the harvesting loss rate of lodged rice and wheat. However, it also poses higher requirements for the on-vehicle evaluation of rice and wheat lodging in the field for real-time control and optimization.

[0003] Many scholars' research on crop lodging detection is mainly used for disaster assessment and field management. Different from traditional manual measurement, the more typical lodging detection technologies mainly rely on satellites, radars, drones, etc., equipped with visible light, multi-spectral, hyperspectral, near-infrared, radar and other sensors, to analyze the sensitivity of the texture, color, vegetation index and other characteristics of single / multi-growth stage crops in a high-throughput large field of view to lodging, so as to realize the identification / classification of the lodging area. Some analyses of the severity of lodging only calculate the ratio of crop height to the average plant height based on the elevation value under remote sensing data. The spatio-temporal span is large, and the spatial information of crop lodging is lacking, with scale limitations, and it is difficult to be applied to the on-vehicle rice and wheat lodging assessment in the field. At the same time, for densely growing rice and wheat crops, there are organ intersections and occlusions between plants, and the phenotypes are inconsistent. The existing lodging phenotype analysis methods based on single-view / single-modal data and carried by agricultural machinery are difficult to be applied to the field of on-vehicle rice and wheat lodging assessment in this patent. Moreover, there is a problem of weak correlation between the rice and wheat phenotype parameters in the dynamic field of view at each moment of the harvester operation in the existing on-vehicle phenotype detection methods. Therefore, it is necessary to study a global assessment method for field rice and wheat lodging based on the inter-frame correlation of the on-vehicle dynamic field of view to guide the precise automatic operation control of the harvester for lodging rice and wheat and improve the R & D and application level of intelligent harvesting equipment.

[0004] The present invention proposes a global assessment method for field rice and wheat lodging based on the inter-frame correlation of the on-vehicle dynamic field of view. Based on the stereo vision pose correction inverse perspective transformation and adaptive grid division, the lodging pose within the single-frame grid area is evaluated according to the two-dimensional and three-dimensional space parameters. The sparse point matching of the upper and lower frames is carried out by combining the primary and secondary movement directions. The coordinate chain at each moment of the harvester operation is constructed based on the grid matching pair between multiple frames. Finally, the three-dimensional pose distribution of rice and wheat lodging is globally evaluated within the operation range. Summary of the Invention

[0005] The present invention discloses a global assessment method for field rice and wheat lodging based on the correlation between vehicle-mounted dynamic field frames. First, a static coordinate model and a static coordinate relationship chain for global assessment of field rice and wheat lodging are constructed, and based on the relationship chain, the basic equation of inverse perspective transformation is compensated and corrected to improve the accuracy and reliability of the vehicle-mounted field of view inverse perspective transformation model for global assessment of field rice and wheat lodging of the present invention; at the same time, the rice and wheat lodging assessment detection area is positioned with the full cutting width as a consideration, and the perspective and inverse perspective transformation of key points are used to achieve uneven adaptive division of the original image, and the grid is divided into the minimum area for rice and wheat lodging detection during harvester operation. Afterwards, based on edge processing and line segment fitting in two-dimensional image space, and multi-straight line main direction extraction in three-dimensional space, a rough assessment of the rice and wheat lodging posture within the grid area is achieved, and based on the KAZE feature extraction of the lodging starting point on the entire two-dimensional image, the rough assessment of the lodging posture with the main direction vector as the needle is corrected to achieve a precise assessment of the lodging posture within the grid area. Next, the grid area is divided into smaller grids to evenly select matching points. A sparse set of matching point pairs is generated for the upper and lower frames using a combination of matching point area estimation based on the primary motion direction and matching point search in both the primary and secondary motion directions. Finally, considering the intersection of the measured area between the upper and lower moments of the harvester operation, a coordinate chain for each moment of the harvester operation is constructed based on the coordinate positions of the grid feature matching points within the overlapping region at different times. Based on this coordinate chain, the calculated lodging pose at each moment of the harvester operation is converted to the same coordinate system, enabling global calculation of the three-dimensional pose of rice and wheat during the harvester operation and assessment of their lodging distribution.

[0006] The technical solution of the present invention is to adopt the following steps:

[0007] (1) Inverse perspective transformation and adaptive grid division based on stereo vision posture correction: a static coordinate model with unchanged mutual relationship under the dynamic environment of continuous operation of the harvester is constructed, as well as a static coordinate relationship chain from the image pixel coordinate system to the world coordinate system and between binocular cameras; considering the posture relationship between the binocular cameras of the present invention, the basic equation of inverse perspective transformation is constructed based on the relationship chain corresponding to the basic camera coordinate system, and the posture relationship of the binocular cameras is used to compensate and correct the basic equation of inverse perspective transformation to improve the accuracy and reliability of the vehicle-borne field inverse perspective transformation model for global evaluation of rice and wheat lodging in the field; in view of the problems of image acquisition for global evaluation of rice and wheat lodging in the field on board, such as large near and small far, and high real-time requirements for vehicle-borne detection, the rice and wheat lodging evaluation detection area is located with the full cutting width as the consideration, and the perspective and inverse perspective transformation of key points are used to realize uneven adaptive division of the original image, and the grid is divided as the minimum area for rice and wheat lodging detection during harvester operation, laying the foundation for the subsequent calculation of rice and wheat lodging evaluation based on the grid, which can reduce the amount of coordinate conversion calculation and improve the real-time performance and accuracy of vehicle-borne detection.

[0008] (2) Evaluation of lodging posture in a single-frame grid area combining two-dimensional and three-dimensional space: The present invention realizes crop lodging evaluation by detecting the stalk status of rice and wheat. Therefore, under the S image channel of HSV space, based on anisotropic diffusion, least squares fitting, morphological opening and closing operations and other operations on the grid area, the edges of rice and wheat stalks in the image grid area are extracted and repaired; by edge processing and line segment fitting of the two-dimensional image, three-dimensional space line segment calculation, and multi-segment main direction extraction, the main direction vector of the grid area is calculated to realize a rough evaluation of the lodging posture of rice and wheat in the grid area; it is difficult to locate the root area of the lodging rice and wheat in the local grid area, and the overall lodging posture starting from the root area. Therefore, based on the KAZE feature extraction of the lodging starting point on the entire two-dimensional image, the rough evaluation of the lodging posture with the main direction vector as the needle is corrected to realize a precise evaluation of the lodging posture in the grid area.

[0009] (3) Sparse point matching of upper and lower frames in combination with the main and secondary motion directions: The rice and wheat lodging postures obtained by the harvester based on a single frame at each operation moment are all displayed based on the world coordinate system at each moment. It is difficult to realize the correlation analysis of rice and wheat target features between vehicle-mounted dynamic field frames and further globally evaluate the lodging of rice and wheat in the field. Therefore, a sparse point matching method of upper and lower frames in combination with the main and secondary motion directions is proposed to lay the foundation for the construction of the coordinate chain of each moment of the harvester operation. Small grids are divided in the grid area to evenly select matching points; the area where the matching points are located is estimated based on the main motion direction of the harvester operation at the corresponding moment of the upper and lower frames; and the initial point of the matching search is calculated based on the motion vectors of the adjacent points in the eight neighborhoods of the point to be matched. Considering the main motion direction characteristics, a flat template of the main direction is constructed. The main and secondary motion directions are combined to achieve fast and accurate matching point search, and a set of sparse matching point pairs of upper and lower frames is obtained.

[0010] (4) Construction of the coordinate chain of harvester operation at each moment based on grid matching pairs between multiple frames: To achieve a global assessment of rice and wheat lodging in the field under the vehicle-mounted dynamic field of view, all points in the operation area need to be mapped to the same spatial coordinate system. Considering the intersection of the test area between the upper and lower moments of the harvester operation, based on the coordinate positions of the grid feature matching point set in the overlapping area at different moments, the transformation pose relationship of the camera base coordinate system at each moment is calculated, and based on this relationship, the image pixel coordinate system, image coordinate system, camera coordinate system, and world coordinate system at each moment are linked to construct the coordinate relationship transformation chain between the harvester operation moments.

[0011] (5) Global calculation of the three-dimensional pose of rice and wheat during harvester operation and assessment of lodging distribution: Based on the established conversion chain of coordinate relationships between different moments during harvester operation, the needle vectors of the lodging poses calculated at each moment during harvester operation are transformed into the world coordinate system at the initial moment of operation, obtaining the three-dimensional pose of rice and wheat during harvester operation in one coordinate system. At the same time, according to the matching regions between adjacent frames, duplicate grid regions are removed, obtaining the three-dimensional pose distribution map of rice and wheat for the entire field from the initial moment to the end moment of operation. Based on the angle between the needle vector of the lodging pose and the horizontal plane, the projection on the horizontal plane, etc., the global lodging region distribution map, lodging angle distribution map, and lodging direction distribution map of rice and wheat can be calculated.

[0012] The present invention proposes a method for global assessment of rice and wheat lodging in the field based on the inter-frame correlation of the vehicle-mounted dynamic field of view. After adopting the above technical solutions, it has the following beneficial effects:

[0013] (1) Aiming at the accuracy and real-time requirements for assessing rice and wheat lodging in the field under the vehicle-mounted dynamic field of view of the harvester, the full cutting width is considered to locate the assessment and detection area of rice and wheat lodging, and the original image is unevenly and adaptively divided through the perspective and inverse perspective transformation of key points. The divided grid is used as the smallest area for detecting rice and wheat lodging during harvester operation. At the same time, considering the problem of perspective distortion in the images for vehicle-mounted field rice and wheat lodging assessment, a static coordinate model and a static coordinate relationship chain are constructed, and based on this, the basic equation of inverse perspective transformation is compensated and corrected to improve the accuracy and reliability of the perspective and inverse perspective transformation models for vehicle-mounted field of view used in rice and wheat lodging assessment.

[0014] (2) Aiming at the resolution contradiction between the lodging characteristics of rice and wheat with individual plant differences and the area-based harvester operation, based on the small grid regions with uneven and adaptive division, the stem edges that can be used to represent the lodging of rice and wheat are extracted and repaired in the S image channel of the HSV space within the grid regions. By fitting the direction vectors of all stem edges within the grid regions and based on the two-dimensional and three-dimensional pose relationships between multiple vectors, the overall lodging state of rice and wheat within a single grid based on the grid region is roughly evaluated. At the same time, aiming at the problem of difficultly locating the root region of lodging rice and wheat within local grid regions and the overall lodging pose starting from the root region, the starting points of lodging are extracted from the entire two-dimensional image based on KAZE features, and the rough assessment of the lodging pose represented by the main direction vector is corrected to achieve accurate assessment of the lodging pose within the grid region.

[0015] (3) When the harvester is operating, it moves at a certain speed and direction while carrying the detection and sensing equipment. Static coordinate models such as the world coordinate system and camera coordinate system based on the harvester also move accordingly. As a result, the rice and wheat lodging poses obtained based on a single frame at each operating moment are represented based on the world coordinate system of each moment. When the pose relationships between the world coordinate systems at each moment are unknown, it is difficult to achieve the correlation analysis of rice and wheat target features between frames in the vehicle-mounted dynamic field of view, and further difficult to achieve the global evaluation of rice and wheat lodging in the field. Therefore, the present invention proposes a method for sparse point matching of upper and lower frame segmentation combining the primary and secondary movement directions, estimating the area range where the matching points are located by combining the primary movement direction of the harvester operation at the corresponding moments of the upper and lower frames, calculating the initial point of the matching search based on the motion vectors of adjacent points within the eight-neighborhood of the point to be matched, constructing a flat template of the primary direction considering the primary movement direction feature, and realizing fast and accurate matching point search by combining the primary and secondary movement directions, obtaining a set of sparse matching point pairs for upper and lower frame segmentation, and laying a foundation for the construction of the coordinate chain at each moment of the harvester operation and the global evaluation of rice and wheat lodging.

[0016] (4) Aiming at the problems that it is difficult to unify the lodging postures between each moment into the same coordinate space and it is difficult to achieve the global evaluation of vehicle-mounted rice and wheat lodging, the present invention calculates the conversion pose relationship of the camera basic coordinate system at each moment based on the coordinate positions of the grid feature matching point sets within the overlapping area between the upper and lower moments of the harvester operation, and constructs a coordinate relationship conversion chain between each moment of the harvester operation based on this; at the same time, the lodging pose representation vectors calculated at each moment of the harvester operation are converted into the world coordinate system at the initial moment of the operation, obtaining the rice and wheat lodging area distribution map, lodging angle distribution map, and lodging direction distribution map of the entire field from the initial moment to the end moment of the operation. Brief Description of the Drawings

[0017] The following further elaborates on the present invention in detail in conjunction with the drawings and specific embodiments.

[0018] Figure 1 Flowchart of the method for global evaluation of rice and wheat lodging in the field based on the inter-frame correlation of vehicle-mounted dynamic field of view of the present invention

[0019] Figure 2 Schematic diagram of the static coordinate model for global evaluation of rice and wheat lodging in the field of the present invention

[0020] Figure 3 Schematic diagram of the positioning and adaptive grid division of the detection area for rice and wheat lodging evaluation of the present invention

[0021] Figure 4 Schematic diagram of single-plant and population rice and wheat lodging

[0022] Figure 5 Schematic diagram of the detection and repair of the edges of rice and wheat stems within the grid area of a single frame of the present invention

[0023] Figure 6 Schematic diagram of two-dimensional line segment fitting and three-dimensional multi-line segment main direction extraction of the present invention

[0024] Figure 7 This is the KAZE feature point detection result diagram of the present invention

[0025] Figure 8 Schematic diagram of small grid division and eight-neighborhood adjacent points of the present invention

[0026] Figure 9 This is the flat matching point search template based on the main direction of harvester movement of the present invention

[0027] Figure 10 Schematic diagram of the coordinate chain of the harvester at each moment of operation of the present invention DETAILED DESCRIPTION

[0028] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0029] The present invention discloses a global assessment method for field rice and wheat lodging based on the correlation between vehicle-mounted dynamic field frames. First, a static coordinate model and a static coordinate relationship chain for global assessment of field rice and wheat lodging are constructed, and based on the relationship chain, the basic equation of inverse perspective transformation is compensated and corrected to improve the accuracy and reliability of the vehicle-mounted field of view inverse perspective transformation model for global assessment of field rice and wheat lodging of the present invention; at the same time, the rice and wheat lodging assessment detection area is positioned with the full cutting width as a consideration, and the perspective and inverse perspective transformation of key points are used to achieve uneven adaptive division of the original image, and the grid is divided into the minimum area for rice and wheat lodging detection during harvester operation. Afterwards, based on edge processing and line segment fitting in two-dimensional image space, and multi-straight line main direction extraction in three-dimensional space, a rough assessment of the rice and wheat lodging posture within the grid area is achieved, and based on the KAZE feature extraction of the lodging starting point on the entire two-dimensional image, the rough assessment of the lodging posture with the main direction vector as the needle is corrected to achieve a precise assessment of the lodging posture within the grid area. Next, the grid area is divided into smaller grids to evenly select matching points. A sparse set of matching point pairs is generated for the upper and lower frames using a combination of matching point area estimation based on the primary motion direction and matching point search in both the primary and secondary motion directions. Finally, considering the intersection of the measured area between the upper and lower moments of the harvester operation, a coordinate chain for each moment of the harvester operation is constructed based on the coordinate positions of the grid feature matching points within the overlapping region at different times. Based on this coordinate chain, the calculated lodging pose at each moment of the harvester operation is converted to the same coordinate system, enabling global calculation of the three-dimensional pose of rice and wheat during the harvester operation and assessment of their lodging distribution.

[0030] See also Figure 1 , the specific steps are as follows:

[0031] 1. Inverse perspective transformation and adaptive grid division based on stereoscopic vision posture correction: Construct a static coordinate model with unchanged mutual relationship in the dynamic environment of continuous operation of the harvester, as well as a static coordinate relationship chain from the image pixel coordinate system to the world coordinate system and between binocular cameras; considering the posture relationship between the binocular cameras of the present invention, the basic equation of inverse perspective transformation is constructed based on the relationship chain corresponding to the basic camera coordinate system, and the posture relationship of the binocular camera is used to compensate and correct the basic equation of inverse perspective transformation to improve the accuracy and reliability of the vehicle-mounted field of view inverse perspective transformation model used for global assessment of field rice and wheat lodging; in view of the problems of image acquisition for global assessment of field rice and wheat lodging on vehicle, such as large near and small far, and high real-time requirements for vehicle-mounted detection, the rice and wheat lodging assessment detection area is located with the full cutting width as the consideration, and the perspective and inverse perspective transformation of key points are used to achieve uneven adaptive division of the original image, and the grid is divided as the minimum area for rice and wheat lodging detection during harvester operation, laying the foundation for the subsequent calculation of grid-based rice and wheat lodging assessment, which can reduce the amount of coordinate conversion calculation and improve the real-time performance and accuracy of vehicle-mounted detection. The specific steps are as follows:

[0032] (1) Constructing a static coordinate model for global assessment of rice and wheat lodging in the field:

[0033] like Figure 2 As shown in the figure, the global assessment of rice and wheat lodging in the field is a dynamic detection of continuous video frames. It is necessary to first build a static coordinate model whose mutual relationship remains unchanged under the dynamic environment of continuous operation of the harvester. The left and right camera coordinate systems of the stereo binocular vision are: c1 -X c1 Y c1 Z c1 and O c2 -X c2 Y c2 Z c2 , and O c1 -X c1 Y c1 Z c1 is the camera base coordinate system; the image pixel coordinate systems constructed in the images acquired by the left and right cameras are O o1 -U1V1 and O o2 -U2V2, the image coordinate systems are O i1 -X i1 Y i1 and O i2 -X i2 Y i2 ; The world coordinate system is O w -X w Y w Z w .O w -X w Y w Z w Xw and Y w axis is on the horizontal plane, the axis is vertically upward, and the origin O w and O c1 are on the same axis perpendicular to the horizontal plane and the distance is h.

[0034] (2) Construction of the static coordinate relationship chain:

[0035] The construction of the static coordinate model relationship chain mainly includes the conversion relationships of O o1 -U1V1 and O i1 -X i1 Y i1 ; o1 H i1 O o2 -U2V2 and O i2 -X i2 Y i2 ; o2 H i2 O i1 -X i1 Y i1 and O c1 -X c1 Y c1 Z c1 ; i1 H c1 O i2 -X i2 Y i2 and O c2 -X c2 Y c2 Z c2 ; i2 H c2 O c1 -X c1 Y c1 Z c1 and O c2 -X c2 Y c2 Z c2 ; c2 H c1 O c1 -X c1 Y c1 Z c1 and O w -X w Y w Z w ; c1 H w where the relationship for the point to be transformed from the representation A in the a coordinate system to the representation B in the b coordinate system is B = a H b A.

[0036] Let O i1 -X i1 Y i1 be the origin O i1 of O o1 -U1V1 with coordinates (u 01 , v 01 ). The origin O i2 of O i2 -X i2 Y i2 has coordinates (u o2 , v 02 ) in O 02 -U2V2. The single-pixel sizes of the left and right camera sensor target surfaces are the same, with lengths and widths of Δx and Δy respectively. Calculate the conversion relationship based on scaling and translation affine transformation o1 H i1 , o2 H i2 as follows:

[0037]

[0038] Let the focal lengths of the left and right cameras be f. Calculate the conversion relationship based on the principle of pinhole imaging i1 H c1 , i2 H c2 as follows:

[0039]

[0040] Based on the poses of the dot calibration board relative to the left and right cameras within the overlapping field of view of the left and right cameras, calculate the relative pose relationship between the left and right cameras. Let R c1 and R c2 be the rotational pose relationships between the left and right cameras and the calibration board respectively, and T c1 and T c2 be the translational pose relationships between the left and right cameras and the calibration board respectively. The calculated conversion relationship c2 H c1 is as follows:

[0041]

[0042] According to the pose relationship between the coordinate system O c1 -X c1 Y c1 Z c1 of the base camera (left camera) and O w -X w Y w Z w calculate the conversion relationship between the camera coordinate system and the world coordinate system c1 H wThe center point O of the camera's basic coordinate system in the present invention c1 and the center point O of the world coordinate system w are set on the same vertical axis and are separated by h; O w -X w Y w Z w is translated to O c1 , and after rotating 90 + θ1 degrees around the X w axis, O c1 -X c1 Y c1 Z c1 can be obtained, and O c1 -X c1 Y c1 Z c1 has no scaling relationship with O w -X w Y w Z w . Therefore, the transformation relationship between O c1 -X c1 Y c1 Z c1 and O w -X w Y w Z w can be calculated based on Equation (4). c1 H w can be calculated based on Equation (4).

[0043]

[0044] Among them, the rotation matrix R is calculated by multiplying single rotation matrices on the left. Based on the right-hand coordinate system for right rotation, then:

[0045]

[0046] (3) Inverse perspective transformation and compensation correction:

[0047] The present invention constructs the basic equation of inverse perspective transformation based on the coordinate relationship chain o1 H i1 , i1 H c1 , c1 H w and, based on c2 H c1 and o2 H i2 , i2 H c2The relationship chain compensates and corrects the basic equation of inverse perspective transformation, improving the accuracy and reliability of the vehicle-mounted field-of-view inverse perspective transformation model for the global assessment of rice and wheat lodging in the field of the present invention. Inverse perspective transformation is to calculate the corresponding spatial top view in the world coordinate system based on the original pixel image obtained by the sensor. Therefore, the inverse perspective transformation equations of the left and right cameras can be constructed based on Equations (6) and (7) respectively, where [x w1i , y w1i , z w1i and [x w2i , y w2i , z w2i are the coordinates of the same spatial point P i obtained based on the coordinate conversion chains of the left and right cameras respectively; [u 1i , v 1i and [u 2i , v 2i are the pixel coordinates of the points p i and p 1i and p 2i on the images of the left and right cameras to which the spatial point P i is mapped, which can be directly read from the image. The parameters Δx, Δy, f, u 01 , v 01 , u 02 , v 02 can be obtained by camera calibration, and the parameters h and θ1 can be obtained by a tape measure and a protractor.

[0048]

[0049] Taking the coordinate conversion of the left camera as the basic equation of inverse perspective transformation, the deviation [x w2i , y w2i of the coordinates [x w1i , y w1i obtained by the right camera inverse perspective transformation equation and the coordinates [x w1i , y w1i obtained by the basic equation, i.e., [x w1i - x w2i , y w1i - y w2i , compensates and corrects the inverse perspective transformation. The compensation and correction are calculated by the gross error rejection method, as shown in Equations (8) and (9), where n is the number of spatial points used for inverse perspective transformation. Considering the calculation error of the coordinates of the same spatial point, the gross error rejection threshold (x t , y t ) can be calculated based on the mean value of the coordinate deviation, as shown in Equations (10) and (11). Based on this, the inverse perspective transformation coordinates (x wi , y wi ) after supplementary correction are obtained.

[0050]

[0051] (4) Positioning of the detection area for rice and wheat lodging assessment:

[0052] Considering that the lodging assessment of rice and wheat in the field is mainly used for the early detection of the real-time operation of the harvester, the rice and wheat area to be detected is within a certain range in front of the harvester. At the same time, considering that the operation speed of the harvester is generally 0.5m / s - 1.8m / s, the present invention considers the operation area of the harvester at a single moment and sets the length of the detection area as L d = 4.8m (this area can be increased or decreased according to requirements). Considering that the maximum width of the harvester operation area does not exceed the full cutting width, that is, the width of the cutting table, based on this and considering a certain margin, the present invention sets the width of the detection area as W d = W g +d, where W g is the width of the cutting table, and d is the detection margin, which is set to 35cm by the present invention (this margin can be increased or decreased according to requirements).

[0053] The image processing of the left and right cameras in the present invention is consistent, so the left camera is taken as an example for illustration. As Figure 3 shown, by performing the Hough transform on the fixed area in the lower part of the original image, the outer edge of the cutting table is extracted to obtain the fitting straight line l 11 of the front end of the cutting table and the fitting straight line l 12 of the rightmost end; calculate the intersection point p 11 of the straight line l 11 and the leftmost edge of the image. Based on p 11 , construct a straight line l 13 parallel to the U1 axis of the image coordinate system; and calculate the intersection point p 12 of l 13 and l 12 ; move the point p 12 along the straight line l 13 by a distance d to obtain the point p 13 ; move the point p 11 along the leftmost edge of the image in the opposite direction of the V1 axis by a distance L d to obtain the point p 14 . Based on the points p 11 , p 13 , p 14 , construct a rectangle with two sides parallel to the V1 and U1 axes respectively as the detection area for rice and wheat lodging assessment.

[0054] (5) Adaptive grid division of the original image based on perspective and inverse perspective transformation:

[0055] Considering the influence of the clarity of the top view after inverse perspective transformation on the lodging detection accuracy, the present invention uses a high-definition original image to detect lodging. At the same time, considering the influence of the perspective of objects in the image (objects appear larger when closer and smaller when farther away) on the division of the detection area, uneven adaptive division of the original image is achieved through key point perspective and inverse perspective transformation, so that the actual spatial area corresponding to each small area after division is equal. First, the key points p 11 , p 14 are calculated based on formulas (8) and (9) to obtain the real points P1 and P4 in the three-dimensional space corresponding to the key points, and the inverse perspective transformation top view coordinates of P1 and P4 are obtained as P1 = (x w1 , y w1 ) and P4 = (x w4 , y w4 ). Equal division is performed according to the distance between P1 and P4 on the X w axis in the world coordinate system, and the number of equal divisions is m x . Specifically, the coordinates (x di , y di , y di ) of the key point P di are calculated based on formula (12):

[0056]

[0057] Based on the obtained equally divided key points P di in space and the inverse calculation of formulas (8) and (9), the image point p 1di after perspective transformation is obtained. It is set that the grid division on the V1 axis in the detection area of the original image is unevenly divided based on the key points p 1di of perspective and inverse perspective transformation, and the grid division on the U1 axis is evenly divided based on the set number of grids m y . Finally, the adaptive grid division of the minimum area for detecting rice and wheat lodging during the operation of the harvester is obtained, laying a foundation for the real-time and accurate detection of on-vehicle rice and wheat lodging.

[0058] 2. Evaluation of lodging poses within a single-frame grid area combining two-dimensional and three-dimensional spaces: The present invention realizes the lodging evaluation of crops by detecting the stem states of rice and wheat. Therefore, in the S image channel of the HSV space, based on operations such as anisotropic diffusion, least squares fitting, and morphological opening and closing operations on the grid area, the edges of rice and wheat stems within the image grid area are extracted and repaired; through edge processing and line segment fitting of the two-dimensional image, three-dimensional space line segment calculation, and extraction of the main direction of multiple line segments, the main direction vector of the grid area is calculated to achieve a rough evaluation of the lodging poses of rice and wheat within the grid area; it is difficult to locate the root area of lodging rice and wheat within the local grid area and the overall lodging pose starting from the root area. Therefore, based on KAZE features, the lodging starting points on the entire two-dimensional image are extracted to correct the rough evaluation of the lodging poses represented by the main direction vector, so as to achieve a precise evaluation of the lodging poses within the grid area. The specific steps are as follows:

[0059] (1) Edge detection and repair in the S image channel:

[0060] The lodging of rice and wheat crops is mainly reflected in the pose states of the stems, such as Figure 4 shown, including breakage, stem lodging, root lodging, etc. Therefore, the present invention realizes the lodging evaluation of crops by detecting the stem states of rice and wheat. As Figure 5 shown, the specific steps are as follows:

[0061] 1) Based on adaptive grid division, for a static in-vehicle lodging evaluation image, the images within each grid area are extracted to obtain a set of grid areas with inconsistent sizes. Since rice and wheat crops grow densely and their organs cross, and the ground area is difficult to reach by light and is darker, the present invention converts the RGB color image to the HSV space and processes it using the saturation S channel whose representative color is close to the spectral color.

[0062] 2) Considering that there are other organ areas such as awns and leaves in rice and wheat crops that interfere with the stems, through anisotropic diffusion of the grid area, the discontinuous image edges are connected, and no smoothing processing is performed in the vertical direction of the main direction to enhance the consistency of the image structure. For the enhanced grid area, least squares fitting is used for edge detection, and the quadratic polynomial parameters are calculated for each pixel point within the area. To extract the long edges corresponding to the stems and reduce the influence of short edges such as awns and grains on the detection, the convolution small window size of the least squares fitting edge detection is designed to be 3×3; the second-order directional derivative of the pixel point perpendicular to the straight line direction is calculated, and the pixel point corresponding to the local maximum value of the second-order directional derivative is used as the edge point, and all the detected edge points are connected to form an edge line.

[0063] 3) Considering the influence of the edges obtained from other non-stem regions on the stem edges after edge detection, as well as problems such as the disconnection and missing gaps between the discontinuous edge lines detected on the stem edges, the present invention repairs the disconnected edges through morphological opening and closing operations, and purifies them based on features such as the length of the edge lines. Finally, a set of continuous long edge lines on the stems is obtained in the grid region, and the number of edge lines is the same as the number of stems. Let the number of stems detected in the grid region at the i-th row and j-th column be s ij 。

[0064] (2) Two-dimensional line segment fitting and three-dimensional multi-line segment main direction extraction: <{

[0065] The present invention calculates the main direction vector of the grid region through edge line segment fitting of the two-dimensional image, three-dimensional space line segment calculation, and multi-line segment main direction extraction, and is used to represent the comprehensive pose of rice and wheat in a single grid region. As Figure 6 shown, the specific steps are as follows:

[0066] 1) In the grid region of the two-dimensional image, based on the least squares method, linear fitting is performed on each pixel of the s ij edges to obtain the two-dimensional straight line of the stem in the grid region at the i-th row and j-th column. Let the starting point and ending point of the k-th edge line segment in the grid region be p ks 7]=(x ks ,y ks ) and p ke [[ID=]23]=(x ke ,y ke ). Taking the abscissa as a reference, the fitting straight line segment equation of the k-th edge is constructed, as shown in Equation (13), where a k and b k are the parameters of the fitting straight line segment equation of the k-th edge respectively, (x, y) are the points on the fitting straight line segment, and the starting point and ending point of the k-th fitting straight line segment are calculated to be p' ks =(x ks ,y' ks ) and p' ke =(x ke ,y' ke ).

[0067] y = a k x + b k , k = 1, 2,... s ij , x ks ≤x≤x ke (13)

[0068] 2) Based on the coordinate transformation of binocular vision, calculate the spatial points P' corresponding to the starting points p' ks and the ending points p' ke of all the fitting straight line segments in the grid regionks and P' ke with three-dimensional coordinates (x wks , y wks , z wks ) and (x wke , y wke , z wke ). The present invention uses unit vectors to represent the three-dimensional main directions of grid regions, and sets the starting point of the direction at the center of the region to jointly represent the regional comprehensive pose of rice and wheat. Therefore, it is necessary to first calculate the vectors corresponding to each group of straight line segments to obtain a group of vector groups (x wke - x wks , y wke - y wks , z wke - z wks ). Based on formulas (14) and (15), taking the average value of the angles of all vectors in the region as the angle of the unit vector of the comprehensive pose of rice and wheat, the unit vector starting point q ij = (x ij0 , y ij0 , z ij0 ), where (α k , β k , γ k ) is the direction of the k-th vector in the region, and (α, β, γ) is the direction of a. [[ID=SS=44]]<SS= [[ID=SS=45]]<SS=

[0069] [[ID=SS=46]]<SS= [[ID=SS=47]]<SS= [[ID=SS=48]]<SS=

[0070] (3) Extraction of the lodging starting point based on KAZE features and correction of the regional main direction:

[0071] It is difficult to determine the root region of lodging rice and wheat in the local direction and the overall evaluation of the lodging pose starting from the root region based on the pose detection of rice and wheat stems in the local grid region. Therefore, the present invention starts from the entire detection region of a single frame, locates the starting point position of lodging rice and wheat in the region based on KAZE feature points, and uses it to correct the direction of the stem pose vector of each grid region to achieve a more accurate single-frame pose evaluation of rice and wheat lodging.

[0072] As Figure 7As shown in the figure, the root area of rice and wheat is exposed from the dense crops in the lodging state, showing obvious edge and point features. Therefore, the midpoint of the root area of the lodged rice and wheat is located based on the distribution of KAZE feature points. First, a non-linear scale space is constructed based on the effective additive operator splitting technique and variable conductance diffusion, and KAZE feature points are extracted for the entire detection area of a single frame in the non-linear scale space. By blurring locally to reduce noise and retain the object boundary, a multi-scale two-dimensional KAZE feature point set is finally obtained. Then, based on the Euclidean distance between KAZE feature points, the point sets with a distance less than a certain distance threshold are clustered, the number of points in each clustering category is counted, and the center of the clustering category with the largest number of points is taken as the midpoint g0=(x0,y0,z0) of the root area of the lodged rice and wheat, which is also used as the starting point of the lodged rice and wheat in a single-frame image (the present invention mainly aims at the continuously lodged state. If there are multiple lodging areas, the root centers of multiple lodging areas can be extracted based on the number of feature points in the clustering category).

[0073] To correct the regional principal direction for indicating the lodging direction of rice and wheat in the grid area of the i-th row and j-th column, first calculate the starting point q of the principal direction of the grid area of the i-th row and j-th column ij The positional relationship relative to the midpoint g0 of the root area is represented by a vector As shown in Equation (16), and based on The relationship with the principal direction vector of each grid area Adjust the positive and negative directions of the vector a to obtain the corrected regional principal direction To solve the problem that it is difficult to determine the roots and tops of lodged rice and wheat in the local grid area, the vector direction starting from the roots of rice and wheat is used as the pointer for its lodging pose

[0074]

[0075] 3. Sparse point matching of upper and lower frame grids by combining the primary and secondary motion directions: The lodging poses of rice and wheat obtained based on a single frame by the harvester at each operation moment are all indicated based on the world coordinate system at each moment, which is difficult to realize the correlation analysis of the rice and wheat target features between the vehicle-mounted dynamic field-of-view frames, and further globally evaluate the lodging of rice and wheat in the field. Therefore, a method of sparse point matching of upper and lower frame grids by combining the primary and secondary motion directions is proposed, which lays a foundation for the construction of the coordinate chain at each operation moment of the harvester. Small grids are divided within the grid area to evenly select matching points; the regional range where the matching points are located is estimated by combining the primary motion direction of the harvester operation at the corresponding moments of the upper and lower frames; and based on the motion vectors of the adjacent points within the eight-neighborhood of the point to be matched, the initial point of the matching search is calculated, a flat template of the primary direction is constructed considering the primary motion direction feature, and fast and accurate matching point search is realized by combining the primary and secondary motion directions to obtain a set of sparse matching point pairs of upper and lower frame grids. The specific steps are as follows:

[0076] (1) Selection of uniform matching points for small grid division:

[0077] In order to reduce the computational complexity and improve the real-time performance of vehicle-mounted detection, the present invention adopts a small grid division method to uniformly select points. r and I r-1 , in image I r Adaptive division of detection area m x ×m y The grid area is evenly divided into 3×3 small grids, and the center point of each small grid is taken as the matching point M rs ,s=1,2,...,m x ×m y ×9.

[0078] (2) Matching point area estimation based on the main motion direction:

[0079] During harvester operation, there are major movements such as uneven harvester speed, vibration, and steering, as well as minor movements caused by wind and the pulling of rice and wheat due to the movement of the harvester. Therefore, the matching point area can be estimated by combining the main motion direction at the corresponding time of the upper and lower frames to reduce the matching point search time.

[0080] Based on the speed sensor installed on the harvester wheel / track bearing, the real-time movement speed v of the harvester is calculated r Considering that the distance between the upper and lower frames of the harvester is very short, the calculation is based on linear motion, and the vertical difference between the upper and lower frames is approximately s. r =v r t distance, converted to pixel distance d r Based on the camera calibration parameters, the distance in space is converted to the image pixel distance s' r . Let image I r Point M on rs The pixel coordinates are (u rs ,v rs ), considering that the harvester carries the camera system and moves forward, in the ideal case of only considering the linear motion of the harvester, point M rs The corresponding spatial point is mapped to image I r-1 Point M on (r-1)s The pixel coordinates are (u rs ,v rs -d r ). Therefore, considering that rice and wheat crops are fixed root crops, their movement range affected by external forces is small, based on the maximum range of their small-amplitude movement, M is set. (r-1)s L in the center p ×W p The pixel interval is the matching point search interval range LW p .

[0081] (3) Combined matching point search of primary and secondary motion directions:

[0082] Considering the natural growth state of rice and wheat crops, their secondary motions due to external forces are consistent within a certain small area. Based on this, the present invention calculates the initial point of the matching search based on the motion vectors of adjacent points within the eight-neighborhood of the point to be matched. Furthermore, considering the characteristics of the main direction of motion, a flat template of the main direction is constructed. Combining the main and secondary motion directions enables fast and accurate matching point search. The main steps are as follows:

[0083] 1) Let image I r The point to be matched is M rs =(u rs ,v rs ), if M rs For I r The first matching point on LW p The center of the matching point is the initial point M' rs0 If M rs Not for I r The first matching point on Figure 8 As shown, let M rs Eight small grid center points M in each area, up, down, left, and right rst ,t=1,2,...,8 The corresponding motion vector is MV rst =(u rst ,v rst ), t=1,2,...,8, calculate MV based on formula (17) rst , the mean of t=1,2,...,8 Based on Positioning M rs The initial point for searching matching points is

[0084]

[0085] 2) Considering that the matching of image points is not only related to the pixel point, but also involves the changes in its neighborhood, the point is taken as the center and the 3×3 pixel area is used as the window. The points are calculated from left to right and from top to bottom. The absolute difference (SAD) between points is obtained based on formula (18) and used as the judgment standard for point matching, where f ri = represents the pixel value of the i-th point in the corresponding small window on the r-th frame. i from 1 to 9 corresponds to the points (u-1, v-1), (u, v-1), (u+1, v-1), ..., (u+1, v+1). Set a threshold T. If SAD < T, the two corresponding points match; if SAD ≥ T, go to step 3.

[0086]

[0087] 3) Considering that the main movement direction of the harvester is the vertical direction of the image, a flat template of the main direction is constructed ( Figure 9 ). Calculate the SAD value of each point. If the SAD value of the template center point is the smallest, then this point is the corresponding matching point; if the SAD value of any other point in the template that is not the center point satisfies SAD<T, then this point is the corresponding matching point; otherwise, select the point with the smallest SAD value among the other points in the template that are not the center point as the new center point, reconstruct the flat template of the new main direction, and go to step 3) until a matching point is found.

[0088] 4) Based on the above process, the image frame I is obtained r and I r-1 Matching point pairs M on all small grids rs and M' rs ,s=1,2,...,m x ×m y ×9.

[0089] 4. Construction of a coordinate chain for each moment of harvester operation based on multi-frame grid matching: To achieve a global assessment of rice and wheat lodging in the field under the vehicle's dynamic field of view, all points within the operation area must be mapped to the same spatial coordinate system. Considering the intersection of the measured area between the previous and next moments of the harvester operation, the coordinate positions of the grid feature matching points within the overlapping area at different times are used to calculate the transformation pose relationship of the camera's base coordinate system at each moment. Based on this relationship, the image pixel coordinate system, image coordinate system, camera coordinate system, and world coordinate system are linked at each moment to construct a coordinate relationship transformation chain for each moment of the harvester operation.

[0090] Assume homework T i The camera base coordinate system at this moment is O c1Ti -X c1Ti Y c1Ti Z c1Ti , T i and T i+1 The transformation relationship of the camera base coordinate system at the moment is c1T(i+1) H c1Ti The key to constructing the coordinate chain of harvester operation at each moment lies in the relationship between the basic coordinate systems of the cameras at each moment. Based on this, the coordinate systems between each moment can be linked. The specific steps of the coordinate chain construction are as follows:

[0091] (1) Figure 10 As shown, let T i At this moment, according to the obtained T i and T i+1 The set of matching points M between the image pairs at the moment (i+1)s and M' (i+1)s , respectively calculate the image point at T iand T i+1 The camera base coordinate system O at the moment c1Ti -X c1Ti Y c1Ti Z c1Ti and O c1T(i+1) -X c1T(i+1) Y c1T(i+1) Z c1T(i+1) The three-dimensional coordinate representation A under (i+1)s and A' (i+1)s ;

[0092] (2) Calculate the translation relationship between O (i+1)s and A' (i+1)s based on the mean of the positional relationships between points, and calculate the rotation relationship between O clTi -X c1Ti Y c1Ti Z c1Ti and O c1T(i+1) -X c1T(i+1) Y c1T(i+1) Z c1T(i+1) based on the positional relationships between the lines and vectors formed by two points of A (i+1)s and A' (i+1)s to obtain the transformation relationship between O c1Ti -X c1Ti Y c1Ti Z c1Ti and O c1T(i+1) -X c1T(i+1) Y c1T(i+1) Z c1T(i+1) ; c1Ti -X c1Ti Y c1Ti Z c1Ti and O c1T(i+1) -X c1T(i+1) Y c1T(i+1) Z c1T(i+1) The transformation relationship c1T(i+1) H c1Ti ;

[0093] (3) If the moment T i is the last moment, then end the calculation of the coordinate chain relationship to obtain c1T1 H c1T0 , c1T2 H c1T1 , …, c1Tn H c1T(n-1) ; Otherwise, set i = i + 1 and continue with steps (1) and (2).

[0094] 5. Global calculation of the three-dimensional pose of rice and wheat during harvester operation and assessment of lodging distribution: Based on the constructed coordinate relationship conversion chain between different moments of harvester operation, the needle vectors of the lodging poses calculated at each moment of harvester operation are converted into the world coordinate system at the initial moment of operation, obtaining the three-dimensional pose of rice and wheat during harvester operation in one coordinate system. Meanwhile, according to the matching regions between adjacent frames, the overlapping grid regions are removed, obtaining the three-dimensional pose distribution map of rice and wheat for the entire field from the initial moment to the end moment of operation. Based on the angle between the needle vector of the lodging pose and the horizontal plane, the projection on the horizontal plane, etc., the global lodging region distribution map, lodging angle distribution map, and lodging direction distribution map of rice and wheat can be calculated. The specific steps are as follows:

[0095] (1) Based on the constructed coordinate relationship conversion chain between different moments of harvester operation, the needle vectors of the lodging poses calculated at each moment of harvester operation are converted into the camera base coordinate system O c1T0 -X c1T0 Y c1T0 Z c1T0 at the moment T0, and based on the relationship between the camera base coordinate system and the world coordinate system, it is further converted into the world coordinate system O w0 -X w0 Y w0 Z w0 at the moment T0, obtaining the three-dimensional pose of rice and wheat during harvester operation in one coordinate system

[0096] (2) According to the matching regions between adjacent frames, the overlapping grid regions are removed, obtaining the three-dimensional pose distribution map of rice and wheat for the entire field from the initial moment to the end moment of operation. Taking the angle with the horizontal plane as the reference for the growth angle of rice and wheat, and setting a threshold. When the angle is less than the threshold, it is determined as lodging. Based on this, the global lodging region distribution map and lodging angle distribution map of rice and wheat for the entire field are calculated. And according to the projection on the horizontal plane, the lodging direction of rice and wheat in each grid region is calculated, obtaining the global lodging direction distribution map of rice and wheat.

[0097] So far, the global assessment of rice and wheat lodging in the field based on the inter-frame correlation of the vehicle-mounted dynamic field of view has been completed.

[0098] It should be understood that the above embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. After reading the present invention, various equivalent modifications made by those skilled in the art to the present invention all fall within the scope defined by the appended claims of this application.

Claims

1. A global evaluation method for lodging of field rice and wheat based on the inter-frame correlation of vehicle-mounted dynamic field of view, characterized in that, The steps include: (1) Inverse perspective transformation and adaptive meshing based on stereo vision pose correction: Constructing a static coordinate model whose relationships remain unchanged under the dynamic environment of continuous operation of the harvester, as well as a static coordinate relationship chain from the image pixel coordinate system to the world coordinate system and between binocular cameras; The basic equations for inverse perspective transformation are constructed based on the relationship chain corresponding to the basic camera coordinate system, and the position relationship of the binocular camera is used to compensate and correct the basic equations for inverse perspective transformation. The rice and wheat lodging assessment and detection area is located based on the full harvesting width, and the perspective and inverse perspective transformation of key points are used to achieve uneven adaptive partitioning of the original image. The grid is divided as the minimum area for rice and wheat lodging detection during harvester operation. (2) Assessment of lodging posture in a single-frame grid area combining two-dimensional and three-dimensional space: The crop lodging assessment is achieved by detecting the stalk status of rice and wheat. Therefore, in the S image channel of the HSV space, the edges of the rice and wheat stalks in the image grid area are extracted and repaired based on anisotropic diffusion, least squares fitting, and morphological opening and closing operations of the grid area; the main direction vector of the grid area is calculated by edge processing and line segment fitting of the two-dimensional image, three-dimensional space line segment calculation, and multi-segment main direction extraction to achieve a rough assessment of the lodging posture of rice and wheat in the grid area; the lodging starting point on the entire two-dimensional image is extracted based on the KAZE feature, and the rough assessment of the lodging posture with the main direction vector as the needle is corrected to achieve a precise assessment of the lodging posture in the grid area; (3) Sparse point matching of upper and lower frames in combination with the main and secondary motion directions: small grids are divided in the grid area to evenly select matching points; the area where the matching points are located is estimated based on the main motion direction of the harvester at the corresponding moment of the upper and lower frames; and the initial point of the matching search is calculated based on the motion vectors of the adjacent points in the eight neighborhoods of the point to be matched, and a flat template of the main direction is constructed considering the main motion direction characteristics. The main and secondary motion directions are combined to achieve fast and accurate matching point search, and a set of sparse matching point pairs of upper and lower frames is obtained; (4) Construction of the coordinate chain of harvester operation at each moment based on grid matching pairs between multiple frames: In order to realize the global evaluation of rice and wheat lodging in the field under the vehicle-mounted dynamic field of view, all points in the operation area are mapped to the same spatial coordinate system. Considering the intersection of the test area between the upper and lower moments of the harvester operation, the coordinate position of the grid feature matching point set at different moments in the overlapping area is used to calculate the transformation pose relationship of the camera basic coordinate system at each moment. Based on this relationship, the image pixel coordinate system, image coordinate system, camera coordinate system, and world coordinate system at each moment are linked to construct the coordinate relationship transformation chain between the harvester operation moments. (5) Global calculation of the three-dimensional pose of rice and wheat during harvester operation and assessment of lodging distribution: Based on the constructed coordinate relationship conversion chain between different moments during harvester operation, the needle vectors of the lodging poses calculated at each moment during harvester operation are converted into the world coordinate system at the initial moment of operation, obtaining the three-dimensional pose of rice and wheat during harvester operation in one coordinate system. At the same time, according to the matching regions between adjacent frames, the overlapping grid regions are removed, obtaining the three-dimensional pose distribution map of rice and wheat for the entire field from the initial moment to the end moment of operation. Based on the angle between the needle vector of the lodging pose and the horizontal plane and its projection on the horizontal plane, the global lodging region distribution map, lodging angle distribution map, and lodging direction distribution map of rice and wheat are calculated.

2. The global evaluation method for lodging of field rice and wheat based on the inter-frame correlation of on-vehicle dynamic field of view according to claim 1, wherein: The specific process of step (1) is as follows: 1.1, Construct a static coordinate model for global assessment of rice and wheat lodging in the field: Let the left and right camera coordinate systems for stereo binocular vision be: O c1 -X c1 Y c1 Z c1 and O c2 -X c2 Y c2 Z c2 Moreover, O c1 -X c1 Y c1 Z c1 is the camera base coordinate system; the image pixel coordinate systems constructed in the images obtained by the left and right cameras are O o1 -U1V1 and O o2 -U2V2 respectively, and the image coordinate systems are O i1 -X i1 Y i1 and O i2 -X i2 Y i2 respectively; the world coordinate system is O w -X w Y w Z w , and for O w -X w Y w Z w the X w and Y w axes are on the horizontal plane, the axis is vertically upward, and the origin O w and O c1 are on the same axis perpendicular to the horizontal plane and the distance between them is h; 1.2, Construct a static coordinate relationship chain: The construction of the static coordinate model relationship chain mainly includes O o1 -U1V1 and O i1 -X i1 Y i1 conversion relationship o1 H i1 ,O o2 -U2V2 and O i2 -X i2 Y i2 conversion relationship o2 H i2 ,O i1 -X i1 Y i1 and O c1 -X c1 Y c1 Z c1 conversion relationship i1 H c1 ,O i2 -X i2 Y i2 and O c2 -X c2 Y c2 Z c2 conversion relationship i2 H c2 ,O c1 -X c1 Y c1 Z c1 and O c2 -X c2 Y c2 Z c2 conversion relationship c2 H c1 ,O c1 -X c1 Y c1 Z c1 and O w -X w Y w Z w conversion relationship c1 H w ,where the relationship for the point to be transformed from the representation A in the a coordinate system to the representation B in the b coordinate system is B = a H b A; Let O i1 -X i1 Y i1 be the origin O i1 at O o1 -U1V1, the coordinates are (u 01 , v 01 ). O i2 -X i2 Y i2 be the origin O i2 at O o2 -U2V2, the coordinates are (u 02 , v 02 ). The single-pixel sizes of the left and right camera sensor target surfaces are the same, and the length and width are Δx and Δy respectively. Calculate the conversion relationship based on scaling and translation affine transformation o1 H i1 、 o2 H i2 are as follows: Assume that the focal lengths of the left and right cameras are both f, and calculate the conversion relationship based on the principle of pinhole imaging i1 H c1 、 i2 H c2 is as follows: Based on the poses of the dot calibration board relative to the left and right cameras within the overlapping field of view of the left and right cameras, calculate the relative pose relationship between the left and right cameras; Let R c1 and R c2 be the rotational pose relationships between the left and right cameras and the calibration board respectively, and T c1 and T c2 be the translational pose relationships between the left and right cameras and the calibration board respectively. The calculated transformation relationship c2 H c1 is as follows: According to the left camera coordinate system O of the base camera c1 -X c1 Y c1 Z c1 and O w -X w Y w Z w pose relationship, calculate the conversion relationship between the camera coordinate system and the world coordinate system c1 H w , set the center point O of the camera base coordinate system c1 and the center point O of the world coordinate system w on the same vertical axis, and the distance between them is h; O w -X w Y w Z w is translated to O c1 , and after rotating 90 + θ1 degrees around the X w axis, O c1 -X c1 Y c1 Z c1 can be obtained, and O c1 -X c1 Y c1 Z c1 and O w -X w Y w Z w have no scaling relationship. Therefore, the conversion relationship between O c1 -X c1 Y c1 Z c1 and O w -X w Y w Z w c1 H w can be calculated based on Equation (4):​ Among them, the rotation matrix R is calculated by left-multiplying single rotation matrices and is rotated clockwise based on the right-hand coordinate system, then: 1.3, Inverse perspective transformation and compensation correction: Based on the coordinate relationship chain o1 H i1 、 i1 H c1 、 c1 H w Construct the basic equation of inverse perspective transformation, and based on c2 H c1 and o2 H i2 、 i2 H c2 compensate and correct the basic equation of inverse perspective transformation according to the relationship chain. Inverse perspective transformation is to calculate the corresponding spatial top view in the world coordinate system based on the original pixel image obtained by the sensor. Therefore, the inverse perspective transformation equations of the left and right cameras are constructed based on Equations (6) and (7) respectively, where [x w1i , y w1i , z w1i and [x w2i , y w2i , z w2i are the coordinates of the same spatial point P i obtained based on the coordinate conversion chains of the left and right cameras; [u 1i , v 1i and [u 2i , v 2i are the pixel coordinates of the points p i and p 1i and p 2i on the images of the left and right cameras where the spatial point P 01 is mapped, which are directly read from the images. The parameters Δx, Δy, f, u 01 , v 02 , u 02 are obtained by camera calibration, and the parameters h and θ1 are obtained by a tape measure and an angle measuring instrument; Taking the coordinate transformation of the left camera as the basic equation of inverse perspective transformation, the coordinates [x w2i , y w2i obtained by using the inverse perspective transformation equation of the right camera and the coordinates [x w1i , y w1i obtained by the basic equation have a deviation [x w1i - x w2i , y w1i - y w2i , and the inverse perspective transformation is compensated and corrected; the compensation and correction are calculated by the gross error elimination method. Based on the formulas (8) and (9) shown, where n is the number of spatial points used for inverse perspective transformation. Considering the calculation error of the coordinates of the same spatial point, the threshold (x , y t , y t ) for gross error elimination is calculated as shown in the formulas (10) and (11). Based on this, the inverse perspective transformation coordinates (x wi , y wi ) after supplementary correction are obtained; 1.4, Locate the detection area for rice and wheat lodging assessment: Considering that the evaluation of rice and wheat lodging in the field is mainly used for the early detection of the real-time operation of the harvester, the rice and wheat areas to be detected are within a certain range in the front end of the harvester. Considering that the maximum width of the harvester operation area does not exceed the full cutting width, that is, the cutter bar width. Based on this and considering a certain margin, the width of the detection area is set to W d = W g + d, where W g is the width of the cutter bar and d is the detection margin; By performing Hough transform on the fixed area in the lower part of the original image, the outer edge of the cutting table is extracted, and the fitting line l at the forefront of the cutting table is obtained. 11 and the fitting line l at the rightmost end 12 ; Calculate the intersection point p 11 of the line l 11 and the leftmost edge of the image. Based on p 11 construct a line l 13 parallel to the U1 axis of the image coordinate system; And calculate the intersection point p 12 of l 13 and l 12 ; Move the point p 12 along the line l 13 by a distance d to obtain the point p 13 ; Move the point p 11 along the leftmost edge of the image in the opposite direction of the V1 axis by a distance L d to obtain the point p 14 . Based on the points p 11 , p 13 , p 14 construct a rectangle with two sides parallel to the V1 and U1 axes respectively as the evaluation and detection area for rice and wheat lodging. 1.5, Adaptive grid division of the original image based on perspective and inverse perspective transformations: Considering the influence of the top view clarity after inverse perspective transformation on the accuracy of lodging detection, the high-definition original image is used to detect lodging; at the same time, considering the influence of the imaging being larger near and smaller far away on the division of the detection area, the non-uniform adaptive division of the original image is achieved through key point perspective and inverse perspective transformation, so that the actual space area corresponding to each small area after division is equal; first, the key point p corresponding to the detection area is divided into 11 、p 14 Based on the calculation of formula (8) and (9), the key points correspond to the real points P1 and P4 in the three-dimensional space, and the inverse perspective transformation top view coordinates of P1 and P4 are obtained. w1 ,y w1 ) and P4=(x w4 ,y w4 ); According to the X coordinates of P1 and P4 in the world coordinate system w The distance on the axis is divided equally, and the number of equal divisions is m x , specifically divide the key points P equally di The coordinates (x di ,y di ) is calculated based on formula (12): Based on the obtained equally - spaced key points P di and the inverse calculation of equations (8) and (9), the image point p after perspective transformation is obtained 1di . It is set that the grid division on the V1 axis of the detection area in the original image is based on the key point p of perspective and inverse perspective transformation 1di to perform unequal division, and the grid division on the U1 axis is based on the set number of grids m y to perform uniform division. Finally, an adaptive grid division of the minimum area for detecting rice and wheat lodging during the operation of the harvester is obtained, laying a foundation for the real - time and accurate detection of on - vehicle rice and wheat lodging 3. A method for global assessment of rice and wheat lodging in the field based on the inter-frame correlation of the on-vehicle dynamic field of view according to claim 1, characterized in that: the specific process of step (2) is as follows: 2.1, Edge detection and repair in the S image channel: 1) Based on adaptive grid division, for a single static on-vehicle lodging assessment image, the images within each grid region are extracted, obtaining a set of grid regions with inconsistent sizes. Since rice and wheat crops grow densely and their organs cross, and the ground area is difficult to reach by light and is darker, the RGB color image is converted to the HSV space, and the saturation S channel, which represents the degree of color needle approaching the spectral color, is used for processing; 2) Considering that there are organ regions in rice and wheat crops that interfere with the stems, anisotropic diffusion is performed on the grid regions to connect discontinuous image edges, and no smoothing processing is performed in the direction perpendicular to the main direction to enhance the consistency of the image structure. For the enhanced grid regions, least squares fitting is used for edge detection. The quadratic polynomial parameters are calculated for each pixel point in the region. To extract the long edges corresponding to the stems and reduce the influence of short edges on detection, the convolution window size of the least squares fitting edge detection is designed to be 3×3; the second-order directional derivative perpendicular to the straight line direction of the pixel point is calculated, and the pixel point corresponding to the local maximum value of the second-order directional derivative is used as the edge point. All the detected edge points are connected to form an edge line; 3) Repair the disconnected edges through morphological opening and closing operations, and purify them based on the length characteristics of the edge lines. Finally, a set of continuous long edge lines on the stems is obtained in the grid area. The number of edge lines is the same as the number of stems. Let the number of stems detected in the grid area at the i-th row and j-th column be s ij ; 2.2, Two-dimensional line segment fitting and extraction of the main direction of three-dimensional multi-line segments: 1) Within the grid area of the two-dimensional image, perform linear fitting on each pixel of the s ij edges to obtain the two-dimensional straight line of the pointer stalk in the grid area of the i-th row and j-th column. Let the starting point and ending point of the k-th edge segment in the grid area be p ks =(x ks , y ks ) and p ke =(x ke , y ke ). Taking the abscissa as a reference, construct the fitting straight line segment equation of the k-th edge as shown in Equation (13), where a k and b k are the parameters of the fitting straight line segment equation of the k-th edge respectively, (x, y) are the points on the fitting straight line segment, and calculate that the starting point and ending point of the k-th fitting straight line segment are p' ks =(x ks , y' ks ) and p' ke =(x ke , y' ke ); y = a k x + b k , k = 1, 2,... s ij , x ks ≤ x ≤ x ke (13) 2) Calculate the starting points p' of all fitted line segments within the grid area based on the coordinate transformation of binocular vision ks and the ending points p' ke corresponding spatial points P' ks and P' ke with three-dimensional coordinates (x wks , y wks , z wks ) and (x wke , y wke , z wke ); Use unit vectors to represent the three-dimensional main direction of the grid area, and set the starting point of the direction at the center of the area to jointly represent the regional comprehensive pose of rice and wheat; First, calculate the vectors corresponding to each group of line segments to obtain a set of vector groups (x wke - x wks , y wke - y wks , z wke - z wks ); Based on equations (14) and (15), use the average angle of all vectors within the area as the angle of the unit vector of the comprehensive pose of rice and wheat to obtain the unit vector starting point q ij = (x ij0 , y ij0 , z ij0 ), where (α k , β k , γ k ) is the direction of the k-th vector within the area, and (α, β, γ) is the direction of : 2.3, Extraction of the lodging starting point based on KAZE features and correction of the main direction of the region: Starting from the entire detection area of a single frame, the starting point position of the lodging rice and wheat in the region is located based on KAZE feature points, and it is used to correct the direction of the stem pose vector of each grid region, realizing more accurate single-frame pose assessment of rice and wheat lodging. The root regions of rice and wheat are exposed from the dense crops in the lodging state, showing relatively obvious edge and point features. Therefore, to locate the midpoint of the root region of lodged rice and wheat based on the distribution of KAZE feature points, first, a nonlinear scale space is constructed based on the effective additive operator splitting technique and variable conductance diffusion, and KAZE feature points are extracted for the entire detection region of a single frame in the nonlinear scale space. By blurring the local area to reduce noise and retain the object boundary, a multi-scale two-dimensional KAZE feature point set is finally obtained. Then, based on the Euclidean distance between KAZE feature points, the point sets with a distance less than a certain distance threshold are clustered, and the number of points in each clustering category is counted. The center of the clustering category with the largest number of points is taken as the midpoint g0=(x0,y0,z0) of the root region of the lodged rice and wheat, which is also the starting point of the lodged rice and wheat within the single-frame image. To correct the regional principal direction for indicating the lodging direction of rice and wheat needles in the grid area of the i-th row and j-th column, first calculate the starting point q of the principal direction of the grid area of the i-th row and j-th column ij The positional relationship relative to the midpoint g0 of the root area is represented by a vector As shown in Equation (16) and based on The relationship with the principal direction vector of each grid area Adjust the direction and sign of the vector To obtain the corrected regional principal direction Take the vector direction starting from the root of rice and wheat as its lodging pose pointer:

4. A global evaluation method for field rice and wheat lodging based on the inter-frame correlation of on-vehicle dynamic fields of view according to claim 1, characterized in that: in step (3), the sparse point matching of the upper and lower frames combined with the primary and secondary movement directions is as follows: 3.1, Selection of evenly matched points for small grid division: To reduce the computational complexity and improve the real-time performance of in-vehicle detection, a method of uniform sampling with small grid division is adopted. Let the frame to be matched be I r and I r-1 , in the m r ×m x grid regions of adaptive division of the detection area in the image I y , divide each grid region into 3×3 small grids evenly, and take the center point of each small grid as the point M to be matched rs , s = 1, 2,..., m x ×m y ×9; 3.2, Estimation of the matching point region based on the primary movement direction: Based on the rotational speed sensors installed on the wheels / crawler bearings of the harvester, the real-time motion speed v of the harvester is calculated and obtained. r , considering that the motion distance of the harvester between the upper and lower frames is very short, and calculating with linear motion, the longitudinal difference between the upper and lower frame images is approximately s. r = v r t distance, which is converted to pixel distance d. r , based on the camera calibration parameters, the distance in space is converted to the image pixel distance s'. r , let the pixel coordinates of point M r on image I rs be (u rs , v rs ). Considering that the harvester is moving forward with the camera system, in the ideal case of only considering the linear motion of the harvester, the pixel coordinates of the point M rs corresponding to the spatial point mapped to point M r-1 on image I (r-1)s are (u rs , v rs - d r ); Therefore, considering that rice and wheat crops are fixed-root growing crops and their motion amplitude affected by external forces is small, based on the maximum range of their small-amplitude motion, an L (r-1)s ×W p pixel interval centered on M p is set as the matching point search interval range LW p ; 3.3, Search for matching points combined with the primary and secondary movement directions: 1) Let the image be I r The point to be matched on it is M rs =(u rs , v rs ). If M rs is the first matching point on I r , then set the center of LW p as the initial point M' rs0 for searching the matching point; if M rs is not the first matching point on I r , set the center points M rs of the eight small grid areas above, below, left, and right of M rst , t = 1, 2,..., 8. The corresponding motion vectors are MV rst =(u rst , v rst ), t = 1, 2,..., 8. Calculate the mean value of MV rst , t = 1, 2,..., 8 according to Equation (17) and based on locate the initial point for searching the matching point of M rs as 2) Considering that the matching of image points is not only related to pixel points but also involves the changes in their neighborhoods, a 3×3 pixel area centered on the point is used as a window, and the points are calculated one by one from left to right and top to bottom. Based on Equation (18), the sum of absolute differences (SAD) between points is obtained and used as the judgment criterion for point matching, where f ri represents the pixel value of the i-th point in the corresponding small window on the r-th frame. i corresponds to the points (u-1, v-1), (u, v-1), (u+1, v-1), …, (u+1, v+1) in sequence from 1 to 9. A threshold T is set. If SAD < T, the corresponding two points are matched; if SAD ≥ T, go to step 3) 3) Considering that the main movement direction of the harvester is the vertical direction of the image, a flat template in the main direction is constructed, and the SAD value of each point is calculated. If the SAD value of the center point of the template is the smallest, then this point is the corresponding matching point; if the SAD value of a certain other point in the template except the center point satisfies SAD < T, then this point is the corresponding matching point; otherwise, the point corresponding to the smallest SAD value among the other points in the template except the center point is selected as the new center point, and a new flat template in the new main direction is reconstructed, and go to step 3) until a matching point is found. 4) Based on the above process, the image frame I is obtained r and I r-1 on all small grids, the matching point pairs M rs and M' rs , where s = 1, 2,..., m x ×m y ×9.

5. A global evaluation method for lodging of field rice and wheat based on the inter-frame correlation of vehicle-mounted dynamic field of view according to claim 1, characterized in that: The specific process of step (4) is as follows: 4.1, Let the operation be T i The camera base coordinate system at time c1Ti -X c1Ti Y c1Ti Z c1Ti , T i and T i+1 The transformation relationship between the camera base coordinate systems at time c1T(i+1) H c1Ti , T i At time, according to the obtained T i and T i+1 The set of matching points M (i+1)s and M' (i+1)s between the image pairs at time, calculate the three-dimensional coordinate representations A i and T i+1 of the image points in the camera base coordinate systems O c1Ti -X c1Ti Y c1Ti Z c1Ti and O c1T(i+1) -X c1T(i+1) Y c1T(i+1) Z c1T(i+1) respectively, and A' (i+1)s ; (i+1)s ; 4.2, according to A (i+1)s and A' (i+1)s the mean value of the positional relationship between points, calculate O c1Ti -X c1Ti Y c1Ti Z c1Ti and O c1T(i+1) -X c1T(i+1) Y c1T(i+1) Z c1T(i+1) the translation relationship, according to A (i+1)s and A' (i+1)s the positional relationship between the line and vector formed by two points, calculate O c1Ti -X c1Ti Y c1Ti Z c1Ti and O c1T(i+1) -X c1T(i+1) Y c1T(i+1) Z c1T(i+1) the rotation relationship, obtain the transformation relationship between O c1Ti -X c1Ti Y c1Ti Z c1Ti and O c1T(i+1) -X c1T(i+1) Y c1T(i+1) Z c1T(i+1) ; c1T(i+1) H c1Ti ; 4.3, if T i is the last moment, then end the calculation of the coordinate chain relationship to obtain c1T1 H c1T0 , c1T2 H c1T1 , …, c1Tn H c1T(n-1) ; otherwise, set i = i + 1 and continue with steps 4.1 and 4.

2.

6. The global evaluation method for lodging of field rice and wheat based on the inter-frame correlation of vehicle-mounted dynamic field of view according to claim 1, characterized in that: The specific process of step (5) is as follows: 5.

1. Based on the coordinate relationship conversion chain constructed for each moment of the harvester operation, the needle vectors of the lodging poses calculated at each moment of the harvester operation are transformed to the camera base coordinate system O at time T0 c1T0 -X c1T0 Y c1T0 Z c1T0 . And based on the relationship between the camera base coordinate system and the world coordinate system, it is transformed to the world coordinate system O at time T0 w0 -X w0 Y w0 Z w0 . Thus, the three-dimensional pose of the rice and wheat during the harvester operation in one coordinate system is obtained 5.2, Based on the matching regions between adjacent frames, remove the overlapping grid regions to obtain the 3D pose distribution map of rice and wheat for the entire field from the initial to the end of the operation. Take the angle with the horizontal plane as the reference for the growth angle of rice and wheat, and set a threshold. If the angle is less than the threshold, it is determined as lodging. Based on this, calculate the global lodging area distribution map and lodging angle distribution map of rice and wheat for the entire field, and according to the projection on the horizontal plane, calculate the lodging direction of rice and wheat in each grid region to obtain the global lodging direction distribution map of rice and wheat.

7. A global evaluation method for lodging of field rice and wheat based on the inter-frame correlation of on-vehicle dynamic field of view according to claim 2, characterized in that: In Step 1.4, the operating speed of the harvester is 0.5 m / s - 1.8 m / s, and the length of the set detection area is L d = 4.8 m.

8. A global evaluation method for lodging of field rice and wheat based on the inter-frame correlation of vehicle-mounted dynamic fields of view according to claim 2, characterized in that: In step 1.4, the detection margin d is set to 35 cm.

Images