A tractor identification and positioning system and method based on multi-source information fusion

Abstract

The application discloses a tractor identification and positioning system and method based on multi-source information fusion, relates to the field of agricultural machinery identification and positioning, and comprises a support installed at the front end of a tractor, wherein a binocular stereo camera and a millimeter wave radar are installed on the support, the center of the left eye of the binocular stereo camera is aligned with the center of the millimeter wave radar, and the transverse coordinates of target identification and positioning information are kept consistent. The application identifies the tractor based on deep learning, positions the tractor by using the binocular stereo camera, can directly obtain the spatial position coordinates of the main machine, improves the precision and speed of tractor visual identification and positioning, adds the millimeter wave radar on the basis of tractor identification and positioning by using a visual sensor, adopts a distributed multi-source information fusion architecture, does not cause the paralysis of a perception system due to the identification failure of a single sensor, makes up for the defects of a single sensor in the robustness of tractor identification and positioning, considers the complex nonlinearity of tractor movement and the noise change of an actual field environment, fuses multi-sensor information based on adaptive square root unscented Kalman filtering, and improves the accuracy and reliability of the fusion result of tractor identification and positioning.

Description

Technical Field

[0001] This invention relates to the field of agricultural machinery identification and positioning, and in particular to a tractor identification and positioning system and method based on multi-source information fusion. Background Technology

[0002] In recent years, with the rapid development of electronic information technology, unmanned agricultural machinery has become a hot topic. Multi-source information fusion technology provides a new method for the development of agricultural machinery navigation, identification, and positioning systems. Multi-source information mainly includes visual, radar, and satellite information. Among these, machine vision navigation must first solve the problem of visual recognition and positioning. It determines the spatial position of objects in front relative to the camera based on image information captured by binocular cameras installed on the agricultural machinery. Visual sensors can acquire more complete information about the field environment and can promptly detect obstacles such as pedestrians and rocks, especially in mountainous and hilly areas where satellite navigation system signals are weak. To improve the reliability of agricultural machinery identification and positioning systems, visual recognition and positioning can be integrated with millimeter-wave radar identification and positioning to compensate for the shortcomings of single sensors in target identification and positioning accuracy. Modern agriculture is becoming increasingly automated, but satellite navigation cannot be applied to fields with weak satellite signals. To achieve master-slave following or multi-machine collaborative navigation operations, agricultural machinery identification and positioning technology based on multi-source information fusion is a key technology that urgently needs to be solved.

[0003] Existing vision-based agricultural target recognition and localization methods are mostly designed for static targets such as fruits, vegetables, flowers, and obstacles. Research on real-time visual recognition and localization of agricultural machinery such as tractors operating in the field is still lacking. Research on target recognition and localization methods based on multi-source information fusion mainly focuses on the recognition and localization of passenger vehicles and obstacles, but there is little research on the recognition and localization of agricultural machinery such as tractors. How to accurately and in real-time identify the tractor host moving ahead using machine vision and deep learning, and how to fuse agricultural machinery recognition and localization information from binocular stereo cameras and millimeter-wave radar to improve accuracy and robustness, all require further research. Therefore, a tractor recognition and localization system and method based on multi-source information fusion is needed to solve these problems. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a tractor identification and positioning system and method based on multi-source information fusion. It employs a distributed multi-source information fusion architecture to fuse the position information of the tractor's main unit, providing accurate target perception information for autonomous following by the slave tractor. The specific technical solution is as follows:

[0005] A tractor identification and positioning system based on multi-source information fusion includes a bracket installed at the front end of the tractor. A binocular stereo camera and a millimeter-wave radar are mounted on the bracket, and the center of the left eye of the binocular stereo camera is aligned with the center of the millimeter-wave radar to keep the lateral coordinates of the target identification and positioning information consistent.

[0006] The tractor's cab is equipped with an embedded host computer and a portable display. The embedded host computer is an NVIDIA AGX Xavier, and the binocular stereo camera is a ZED 2. Images are transmitted to the embedded host computer via USB, and commands are exchanged.

[0007] The millimeter-wave radar, model ARS408-21, transmits data and commands to the embedded host computer via CAN communication.

[0008] The portable display is connected to the embedded host computer via a Type-C interface.

[0009] The present invention also discloses a tractor identification and positioning method based on multi-source information fusion, including a visual identification and positioning module, a millimeter-wave radar identification and positioning module, and multi-sensor information fusion;

[0010] The visual recognition and positioning module identifies the tractor based on deep learning, uses a binocular stereo camera to locate the tractor, obtains the spatial coordinates of the host, and uses model pruning and knowledge distillation to make lightweight improvements to YOLO v4.

[0011] The multi-sensor fusion includes data synchronization, target measurement matching, information fusion based on adaptive square root unscented Kalman filtering, and target motion model construction. It fuses information from visual and millimeter-wave radar identification and positioning, specifically including the following steps:

[0012] S1. Spatially and temporally unify the horizontal and vertical coordinates of the tractor obtained separately by the binocular stereo camera and the millimeter-wave radar. Spatially unifying means unifying the target position identified by the millimeter-wave radar to the binocular stereo camera coordinate system based on the installation position of the two sensors. Temporally unifying means synchronizing the two sensors through software triggering based on the acquisition frequency of the two sensors.

[0013] S2. Match target measurements based on Mahalanobis distance, associate the measured and predicted values ​​of the target ahead, calculate the different attributes of the sample according to the confidence level, and determine which data were generated by the same tractor.

[0014] S3. Information fusion based on adaptive square root unscented Kalman filter, including obtaining Sigma point set, time update, and measurement update;

[0015] S4. Using a constant velocity model as the target motion model, tracking filtering through the constant velocity model can easily predict the measured value, making the fused measured value closer to the true value.

[0016] Preferably, in step S3, the nonlinear system is represented by the following equation, where the random variable X is disturbed by Gaussian white noise W(k), and the observed variable Z is disturbed by Gaussian white noise V(k):

[0017]

[0018] In the formula, f is the state equation function of the nonlinear system; h is the observation equation function of the nonlinear system; x is the state vector; and k represents time.

[0019] Preferably, in step S3, the Sigma point set is obtained:

[0020]

[0021] In the formula, n ut λ is the dimension of the random variable X; ut S is the scaling factor; S is the covariance matrix of the state variables;

[0022] Updated in time:

[0023]

[0024]

[0025]

[0026]

[0027] In the formula, i is the sampling point number, w m w represents the mean weights of the sigma points. p The covariance weights of the sigma points. It is an estimator of the random variable X; Let be the process noise estimation matrix, qr denotes QR decomposition, and choleupdate denotes Cholesky decomposition.

[0028] Preferably, in step S3, the measurement update includes:

[0029] a) Generate a new Sigma point set

[0030]

[0031] b) Substitute the new Sigma point set into the observation equation and calculate the residuals.

[0032] Z (i)(k+1|k)=h[X (i) (k+1|k)] (33)

[0033]

[0034]

[0035] In the formula, is a predicted observation mean; is the difference between the observed value and the observation mean;

[0036] c) Estimate the measurement noise matrix

[0037]

[0038]

[0039]

[0040] In the formula, d(k+1)=(1-b) / (1-b k+2 ), b is the forgetting factor, with a value range of 0<b<1; diag represents constructing a diagonal matrix; is a measurement noise estimation matrix;

[0041] d) Calculate the filtering gain

[0042]

[0043]

[0044]

[0045]

[0046] In the formula, P xz is the covariance matrix of the system state quantity and the observed quantity; S zz is the covariance matrix of the system observed quantity, and K is the Kalman filtering gain;

[0047] e) Update the system state quantity

[0048]

[0049] f) Calculate the square root of the posterior state variance

[0050] G(k+1|k+1)=K(k+1)S zz (k+1|k+1) (44)

[0051] S(k+1|k+1)=cholupdate{S(k+1|k),G,-1} (45)

[0052] In the formula, G is the posterior state matrix of the system;

[0053] g) Update the system process noise array

[0054]

[0055]

[0056]

[0057] Preferably, in step S4, under the discrete state of the system, the discrete state equation of the motion model is:

[0058] X(k+1)=F(k)X(k)+W(k) (49)

[0059] The state vector X is:

[0060] X = [y fus x fus v yfus v xfus ] T (50)

[0061] In the formula, y fus —Host's vertical position; x fus —Horizontal position of the host; v yfus —Host longitudinal speed;

[0062] v xfus —Host horizontal speed;

[0063] The system transition matrix F(k) is:

[0064]

[0065] The measured value vector is: Z = [y fus x fus ] T (52)

[0066] Beneficial effects of this invention:

[0067] 1. This invention uses deep learning to identify tractors, and employs model pruning and knowledge distillation to make lightweight improvements to YOLO v4, thereby improving the real-time performance of the algorithm. It uses a binocular stereo camera to locate the tractor, eliminating the need to develop an image stereo matching algorithm and directly obtaining the host's spatial position coordinates, which greatly improves the accuracy and speed of tractor visual recognition and positioning.

[0068] 2. Based on the visual sensor for tractor identification and positioning, this invention adds millimeter-wave radar and adopts a distributed multi-source information fusion architecture. This prevents the perception system from being paralyzed due to the failure of a single sensor, thus making up for the shortcomings of a single sensor in the robustness of tractor identification and positioning.

[0069] 3. This invention takes into account the complex nonlinearity of tractor movement and the noise changes in the actual field environment. Based on adaptive square root unscented Kalman filtering, it fuses information from multiple sensors and can adaptively adjust the process noise covariance matrix Q and the measurement noise covariance matrix R. By using QR decomposition and Cholesky decomposition, the positive definiteness of the covariance matrix is ​​maintained during the algorithm iteration calculation process, thereby improving the accuracy and reliability of the tractor identification and positioning fusion results. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the tractor identification and positioning system of the present invention;

[0071] Figure 2 This is a schematic diagram of the tractor identification and positioning method of the present invention;

[0072] Figure 3 This is the technical approach of the visual recognition and positioning module of the present invention;

[0073] Figure 4 This is a flowchart of the multi-source information fusion module of the present invention.

[0074] Among them, 1-tractor; 2-binocular stereo camera; 3-millimeter-wave radar; 4-embedded host computer; 5-portable display; 6-stand. Detailed Implementation

[0075] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0076] like Figures 1 to 4 As shown, a tractor identification and positioning system based on multi-source information fusion includes a bracket 6 installed at the front end of the tractor 1. A binocular stereo camera 2 and a millimeter-wave radar 3 are installed on the bracket 6, and the center of the left eye of the binocular stereo camera 2 is aligned with the center of the millimeter-wave radar 3, so that the lateral coordinates of the target identification and positioning information remain consistent.

[0077] The cab of the tractor 1 is equipped with an embedded host computer 4 and a portable display 5. The embedded host computer 4 is an NVIDIA AGX Xavier, and the binocular stereo camera 2 is a ZED 2. The image is transmitted to the embedded host computer 4 via USB, and commands are exchanged.

[0078] The millimeter-wave radar 3 is model ARS408-21, and it transmits data and commands to the embedded host computer 4 via CAN communication.

[0079] The portable display 5 is connected to the embedded host computer 4 via a Type C interface.

[0080] The present invention also discloses a tractor identification and positioning method based on multi-source information fusion, including a visual identification and positioning module, a millimeter-wave radar identification and positioning module, and multi-sensor information fusion;

[0081] The visual recognition and positioning module identifies the tractor based on deep learning, uses a stereo camera 2 to locate the tractor 1, obtains the spatial coordinates of the host machine, and performs lightweight improvements to YOLO v4 using model pruning and knowledge distillation. Based on the improved YOLO v4 and stereo camera 2, this invention proposes a tractor recognition and positioning method that can be implemented in real-time on actual vehicles. Its technical approach is as follows: Figure 3 As shown, the specific work steps are as follows:

[0082] (1) Using a stereo camera 2, different tractors 1 were photographed at different angles and distances on sunny and cloudy days. The image format was .png. LabelImg software was used to manually label all the images. The cab of tractor 1 was used as the recognition target to construct a tractor dataset containing training and test sets.

[0083] (2) Input the tractor training set into the YOLO v4 deep learning network for basic training to obtain a high-precision tractor 1 recognition model weight file. The high-precision model has too many parameters and too much computation, which cannot meet the real-time recognition requirements.

[0084] (3) Perform sparse training on the weight file of the tractor 1 recognition model generated by the basic training, and use L1 regularization to adjust the γ coefficient of the BN (Bach Normalization) layer of the YOLO v4 tractor recognition model to make the model obtained by the basic training sparse.

[0085] (4) After sparse training is completed, the contribution of each input layer is evaluated based on the γ value of the BN layer. The channel pruning algorithm is used to remove the low-contribution channels and keep only the high-contribution channels. Different channel pruning ratios are selected, and multiple model pruning experiments are conducted to select the optimal channel pruning ratio.

[0086] (5) Channel pruning mainly reduces the size of the model, while layer pruning mainly affects the inference speed of the model. The CBM before each shortcut layer is evaluated, the mean γ of each layer is sorted, and then the shortcut layer with the smallest mean is pruned according to the layer pruning algorithm. The optimal number of layers to be pruned is determined through multiple trials.

[0087] (6) After pruning, the structure and quality of the Tractor 1 recognition model have changed significantly, especially the accuracy has decreased. Fine-tuning training based on knowledge distillation can effectively restore the accuracy. The teacher-student knowledge distillation strategy is adopted to carry out knowledge transfer training, thereby improving the accuracy of the student network and enabling the Tractor 1 recognition model to meet the requirements of the task for both accuracy and speed.

[0088] (7) In the program, call the stereo camera 2SDK to obtain the depth map of tractor 1, and obtain the positioning coordinates (x, y) of tractor 1 relative to stereo camera 2. bc y bc ).

[0089] The specific working steps of the millimeter-wave radar identification and positioning module are as follows:

[0090] (1) The millimeter-wave radar 3 has a detection range of up to 250m and can detect 256 targets at the same time. A large number of invalid target data will be generated within the same detection cycle, which will interfere with the identification and detection of the host tractor 1 in front. During the master-slave follow operation of tractor 1, a fixed horizontal and vertical distance needs to be maintained. Based on these conditions, the present invention determines the target selection strategy and filters out data of stationary targets outside the set distance.

[0091] (2) Add a CAN_TTL conversion module between the millimeter-wave radar 3 and the embedded host computer 4. The TX and RX interfaces of the CAN_TTL module are connected to the TX and RX I / O ports of the embedded host computer, respectively.

[0092] (3) Initialize and configure the millimeter-wave radar 3, parse the CAN messages according to the dbc file, and read the required tractor host's position information (x) in real time through the embedded host computer 4. mwr y mwr ).

[0093] The multi-sensor information fusion includes data synchronization, target measurement matching, information fusion based on adaptive square root unscented Kalman filtering, and target motion model construction. It fuses information from visual and millimeter-wave radar identification and positioning, specifically including the following steps:

[0094] S1. The horizontal and vertical coordinates of the tractor 1 obtained separately by the stereo camera 2 and the millimeter-wave radar 3 are unified in space and time. Spatial unification is to unify the target position identified by the millimeter-wave radar 3 to the coordinate system of the stereo camera 2 according to the installation position of the two sensors. Temporal unification is to synchronize the two sensors through software triggering according to the acquisition frequency of the two sensors.

[0095] S2. Match target measurement values ​​based on Mahalanobis distance, associate the measured and predicted values ​​of the target ahead, calculate the different attributes of the sample according to the confidence level, and determine which data were generated by the same tractor 1.

[0096] S3. Information fusion based on adaptive square root unscented Kalman filter, including obtaining Sigma point set, time update, and measurement update;

[0097] S4. Using a constant velocity model as the target motion model, tracking filtering through the constant velocity model can easily predict the measured value, making the fused measured value closer to the true value.

[0098] In step S3, the nonlinear system is represented by the following equation: the random variable X is disturbed by Gaussian white noise W(k), and the observed variable Z is disturbed by Gaussian white noise V(k):

[0099]

[0100] In the formula, f is the state equation function of the nonlinear system; h is the observation equation function of the nonlinear system; x is the state vector; and k represents time.

[0101] In step S3, the Sigma point set is obtained:

[0102]

[0103] In the formula, n ut λ is the dimension of the random variable X; ut S is the scaling factor; S is the covariance matrix of the state variables;

[0104] Updated in time:

[0105] X (i) (k+1|k)=f[k,X (i) (k|k)] (54)

[0106]

[0107]

[0108]

[0109] In the formula, i is the sampling point number, w m w represents the mean weights of the sigma points. p The covariance weights of the sigma points. It is an estimator of the random variable X; Let be the process noise estimation matrix, qr denotes QR decomposition, and choleupdate denotes Cholesky decomposition.

[0110] In step S3, the measurement update comprises:

[0111] a) generating a new set of Sigma points

[0112]

[0113] b) substituting the new set of Sigma points into the observation equation and calculating a residual

[0114]

[0115]

[0116]

[0117] wherein, is a predicted observation mean; is a difference between an observed value and the observation mean;

[0118] c) estimating a measurement noise matrix

[0119]

[0120]

[0121]

[0122] wherein, d(k+1)=(1-b) / (1-b k+2 ), b is a forgetting factor, with a value range of 0<b<1; diag represents constructing a diagonal matrix; is a measurement noise estimation matrix;

[0123] d) calculating a filtering gain

[0124]

[0125]

[0126]

[0127]

[0128] wherein, P xz is a covariance matrix of system state variables and observation variables; S zz is a covariance matrix of system observation variables, and K is a Kalman filtering gain;

[0129] e) updating the system state variables

[0130]

[0131] f) Calculate the square root of the posterior state variance.

[0132] G(k+1|k+1)=K(k+1)S zz (k+1|k+1) (70)

[0133] S(k+1|k+1)=cholupdate{S(k+1|k),G,-1} (71)

[0134] In the formula, G is the posterior state matrix of the system;

[0135] g) Update the system process noise array

[0136]

[0137]

[0138]

[0139] In step S4, under the discrete state of the system, the discrete state equation of the motion model is:

[0140] X(k+1)=F(k)X(k)+W(k) (75)

[0141] The state vector X is:

[0142] X = [y fus x fus v yfus v xfus ] T (76)

[0143] In the formula, y fus —Host's vertical position; x fus —Horizontal position of the host; v yfus —The longitudinal speed of the host machine; v xfus —Host horizontal speed;

[0144] The system transition matrix F(k) is:

[0145]

[0146] The measured value vector is: Z = [y fus x fus ] T (78)

[0147] As is known from common technical knowledge, this invention can be implemented through other embodiments that do not depart from its spirit or essential characteristics. Therefore, the disclosed embodiments described above are merely illustrative in all respects and are not the only ones. All modifications within the scope of this invention or its equivalents are included in this invention.

Claims

1. A method for identifying and locating a tractor based on multi-source information fusion, applied to a tractor identification and positioning system based on multi-source information fusion, characterized in that: The system includes a bracket (6) installed at the front end of the tractor (1), on which a binocular stereo camera (2) and a millimeter-wave radar (3) are mounted, and the center of the left eye of the binocular stereo camera (2) is aligned with the center of the millimeter-wave radar (3) so that the lateral coordinates of the target identification and positioning information remain consistent. The cab of the tractor (1) is equipped with an embedded host computer (4) and a portable display (5). The embedded host computer (4) is an NVIDIA AGX Xavier, and the binocular stereo camera (2) is a ZED 2. The image is transmitted to the embedded host computer (4) via USB and commands are exchanged. The millimeter-wave radar (3) is model ARS408-21, and transmits data and commands to the embedded host computer (4) via CAN communication; The portable display (5) is connected to the embedded host computer (4) via a Type C interface; The method includes a visual recognition and positioning module, a millimeter-wave radar recognition and positioning module, and multi-sensor information fusion. The visual recognition and positioning module identifies the tractor based on deep learning, uses a binocular stereo camera (2) to locate the tractor (1), obtains the spatial coordinates of the host, and uses model pruning and knowledge distillation to make lightweight improvements to YOLO v4. The multi-sensor information fusion includes data synchronization, target measurement matching, information fusion based on adaptive square root unscented Kalman filtering, and target motion model construction. It fuses information from visual and millimeter-wave radar identification and positioning, specifically including the following steps: S1. The horizontal and vertical coordinates of the tractor (1) in front obtained separately by the binocular stereo camera (2) and the millimeter-wave radar (3) are unified in space and time. The spatial unification is to unify the target position identified by the millimeter-wave radar (3) into the coordinate system of the binocular stereo camera (2) according to the installation position of the two sensors. The temporal unification is to synchronize the two sensors through program soft triggering based on the adoption frequency of the two sensors. S2. Match target measurement values ​​based on Mahalanobis distance, associate the measured and predicted values ​​of the target ahead with data, calculate the different attributes of the sample according to the confidence level, and determine which data were generated by the same tractor (1). S3. Information fusion based on adaptive square root unscented Kalman filter, including obtaining Sigma point set, time update, and measurement update; S4. Using a constant velocity model as the target motion model, tracking filtering through the constant velocity model can easily predict the measured value, making the fused measured value closer to the true value. In step S4, under the discrete state of the system, the discrete state equation of the motion model is: (23) State vector X for: (24) In the formula, y fus —The vertical position of the host computer; x fus —Horizontal position of the host; v yfus —Host longitudinal speed; v xfus —Host horizontal speed; System transition matrix F ( k )for: (25) The measured value vector is: (26).

2. The identification and positioning method of a tractor identification and positioning system based on multi-source information fusion as described in claim 1, wherein in step S3, the nonlinear system is represented by the following formula, where the random variable X is disturbed by Gaussian white noise W(k), and the observed variable Z is disturbed by Gaussian white noise V(k): In the formula, f is the state equation function of the nonlinear system; h is the observation equation function of the nonlinear system; x is the state vector; and k represents time.

3. The identification and positioning method for a tractor identification and positioning system based on multi-source information fusion as described in claim 2, wherein in step S3, the Sigma point set is obtained: (1) In the formula, n ut λ is the dimension of the random variable X; ut S is the scaling factor; S is the covariance matrix of the state variables; Updated in time: (2) (3) (4) (5) In the formula, i is the sampling point number, w m w represents the mean weights of the sigma points. p The covariance weights of the sigma points. It is an estimator of the random variable X; Let be the process noise estimation matrix, qr denotes QR decomposition, and choleupdate denotes Cholesky decomposition.

4. The identification and positioning method of a tractor identification and positioning system based on multi-source information fusion as described in claim 3, wherein step S3, measurement update includes: a) Generate a new Sigma point set (6) b) Substitute the new Sigma point set into the observation equation and calculate the residuals. (7) (8) (9) In the formula, This is the predicted observation mean; It is the difference between the observed value and the observed mean; c) Estimating the measurement noise array (10) (11) (12) In the formula, , b It is the forgetting factor, with a value range of 0 < b <1; diag indicates the construction of a diagonal matrix; For measuring noise estimation matrix; d) Calculate the filter gain (13) (14) (15) (16) In the formula, P xz S is the covariance matrix of the system state variables and the observed variables; zz Let K be the covariance matrix of the system observations, and K be the Kalman filter gain. e) Update system state variables (17) f) Calculate the square root of the posterior state variance. (18) (19) In the formula, G is the posterior state matrix of the system; g) Update the system process noise array (20) (21) (22)。

Images