A damping camera-imu device and a navigation device and a navigation method in an agricultural machine seedling transplanting system

Abstract

The application discloses a damping camera-IMU device and a navigation device and a navigation method in a rice transplanter system, and the system comprises a hardware part and a software part; the hardware part adopts a four-eye wide-angle lens, an independently designed multi-camera, an IMU arrangement and a suspension damping scheme, so that the measurement accuracy and the anti-vibration performance are ensured; the software part is modified and optimized based on an OpenVINS algorithm, so that feature point tracking, camera pose estimation and IMU data fusion are realized, and the navigation accuracy and robustness are improved; through the independently designed visual hardware system and the improved agricultural machine visual navigation software, the function of stably positioning the agricultural machine under the premise of limiting the cost is realized, the cost of intelligent transformation of the agricultural machine is reduced, and the super-wide field of view and the stereoscopic vision function are provided, so that the application can be reused for obstacle avoidance, target detection and other tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to an automatic positioning and navigation system for agricultural machinery, specifically a shock-absorbing camera-IMU device and navigation device and method for an agricultural transplanter, particularly suitable for precise attitude estimation and positioning in environments without GPS signals or with limited GPS signals. BACKGROUND

[0002] The advent of transplanter has greatly reduced the intensity of agricultural transplanting, which is a laborious task. However, existing transplanters still face the problem of long working time due to slow driving speed and large working range, and the problem of great damage to the driver's health due to large working noise and vibration. At the same time, due to the application scenario of the equipment, it is also determined that it must have high stability (it can operate continuously in rainy days and poor GPS signal conditions); large-area popularization should be based on the existing agricultural machinery for modification, which requires low modification cost. Therefore, the present application proposes a camera-IMU structure for high-vibration environments (agricultural machinery), which combines visual-IMU mileage calculation method and closed-loop control system to achieve precision control of ±7.5 cm in more than 80% of the 25m operating range. At the same time, the visual system with ultra-wide field of view and front binocular stereo vision can also be used for obstacle avoidance (collision avoidance), target detection (seeding condition detection) and other tasks, realizing the reuse of hardware and reserving space for higher and more demanding tasks.

[0003] Although the existing transplanter has reduced the intensity of agricultural transplanting labor, it still has the problems of long working time and large working noise and vibration. In order to meet the demand of automated agriculture, the present application aims to provide a camera-IMU structure with high stability and low price, and a shock-absorbing device to solve the problems of existing transplanters.

[0004] Agricultural machinery navigation mainly relies on GPS system for positioning. However, due to the particularity of the farmland environment, such as crop shading, signal interference and other factors, the stability and accuracy of GPS signals are limited. Therefore, a method and system capable of realizing accurate navigation in environments without GPS signals or with limited GPS signals is needed. Existing visual navigation systems mainly based on monocular or binocular cameras for attitude estimation and positioning. However, in the farmland environment, affected by factors such as crop shading and vibration interference, traditional visual navigation systems often have difficulty in obtaining stable and accurate navigation results. SUMMARY

[0005] The present application improves the prior art, and provides a damping camera-IMU device, a navigation device and a navigation method in a rice transplanter system, which realizes accurate positioning and navigation of the rice transplanter in an environment without GPS signals or with limited GPS signals through accurate attitude estimation and positioning.

[0006] The present application is realized by the following technical solutions:

[0007] The present application discloses a damping camera-IMU device in a rice transplanter system, which comprises a camera-IMU structural member, a camera arranged in front of a side plate of the camera-IMU structural member, an IMU arranged in the middle of an upper plate of the camera-IMU structural member, an I-shaped member fixedly connected to the upper plate of the camera-IMU structural member, a punched steel plate located above the I-shaped member, a T-shaped structural member fixed to the upper surface of the punched steel plate, a through hole formed in the T-shaped structural member for a steel pipe on the rice transplanter to pass through, a sponge pad above the I-shaped member, and a diagonal tension spring connecting the I-shaped member and the punched steel plate.

[0008] As a further improvement, the camera-IMU structural member of the present application is a cover structure containing a cavity, which comprises an upper plate and four side plates, and at least two cameras are mounted on the front side plate of the side plates.

[0009] As a further improvement, the front side plate of the present application comprises a front plate and two inclined plates on both sides of the front plate, and four cameras are mounted on both ends of the front plate and the two inclined plates, wherein the cameras are four-eye wide-angle lenses, and the angle between the front plate and the inclined plates is 135 degrees.

[0010] As a further improvement, the present application is characterized in that the IMU is inverted and mounted on the central part of the lower surface of the upper plate.

[0011] As a further improvement, the present application is characterized in that the two horizontal edges of the I-shaped member correspond to two punched steel plates respectively, the two punched steel plates are parallel to the corresponding two horizontal edges respectively, and the two ends of the punched steel plates are connected to the two ends of the corresponding horizontal edges through diagonal tension springs.

[0012] As a further improvement, the diagonal tension spring of the present application is in a stretched state, the distance between the two punched steel plates is greater than the vertical edge of the I-shaped member, and the punched steel plates are located above the horizontal edges.

[0013] As a further improvement, the angle of the diagonal tension spring of the present application is 45 degrees, and the IMU is integrated on the collection plate.

[0014] The present application also discloses a camera-IMU navigation device in a rice transplanter system, which comprises a camera-IMU collection plate, an IMU fixedly connected to the camera-IMU collection plate, a camera connected to the camera-IMU collection plate, and an industrial computer connected to the camera-IMU collection plate.

[0015] The camera-IMU acquisition board is responsible for controlling the sampling trigger of the camera and the timestamp synchronization between multiple cameras, ensuring accurate acquisition of image data and close proximity between different cameras;

[0016] The camera includes two cameras for forming a forward binocular combination, and another two cameras facing left front and right front, providing a wider field of view;

[0017] The IMU obtains attitude information such as acceleration and angular velocity, and the attitude information contains the attitude change and motion state of the agricultural machine;

[0018] The industrial computer is responsible for processing the received camera and IMU inertial measurement data, and running the positioning and navigation algorithm to output the attitude estimation and positioning result of the agricultural machine.

[0019] The application also discloses an autonomous navigation method of a camera-IMU device in an agricultural machine transplanting system, comprising: obtaining image data and IMU inertial measurement data; the image data and the IMU inertial measurement data are obtained by calling the camera through the camera-IMU acquisition board; the image data includes image data of four cameras, which are used for feature point extraction, tracking and attitude estimation; the IMU inertial measurement data includes attitude information such as acceleration and angular velocity, which is used for attitude estimation assistance;

[0020] Using the image data and the inertial measurement data, the state of the Kalman filter is initialized, including the pose and velocity of the camera and the bias of the IMU, to obtain the pose and velocity of the camera and the bias of the IMU in the initial state; this step only needs to be executed once at the beginning of the algorithm, and is used to initialize the initial pose and initial velocity of the camera and the initial bias of the IMU mentioned in step 4;

[0021] Through the image data, a feature extraction algorithm (such as ORB, SIFT, etc.) is used to extract feature points in the image data, to obtain pixel coordinates and descriptors of the feature points;

[0022] A feature point matching algorithm is run on the descriptors, and the feature points are tracked and matched between consecutive image frames or images of different cameras at the same time, to establish their corresponding relationship, and the matched feature points are obtained;

[0023] By matching the feature points and the inertial measurement data, a Kalman filter is used for visual and inertial state estimation, and an observation update and a prediction update are used to update the pose and velocity of the camera and the bias of the IMU, so as to obtain the updated pose and velocity of the camera and the bias of the IMU; the observation update and the prediction update are alternately performed to realize continuous state estimation and update. The observation update uses visual measurement to correct the estimation error, and the prediction update uses inertial measurement to predict the motion of the system. In this way, by continuously iterating the observation update and the prediction update, the Kalman filter gradually converges to more accurate pose and velocity estimation of the camera and bias estimation of the IMU. The results of the updated pose and velocity of the camera and the bias of the IMU are output.

[0024] As a further improvement, the present application observation update step:

[0025] By matching the feature points, the state of the Kalman filter is updated using visual measurement, the change of the pixel coordinates of the feature points in the two frames and the inertial measurement, the acceleration and angular velocity of the IMU;

[0026] Using visual measurement, the observation of the feature points is compared with the corresponding feature points in the Kalman filter, and the observation error is calculated using the re-projection error;

[0027] According to the observation error, the Kalman gain is calculated, which is used to correct the state estimation value of the Kalman filter, so as to obtain the updated pose, velocity and bias estimation;

[0028] Prediction update step:

[0029] By the inertial measurement data, the change of the pose, velocity and bias of the camera and the IMU is predicted by integrating the acceleration and angular velocity of the IMU;

[0030] According to the prediction model, the prediction step of the Kalman filter is used to update the prediction estimation of the state.

[0031] The present application relates to a camera-IMU structure and a damping device in an agricultural machine system, which are used to realize accurate IMU data acquisition and multi-view visual information acquisition in a high-vibration environment. The camera-IMU structure adopts a four-view wide-angle lens, and a multi-camera and IMU arrangement and mounting structure is designed. The damping device adopts a spring suspension and sponge pad mounting method, which effectively reduces the interference of high-frequency vibration on the IMU. The present application has the characteristics of low price and high stability, and is suitable for the modification of existing agricultural machines to meet the needs of automated agriculture.

[0032] The application is used for fixing the camera-IMU structure on the agricultural machine and reducing the interference of high-frequency vibration on the IMU, and comprises spring suspension and sponge pad, and the stability of the structure and the effective elimination of vibration are ensured through the setting of the angle of the cable-stayed structure; the combination of the spring suspension and the sponge pad makes the structure have better damping effect, the interference of high-frequency vibration on the IMU is reduced through the spring suspension and the sponge pad, and accurate sensing in the high-vibration environment is realized, and the accuracy of positioning and navigation realized by subsequent algorithms is improved.

[0033] The beneficial effects of the application are as follows:

[0034] 1. The spring and the sponge play a damping role, the spring can provide tension, and the sponge is opposite to the spring, because the vibration generated by the diesel engine of the agricultural machine is mainly up-down vibration, therefore, the design can provide damping effect in the most intense longitudinal direction, and the setting of the cable-stayed spring can ensure that the camera-IMU structure has stability in the front-back and left-right dimensions; the I-shaped piece has the functions of strengthening the strength of the camera-IMU structure, providing mounting hole positions for the spring, strengthening the self-weight of the I-shaped piece and the camera-IMU structure assembly, enhancing the tightness of the spring and strengthening the stability. The main function of the punched steel plate is to provide distance for the cable-stayed spring in the front-back direction and provide hole positions for the spring installation. The main function of the T-shaped structure is to connect the steel pipe on the agricultural machine with the punched steel plate to provide a stable mounting platform.

[0035] The arrangement mode of the camera is that the two front-view cameras are oriented in the same direction, and the baseline distance is 20 cm, the accuracy of multi-view visual information acquisition and attitude estimation is realized through the self-designed multi-camera and IMU arrangement. Among them, the two front-view cameras are used for binocular stereo matching, the left and right sides have overlapping fields of view with the front view, and the total field of view reaches 210°. The low-cost IMU has a sampling rate of 200hz, the cost is low, and the IMU is integrated on the video acquisition board, which further simplifies the installation difficulty, enhances the stability and ensures the measurement accuracy. Four cameras are used because the front-view binoculars can form a stereo vision, which can strengthen the recovery of depth, and the left and right binoculars can expand the field of view and better track feature points in the farmland. The number of cameras can be reduced to a monocular camera, but the robustness and accuracy of the algorithm will decrease due to the reduction of the viewing angle and the lack of binocular stereo vision.

[0036] 3. The application improves the consistency of the system through hardware trigger synchronization; the IMU is centrally designed, so that the vibration amplitude at the IMU is smaller than the maximum amplitude of the four cameras in various cases.

[0037] 4、The application realizes accurate pose estimation and positioning in the environment without GPS signal or limited GPS signal through the self-designed visual hardware system and improved OpenVINS algorithm. The hardware part includes four wide-angle lenses, multi-camera and IMU arrangement and suspension damping scheme, providing ultra-wide field of view and accuracy of pose estimation. The software part is based on the improved OpenVINS algorithm, and through adjusting parameters and fusing multi-view information, accurate pose estimation and positioning are realized. In addition, the hardware system of the application has low cost, is suitable for modification of existing agricultural machines, reduces the cost and implementation difficulty of intelligent modification of agricultural machines, has wide application prospect, and provides lower cost, more stable and more accurate agricultural machine navigation solution in farmland operation.

[0038] The application relates to an intelligent visual navigation system for agricultural machines, which realizes stable positioning of agricultural machines under the premise of limited cost through a self-designed visual hardware system and improved visual navigation software for agricultural machines, reduces the cost of intelligent modification of agricultural machines, and provides ultra-wide field of view and stereo vision function, which can be reused for obstacle avoidance and target detection. The system comprises a hardware part and a software part. The hardware part adopts four wide-angle lenses, self-designed multi-camera, IMU arrangement and suspension damping scheme, thereby guaranteeing measurement accuracy and anti-vibration performance. The software part is modified and optimized based on the OpenVINS algorithm, realizes feature point tracking, camera pose estimation and IMU data fusion, and improves the accuracy and robustness of navigation.

[0039] The system fully utilizes the self-designed visual hardware system and optimized visual navigation software for agricultural machines, realizes stable and accurate positioning under the premise of limited IMU and camera cost, reduces the cost of intelligent modification of agricultural machines, and has ultra-wide field of view and stereo vision function, which can be used for obstacle avoidance and target detection.

[0040] 5、Spring suspension and sponge pad mounting: the application adopts a spring suspension and sponge pad mounting damping scheme, which can effectively reduce the interference of high-frequency vibration on the IMU. Through the design of the inclined spring, the stability in the front-back and left-right dimensions is guaranteed. This damping scheme has good damping effect in the high-vibration environment of agricultural machines, and improves the accuracy of visual and IMU data.

[0041] 6、Wide field of view: the camera system adopting four wide-angle lens arrangement realizes front-view binocular stereo matching and wide field of view. Through the combination of front-view cameras and left and right view cameras, the total field of view reaches 210 degrees, which can cover a wider farmland area and improve the target detection and environment perception capability.

[0042] 7. High stability: The stability of the camera-IMU structure during the operation of the agricultural machine is ensured by the design of the I-beam, inclined spring, and punched steel plate structure. At the same time, the shock-absorbing device of the spring and sponge can effectively suppress the interference of agricultural machine vibration on IMU data, improving the accuracy of positioning and navigation.

[0043] 8. Low cost: Compared with the use of inertial navigation and differential GPS equipment with a price of tens of thousands of yuan, the system cost of the invention is lower. Using a low-cost camera and IMU, combined with visual-IMU mileage calculation method and closed-loop control system, the precision control can be realized within the price range of about 3000 yuan. Such a low-cost solution makes the device more easily promoted and applied in the field of agricultural machinery.

[0044] 9. Hardware reuse: The vision system in the device can not only be used for positioning and navigation, but also for obstacle avoidance, target detection, etc., realizing hardware reuse. This reuse design improves the functionality and flexibility of the device, leaving room for more task requirements.

[0045] 10. Add a binocular matching step to improve the accuracy and robustness of feature point matching by using a binocular camera for feature point matching;

[0046] Improve the use of a four-camera camera to increase the stability of feature point tracking and matching, and use a wider field of view to achieve effective tracking in a feature-repeating single farmland environment; use four cameras for feature extraction and tracking, obtain the correspondence of feature points between consecutive image frames through optical flow method or feature point matching algorithm, calculate the camera pose change through the analysis of feature point disparity, and optimize the accuracy of pose estimation through nonlinear optimization method;

[0047] Adjust the IMU noise, image downsampling ratio, initialization threshold, static detection, etc. to improve the robustness and stability of the algorithm; use IMU data for auxiliary pose estimation, improve the accuracy of pose estimation through IMU pre-integration and nonlinear optimization;

[0048] Introduce a static detection mechanism to reduce false matching in the static state and improve the robustness of navigation by distinguishing image features in the static and moving states of the agricultural machine;

[0049] Reduce the number of feature points to reduce the computational load and speed up the algorithm;

[0050] After processing and fusion of image data and IMU data, the pose estimation and positioning of the agricultural machine are realized.

[0051] Inertial part vision and inertial fusion: the information of the visual part and the inertial part is fused through filtering or optimization method to obtain more accurate and stable pose estimation and positioning results;

[0052] Robust optimization: Through optimization measures such as binocular matching, feature point management and parameter adjustment, the performance and stability of the algorithm in the agricultural machinery navigation scene are improved.

[0053] 11. Optimization for agricultural machinery navigation scene, such as binocular matching, feature point management and parameter adjustment. Binocular matching uses binocular camera to match feature points, improving matching accuracy and robustness. Feature point management includes adding new feature points, deleting invalid feature points and updating feature point depth estimation to maintain the stability and consistency of feature points. Parameter adjustment includes adjusting IMU noise, image downsampling ratio, initialization threshold and static detection parameters to improve the robustness and stability of the algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is the overall structure schematic diagram of the present application;

[0055] Figure 2 is the front view schematic diagram of the overall structure of the present application;

[0056] Figure 3 is the left view schematic diagram of the overall structure of the present application;

[0057] Figure 4 is the schematic diagram of the high-vibration environment vision-IMU positioning and navigation system for agricultural machinery of the present application;

[0058] Figure 5 is the flowchart of the software system of the present application.

[0059] In the figure, 1 is a T-shaped structural member, 2 is a punched steel plate, 3 is a cable-stayed spring, 4 is a sponge, 5 is an I-beam, 6 is a camera IMU structural member, 7 is a front plate, 8 is an inclined plate, 9 is a horizontal edge, and 10 is a vertical edge. DETAILED DESCRIPTION

[0060] The embodiments of the present application involve both hardware and software. In terms of hardware, the present application requires that the agricultural machinery be equipped with a four-lens wide-angle lens, and that the camera and IMU be arranged and installed according to the design requirements to ensure the stability and measurement accuracy of the system. In terms of software, the present application requires the use of improved OpenVINS algorithm to realize the pose estimation and positioning of agricultural machinery through feature point extraction, tracking and pose estimation.

[0061] The embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings:

[0062] Figure 1 is the overall structure schematic diagram of the present application; Figure 2 is the front view schematic diagram of the overall structure of the present application; Figure 3It is the left view schematic diagram of the overall structure of the present application; the present application discloses a damping camera-IMU device in a rice transplanter system, the device comprises a camera IMU structural member 6, a camera arranged at the front of the side plate of the camera IMU structural member 6, an IMU arranged at the middle part of the upper plate of the camera IMU structural member 6, an I-shaped piece 5 connected and fixed to the upper plate of the camera IMU structural member 6, a punched steel plate 2 located above the I-shaped piece 5, a T-shaped structural member 1 fixed to the upper surface of the punched steel plate 2, a through hole is formed in the T-shaped structural member 1 for the steel pipe on the agricultural machine to pass through, a sponge 4 is padded above the I-shaped piece 5, and the I-shaped piece 5 and the punched steel plate 2 are connected through a cable-stayed spring 3. The sponge 4 is clamped between the I-shaped piece 5 and the steel pipe on the agricultural machine, in the embodiment, four cameras are mounted on the front side plate of the side plate, the front side plate comprises a front plate 7 and an inclined plate 8 on both sides of the front plate 7, cameras are mounted at both ends and both sides of the front plate 7, and a total of four cameras are arranged, the cameras are four wide-angle lenses, and the angle between the front plate 7 and the inclined plate 8 is 135 degrees. The punched steel plate 2 is connected with the steel pipe on the agricultural machine through the T-shaped structural member 1. The camera IMU structural member 6 is connected with the I-shaped piece 5 through screws.

[0063] The IMU is invertedly arranged at the middle part of the lower surface of the upper plate. The upper plate is perpendicular to the side plate at 90 degrees.

[0064] The two horizontal edges 9 of the I-shaped piece 5 correspond to two punched steel plates 2 respectively, the two punched steel plates 2 are parallel to the corresponding two horizontal edges 9, the two ends of the punched steel plate 2 are connected with the two ends of the corresponding horizontal edge 9 through the cable-stayed spring 3, the cable-stayed spring 3 is in a stretched state, the distance between the two punched steel plates 2 is greater than the vertical edge 10 of the I-shaped piece 5, and the punched steel plate 2 is located outward and above the horizontal edge 9. The angle of the cable-stayed spring 3 is 45 degrees, and the IMU is integrated on the collection plate. The I-shaped piece 5 and the punched steel plate 2 are connected through four cable-stayed springs 3.

[0065] The use process of the device is as follows:

[0066] The damping camera-IMU device is installed on the agricultural machine. The steel pipe of the T-shaped structural member 1 is connected with the steel pipe on the agricultural machine, and the connection is stable.

[0067] The camera-IMU structural member is the core component of the device and is located at the bottom of the device. The structural member comprises an upper plate and surrounding side plates. At least one camera is mounted on the front side plate of the side plate. According to the schematic diagram in the drawing, the cameras can be mounted at both ends of the front side plate and the inclined side plate, and a total of four cameras are arranged. The cameras can be selected from wide-angle lenses, the included angle between the front side plate and the inclined side plate is 135 degrees, and the total field of view reaches 210° when the four wide-angle lenses are selected. The cameras are used for collecting visual data.

[0068] The IMU is located at the middle part of the lower surface of the upper plate of the camera-IMU structural member. The IMU is used for collecting acceleration and angular velocity data.

[0069] I-beam 5 is located on the upper part of the camera-IMU structure and is fixed on the upper plate of the camera-IMU structure by screw connection. I-beam 5 has the effect of strengthening the strength of the camera-IMU structure and providing mounting hole positions for the spring. In addition, I-beam 5 also increases the self-weight of the camera-IMU structure, enhances the tightness of the spring to achieve the effect of enhancing the overall stability.

[0070] Punched steel plate 2 is located obliquely above I-beam 5 and is fixed on the lower surface of T-shaped structure 1. T-shaped structure 1 is provided with a through hole for the steel pipe on the agricultural machine to pass through. The lower surface of T-shaped structure 1 is a flat surface provided with screw holes, which can be fixed with punched steel plate 2 through screws, providing a stable mounting platform for punched steel plate 2.

[0071] Sponge 4 is placed above I-beam 5, which is used for shock absorption and buffering of high-frequency vibration. Sponge 4 is located between I-beam 5 and the steel pipe on the agricultural machine.

[0072] Cable-stayed spring 3 is located between I-beam 5 and punched steel plate 2 through the hole positions connecting them. Cable-stayed spring 3 is in a stretched state and has an angle of 45 degrees obliquely inward. Its role is to provide stability and shock absorption effect. Two punched steel plates 2 are located obliquely above the two horizontal edges 9 of I-beam 5 and are connected to cable-stayed spring 3.

[0073] Front side plate: The front side plate is part of the side plate of the camera-IMU structure. It faces directly forward and is flat. Up to two cameras can be installed on the front side plate, with a minimum of one camera. The camera faces forward and is used to capture the field of view in front.

[0074] Oblique side plate: The oblique side plate is part of the side plate of the camera-IMU structure and is located on both sides of the front side plate. The oblique side plate forms an angle with the front side plate to form the side plate of the camera-IMU structure together. Up to two cameras can be installed on the oblique side plate, with a minimum of zero cameras. The camera faces obliquely forward and has a certain field of view overlap with the camera on the front side plate, and can expand the field of view.

[0075] Horizontal edge 9: Horizontal edge 9 is the leftmost and rightmost of the three edges of I-beam 5. I-beam 5 has two horizontal edges 9, corresponding to two punched steel plates 2 respectively. The punched steel plate 2 is parallel to the horizontal edge 9 and is connected to the I-beam 5 through the cable-stayed spring 3. The role of the horizontal edge 9 is to provide a structural basis for connecting the punched steel plate 2, and also to provide the effect of strengthening the strength of the camera-IMU structure and the structural basis for connection.

[0076] Vertical edge 10: Vertical edge 10 is the middle one of the three edges of I-beam 5. I-beam 5 has one vertical edge 10 connecting the two horizontal edges 9. The role of the vertical edge 10 is to strengthen the structural strength of the I-beam 5 and provide the function of fixation and support.

[0077] The whole device operates by hardware trigger synchronization mode, ensuring that the trigger delay of the four cameras is about 1 ms, and the IMU and the camera use the same clock to improve the consistency of visual data.

[0078] The damping structure of the device plays a damping role. The spring and the sponge 4 work together. The spring provides tension, which counteracts the sponge 4. Since the vibration generated by the agricultural diesel engine is mainly up and down vibration, this design can provide the best damping effect at the position where the longitudinal vibration is the strongest. At the same time, the arrangement of the inclined spring 3 ensures the stability of the camera-IMU structure in the front and rear and left and right dimensions.

[0079] Through the arrangement of the four wide-angle lenses, the camera system of the device provides a wide field of view. The two front-facing cameras face the same direction, with a baseline distance of 20 cm, achieving binocular stereo matching. The two left and right facing cameras have a certain field of overlap with the front facing cameras, ensuring a wide field of view. The total field of view reaches 210 degrees. The IMU is integrated on the acquisition board, further simplifying the installation difficulty and enhancing the stability.

[0080] Through the above steps, the damping camera-IMU device in the agricultural transplanting system can be successfully installed and operated, achieving accurate IMU data acquisition and multi-view visual information acquisition, meeting the needs of automated agriculture.

[0081] The device is used as follows:

[0082] Four wide-angle lenses are selected, and the self-designed multi-camera and IMU arrangement and installation structure: the front two cameras are used as binoculars for stereo matching, facing the same direction, with a 20 cm baseline length, and the left and right sides have overlapping fields of view with the front view, about 75°, which can ensure the accuracy of calibration, has no middle blind area, and the key front view area can be captured by multiple cameras to increase robustness, and can maximize the use of the field of view advantage of the multi-view wide-angle lens, with a total field of view of 210°. The IMU is installed in the middle of the four cameras and on the same board to ensure measurement accuracy. At the center of the four cameras, which is also the center of the entire installation structure, the relative vibration is the smallest, which can be understood as the mean value of the vibration of the four cameras. The left and right vibrations are up and down, and the position change is 0 for it.

[0083] The camera is hardware trigger synchronized: it can ensure that the trigger delay of the four cameras is about 1 ms, improving the accuracy of the front binocular matching and the consistency and reliability of the visual-IMU positioning and navigation algorithm during the operation of the entire system.

[0084] The damping structure is used to fix the above structure on the agricultural machine, while reducing high-frequency vibration to enhance the effectiveness of the IMU.

[0085] Autonomous design of suspension damping scheme, spring and sponge 4: reduce the high frequency noise interference to IMU. The design starting point of this is to find out that if the camera-IMU structure is directly hard connected on the steel pipe in front of the rice transplanter, due to the violent shaking brought by the diesel engine at start-up, and the subsequent running process still exists the shaking of diesel engine and the shaking of hard wheels when running, the visual-IMU positioning and navigation algorithm is difficult to initialize and effectively run in the subsequent process, the reasons are as follows: the high frequency and large amplitude shaking of IMU during initialization is too large, so that the low-cost IMU cannot obtain the instantaneous speed change due to insufficient sampling rate, resulting in that the output does not match the actual. The shaking of the vehicle body and its own structure leads to the blurring of the picture collected by the camera, or the shaking leads to the distortion of the external parameter of the multi-camera-IMU system. High frequency "hard" vibration is also easy to cause damage to the equipment and reduce its service life.

[0086] In the design selection of the damping scheme, the spring suspension and sponge 4 pad are selected. The selection of suspension can avoid the increase of shaking amplitude caused by the out-of-sync shaking of the structure and the vehicle body during vibration by using its own gravity, and the use of sponge 4 pad can further eliminate high frequency noise and low frequency vibration, and also eliminate the reciprocating vibration of the spring. It is worth noting that when designing the spring suspension structure, the four springs are intentionally set at an angle of diagonal pull, so that the multi-camera-IMU structure can be maximally stabilized in the front-back, left-right directions, and the vibration in the up-down direction which is the most serious high frequency vibration can be effectively eliminated.

[0087] The application also discloses a camera-IMU navigation device in a rice transplanter system, which comprises a camera-IMU acquisition board, an IMU fixed on the camera-IMU acquisition board, a camera connected with the camera-IMU acquisition board, and an industrial computer connected with the camera-IMU acquisition board.

[0088] The camera-IMU acquisition board is responsible for controlling the sampling trigger of the camera and the timestamp synchronization between the multiple cameras, ensuring the accurate acquisition of image data and the close time between different cameras.

[0089] The camera comprises two cameras for forming a front binocular combination, and two other cameras facing left front and right front, which provide a wider field of view.

[0090] The IMU obtains attitude information such as acceleration and angular velocity, and the attitude information contains the attitude change and motion state of the agricultural machine.

[0091] The industrial computer is responsible for processing the received camera and IMU inertial measurement data, and running a positioning and navigation algorithm to output the attitude estimation and positioning result of the agricultural machine.

[0092] The application also discloses an autonomous navigation method of a camera-IMU device in a rice transplanter system, comprising the following steps: obtaining image data and IMU inertial measurement data; the image data and the IMU inertial measurement data are obtained by calling a camera through a camera-IMU acquisition board; the image data comprises image data of four cameras and is used for feature point extraction, tracking and pose estimation; the IMU inertial measurement data comprises acceleration and angular velocity and the like and is used for assisting the pose estimation;

[0093] Using the image data and the inertial measurement data, the state of the Kalman filter is initialized, including the pose and velocity of the camera and the bias of the IMU, so as to obtain the pose and velocity of the camera and the bias of the IMU in the initial state; this step only needs to be performed once at the beginning of the algorithm and is used to initialize the initial pose and initial velocity of the camera and the initial bias of the IMU mentioned in step 4;

[0094] Through the image data, a feature extraction algorithm (such as ORB, SIFT and the like) is used to extract feature points in the image data, so as to obtain pixel coordinates and descriptors of the feature points;

[0095] A feature point matching algorithm is run on the descriptors, the feature points are tracked and matched between continuous image frames or images of different cameras at the same time, and a corresponding relationship is established, so as to obtain the matched feature points;

[0096] Through the matched feature points and the inertial measurement data, a Kalman filter is used for visual and inertial state estimation, observation update and prediction update are used to update the pose and velocity of the camera and the bias of the IMU, so as to obtain the updated pose and velocity of the camera and the bias of the IMU; the observation update and the prediction update are alternately performed to realize continuous state estimation and update. The observation update corrects the estimation error by using visual measurement, and the prediction update predicts the motion of the system by using inertial measurement. In this way, by continuously iterating the observation update and the prediction update, the Kalman filter gradually converges to more accurate pose and velocity estimation of the camera and bias estimation of the IMU. The results of the updated pose and velocity of the camera and the bias of the IMU are output.

[0097] As a further improvement, the observation update step of the application is:

[0098] Through the matched feature points, the state of the Kalman filter is updated by using visual measurement, the change of the pixel coordinates of the feature points in two frames and the inertial measurement, the acceleration and angular velocity of the IMU;

[0099] By using the visual measurement, the observation of the feature points is compared with the corresponding feature points in the Kalman filter, the reprojection error is calculated, and the observation error is obtained;

[0100] According to the observation error, a Kalman gain is calculated, which is used to correct the state estimation of the Kalman filter, thereby obtaining updated pose, velocity and bias estimates;

[0101] Prediction update step:

[0102] By means of the inertial measurement data, the changes in the pose, velocity and bias of the camera and the IMU are predicted by integrating the acceleration and angular velocity of the IMU;

[0103] According to the prediction model, the prediction step of the Kalman filter is used to update the prediction estimate of the state.

[0104] Embodiment

[0105] Step 1: Data acquisition and transmission

[0106] 1. Obtain camera data: Through the acquisition board, image data is obtained from up to four cameras in a hardware trigger synchronization manner. These cameras include two cameras for forming a forward binocular combination, and another two cameras facing left front and right front, providing a wider field of view. The acquisition board is responsible for controlling the triggering and synchronization of the cameras, ensuring accurate acquisition of image data.

[0107] 2. Obtain IMU data: The acquisition board is also connected to the inertial measurement unit (IMU) through an interface to obtain attitude information such as acceleration and angular velocity from the IMU. These information contains the attitude changes and motion state of the agricultural machine. The IMU collects and transmits data at a frequency of up to 200 Hz, and uses the same timestamp as the camera data.

[0108] 3. Transmit data to the computing platform: The acquisition board and the computing platform (such as Intel NUC11) are connected through a USB3.0 connection line to realize data transmission. USB3.0 has the characteristics of high-speed transmission, which can meet the real-time requirements.

[0109] Step 2: Software processing and algorithm execution

[0110] 1. Obtain image data and IMU inertial measurement data: Call the camera through the camera-IMU acquisition board to take pictures to obtain image data and IMU inertial measurement data. Image data includes image data of four cameras for feature point extraction, tracking and pose estimation. IMU inertial measurement data includes attitude information such as acceleration and angular velocity for auxiliary attitude estimation.

[0111] 2. Initialize the state of the Kalman filter: Use image data and inertial measurement data to initialize the Kalman filter, including the pose and velocity of the camera and the bias of the IMU. This step is only executed once at the beginning of the algorithm to initialize the initial pose and velocity of the camera and the initial bias of the IMU.

[0112] 3. Feature point extraction and matching: Through image data, feature extraction algorithms (such as ORB, SIFT, etc.) are used to extract feature points in the image, and the pixel coordinates and descriptors of the feature points are obtained. Then run the feature point matching algorithm to track and match the feature points between consecutive image frames or images of different cameras at the same time, establish their corresponding relationship, and obtain the matched feature points.

[0113] 4. Visual and inertial state estimation: Using the matched feature points and inertial measurement data, a Kalman filter is used for visual and inertial state estimation. The filter alternates between continuous state estimation and update through observation update and prediction update. The observation update uses visual measurement (change of pixel coordinates of feature points) and inertial measurement (acceleration and angular velocity of IMU) to correct the estimation error, calculate the Kalman gain, and use it to correct the state estimation value of the filter, thereby obtaining the updated camera pose and velocity and IMU bias. The prediction update predicts the change of camera and IMU pose, velocity and bias through inertial measurement data, integrates the acceleration and angular velocity of IMU, and uses the prediction step of the filter to update the predicted estimation of the state.

[0114] 5. Output the updated results: Output the updated camera pose and velocity and IMU bias results for subsequent autonomous navigation of the agricultural machine transplanting system.

[0115] The above is only the preferred embodiment of the present application, it should be noted that for those skilled in the art, without departing from the core technical features of the present application, a number of improvements and refinements can also be made, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A vibration-damping camera-IMU device in a rice transplanting system, characterized in that, The device includes a cavity-containing cover-type camera IMU structure, a quad-lens wide-angle camera positioned in front of the front side plate of the camera IMU structure, an IMU invertedly mounted on the center of the lower surface of the upper plate of the camera IMU structure, an I-beam connected and fixed to the upper plate of the camera IMU structure, two perforated steel plates located above the I-beam and parallel to its horizontal edge, and a T-shaped structure fixed to the upper surface of the perforated steel plates; the T-shaped structure has through holes adapted to fit agricultural machinery steel pipes for use on agricultural machinery. The steel pipe is inserted; a sponge is placed on top of the I-beam; the two ends of the two horizontal sides of the I-beam are connected to the two ends of the corresponding perforated steel plates by inclined tension springs at a 45-degree angle. The inclined tension springs are in a stretched state, the distance between the two perforated steel plates is greater than the length of the vertical side of the I-beam, and the perforated steel plates are located on the outer upper part of the horizontal side of the I-beam; the I-beam is used to enhance the strength of the camera IMU structural components, provide mounting holes for the inclined tension springs, and increase the tension of the inclined tension springs by increasing the self-weight of the camera IMU structural component assembly.

2. The vibration-damping camera-IMU device in the rice transplanting system according to claim 1, characterized in that, The camera IMU structure includes a top plate and four side plates, with at least two cameras mounted on the front side plate of the side plates.

3. The vibration-damping camera-IMU device in the rice transplanting system according to claim 2, characterized in that, The front panel includes a front panel and inclined plates on both sides of the front panel. Cameras are installed at both ends of the front panel and on the inclined plates on both sides, for a total of 4 cameras. The cameras are quad-lenses with wide-angle lenses. The angle between the front panel and the inclined plates is 135 degrees.

4. A camera-IMU navigation device in a rice transplanting system, characterized in that, Includes the vibration-damping camera-IMU device as described in any one of claims 1-3, a camera-IMU acquisition board connected to the camera IMU structural component in the vibration-damping camera-IMU device, and an industrial control computer connected to the camera-IMU acquisition board; The camera-IMU acquisition board is responsible for controlling the sampling triggering of the camera and the timestamp synchronization between multiple cameras. Ensure accurate acquisition of image data and that the data from different cameras is as similar as possible; The camera includes two cameras for forming a forward-facing binocular combination, and two additional cameras facing the left front and right front, providing a wider field of view; The IMU acquires attitude information including acceleration and angular velocity, and the attitude information includes the attitude changes and motion state of the agricultural machinery; The industrial control computer is responsible for processing the received camera and IMU inertial measurement data, running the positioning and navigation algorithm, and outputting the attitude estimation and positioning results of the agricultural machinery.

5. A navigation method for a camera-IMU navigation device in a rice transplanting system according to claim 4, characterized in that, include: Image data and IMU inertial measurement data are acquired; the image data and IMU inertial measurement data are acquired by calling the camera through the camera-IMU acquisition board; the image data includes image data from four cameras, which are used for feature point extraction, tracking and attitude estimation; the IMU inertial measurement data includes attitude information containing acceleration and angular velocity, which is used to assist in attitude estimation. Using image data and inertial measurement data, initialize the state of the Kalman filter, including the camera pose and velocity and the IMU bias, to obtain the camera pose and velocity and the IMU bias in the initial state. By using image data, feature extraction algorithms are used to extract feature points from the image data, and the pixel coordinates and descriptors of the feature points are obtained. The descriptor is run with a feature point matching algorithm to track and match feature points between consecutive image frames or images from different cameras at the same time, establishing their correspondence and obtaining the matched feature points. Using the matched feature points and inertial measurement data, a Kalman filter is used to estimate the visual and inertial states. The camera pose and velocity, as well as the IMU bias, are updated using observation updates and prediction updates, resulting in the updated camera pose and velocity, as well as the IMU bias. The updated camera pose and velocity, as well as the IMU bias, will be output.

6. The navigation method of the camera-IMU navigation device in the rice transplanting system according to claim 5, Its features are, The observation update steps are as follows: The state of the Kalman filter is updated using visual and inertial measurements based on the matched feature points. By using visual measurement, the observed feature points are compared with the corresponding feature points in the Kalman filter, and the reprojection error is calculated to obtain the observation error; Based on the observation error, the Kalman gain is calculated and used to correct the state estimate of the Kalman filter, thereby obtaining the updated pose, velocity and bias estimates. The prediction update steps are as follows: Using inertial measurement data, the changes in pose, velocity, and bias of the camera and IMU are predicted, and the IMU's acceleration and angular velocity are integrated. Based on the prediction model, the prediction step of the Kalman filter is used to update the predicted estimate of the state.

Images