A robot autonomous ultra-close panoramic detection and positioning system and method for crop diseases

Abstract

The present invention discloses a robot autonomous ultra-close panoramic detection and positioning system and method for crop diseases, and relates to the field of agricultural robots. The robot autonomous ultra-close panoramic detection and positioning system includes an ultra-close canopy adaptive detection device and a canopy profiling panoramic capture device. Through the ultra-close canopy adaptive adjustment and canopy morphology matching profiling, the robot can autonomously adapt to and adjust the canopy with large differences in morphology. Through the canopy panoramic capture and splicing method, the robot can fully acquire the ultra-close three-dimensional image information of the canopy. Then, through the disease multi-layer feature model training and the disease pre-inspection and re-inspection step-by-step diagnosis method, the robot can autonomously focus on and accurately diagnose the diseased part. The present invention solves the problems of missed detection and false detection of diseases when the robot is operating, and improves the robot's detection accuracy of diseases under complex canopies.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural robots, and in particular to a robot autonomous ultra-close panoramic detection and positioning system and method for crop diseases. Background Art

[0002] In recent years, China has become a country with a high incidence of crop diseases, and crop disease detection has become an important issue in my country's agricultural production. At present, disease inspection robots mainly rely on large-scale field scene row-by-row full-crown shooting and post-processing of massive data. In addition, the robot is equipped with a single-angle visual system for shooting, which is affected by factors such as the distance, angle, and occlusion between the crop and the camera and cannot reliably obtain the early disease spot characteristics of different parts of the plant. At the same time, the disease detection method that only relies on RGB images is easily affected by field lighting changes, leaf distortion, and the angle and posture of the disease spot, and is suitable for detection under a certain degree of disease outbreak. At present, there is an urgent need for an efficient and accurate disease detection system and method to improve the disease prevention ability of agricultural production and realize the refined and intelligent management and control of crop growth links. Summary of the invention

[0003] In view of this, the present invention provides a robot autonomous ultra-close panoramic detection and positioning system and method for crop diseases.

[0004] The present invention achieves the above technical objectives through the following technical means.

[0005] A robot autonomous ultra-close panoramic detection and positioning method for crop diseases, comprising:

[0006] The RGB image taken by the depth camera at the center of the flexible deformable rack is used as the base image, and the RGB image taken by the depth camera at both ends of the flexible deformable rack is used as the auxiliary image. Similar feature points are detected between the auxiliary image and the base image to achieve the stitching of RGB images from different viewpoints and establish the canopy panoramic RGB image A. According to the coordinate alignment relationship between the color-depth and infrared thermal cameras, the canopy panoramic 3D depth point cloud dataset B and the panoramic infrared thermal image C are stitched together.

[0007] According to the order of RGB color features, RGB morphological features and thermal image temperature difference features corresponding to the disease types, they are successively implanted into the deep learning network for successive training of multi-level features to obtain feature maps, and the features are weighted and aggregated to obtain a canopy multi-disease multi-feature solidified comparison model.

[0008] The panoramic RGB image A is input into the canopy multi-disease multi-feature solidification comparison model, and the RGB color and RGB morphological feature maps are compared online in sequence to determine the disease area with the highest comprehensive similarity between the panoramic RGB image and the color-morphological features;

[0009] The panoramic infrared thermal image C is input into the canopy multi-disease multi-feature solidification comparison model to perform online comparison of the thermal temperature difference characteristics between the diseased and healthy areas. If the panoramic infrared thermal image C is detected to have disease temperature characteristics, the initial inspection of the canopy indicates that there is a disease and the suspicious area C is located. 1 , otherwise there is no disease;

[0010] According to the initial examination of suspicious areas C 1 The control system performs coordinate matrix conversion and alignment of the infrared thermal camera and the depth camera to locate the suspicious area B in the canopy panoramic 3D depth point cloud dataset B. 1 , the control system drives the servo motor to the suspicious area B 1 The depth camera and the infrared thermal camera automatically focus on the suspicious area and obtain the RGB image A of the suspicious area respectively. 2 and infrared thermal image C 2 ; The control system then converts the A 2 and C 2 Input into the canopy multi-disease multi-feature solidification comparison model to complete the re-inspection of the suspicious area to determine the type of disease;

[0011] The RGB color features and RGB morphological features are acquired from a panoramic RGB image, and the thermal image temperature difference features are acquired from a panoramic infrared thermal image.

[0012] Further technical solutions include:

[0013] The coordinate difference H along the Z direction between the center of the depth camera's field of view and the canopy point cloud coordinates directly below 4 Online comparison is performed with the preset height △H between the depth camera and the canopy surface, and the depth camera height is adjusted to adapt to the canopy super close view;

[0014] The flexible deformable rack matches the plant canopy morphology.

[0015] A further technical solution is that the specific process of adjusting the height of the depth camera is as follows:

[0016] From the coordinate difference H 4 Compare online with the preset height △H to determine the number of revolutions N and direction of the servo lift motor, and:

[0017]

[0018] Where C is the distance that the servo lift motor moves in one circle;

[0019] When △HH 4 >0, the servo lift motor rotates forward to drive the depth camera upward, so that △H=H 4 ; When △HH 4<0, the servo lift motor reverses and drives the depth camera downward, so that △H=H 4 .

[0020] A further technical solution is that the flexible deformable rack matches the plant canopy morphology by: before the robot works, the flexible deformable rack is bent into a shape corresponding to the plant canopy.

[0021] In a further technical solution, the disease type is determined according to the following method:

[0022] The color and morphological features of the panoramic RGB images are used to establish a dual-semantic contextual feature label set of primary and secondary features. The temperature difference between the diseased and healthy areas of the panoramic infrared thermal images is used to establish a single-semantic contextual feature label set of tertiary features. The RGB color-morphology and infrared thermal image temperature difference are marked one by one, and a context-fused matching training set of multi-category crop canopy disease features is constructed to determine the disease type.

[0023] In a further technical solution, when the base image and the auxiliary image are acquired, the depth camera is set at a horizontal viewing angle [-α, α] and a vertical viewing angle [-β 3 ,β 3 ] to collect information, the infrared thermal camera 4 is set at a horizontal viewing angle [-α, α] and a vertical viewing angle [-β 1 ,β 2 ] to collect information, and:

[0024]

[0025]

[0026]

[0027]

[0028] Where: W 1 is the statistical canopy lateral extension, W 2 is the statistical lateral and longitudinal extension of the canopy, S is the horizontal spacing between the depth camera and the infrared thermal camera, and △H is the preset height.

[0029] A robot autonomous ultra-close panoramic detection and positioning system for crop diseases, comprising:

[0030] The canopy ultra-close-range adaptive detection device includes a depth camera and a servo lifting rod. Based on the preset height between the depth camera and the canopy surface, the servo lifting rod drives the depth camera and the infrared thermal camera to move;

[0031] A canopy profiling panoramic capture device includes a servo motor, a gear and a flexible deformable rack. The servo motor is connected to the gear, and the gear is meshed with the flexible deformable rack. The canopy ultra-close-range adaptive detection device and the servo motor are installed as a whole through a bracket to achieve the movement of the canopy ultra-close-range adaptive detection device between the center position and the two end positions of the flexible deformable rack.

[0032] In the above technical solution, the flexible deformable rack is located in the gap between the bearing and the gear, and the gear is fixed to the inner side H below the transmission bracket. 3 The bearing is fixed on the inner side H below the transmission bracket. 1 At, and H 1 >H 3 .

[0033] In the above technical solution, a proximity switch II is installed at the center position of the flexible deformable rack, and a proximity switch I and a proximity switch III are installed at both ends of the flexible deformable rack respectively.

[0034] In the above technical solution, both ends of the flexible deformable rack can be detachably mounted on the robot bracket.

[0035] The beneficial effects of the present invention are as follows: in view of the key problems of the single viewing angle and incomplete image information of existing disease detection, the present invention solves the problem of the robot's autonomous adaptation and adjustment to the large difference in canopy morphology through ultra-close-up adaptive adjustment of the canopy and canopy morphology matching and profiling; through the canopy panoramic capture and splicing method, the robot can fully acquire the ultra-close-up three-dimensional image information of the canopy; and then through the disease multi-layer feature model training and the disease pre-inspection-re-inspection step-by-step diagnosis method, the robot can autonomously focus on and accurately diagnose the diseased part; the present invention solves the problems of missed detection and false detection of diseases when the robot is operating, and improves the robot's detection accuracy of diseases under complex canopies. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a partial structural diagram of the robot autonomous ultra-close panoramic detection and positioning system for crop diseases of the present invention;

[0037] Figure 2 This is a schematic diagram of the arc-shaped canopy profiling of the present invention;

[0038] Figure 3 This is a schematic diagram of the straight canopy profiling of the present invention;

[0039] Figure 4 This is a schematic diagram of the inverted V-shaped canopy profiling of the present invention;

[0040] Figure 5 This is a schematic diagram of the robot obtaining canopy information according to the present invention;

[0041] Figure 6This is a schematic diagram of horizontal viewing angle compression according to the present invention;

[0042] Figure 7 This is a schematic diagram of the vertical viewing angle compression of the present invention;

[0043] In the figure, 1. servo lifting rod, 2. dual camera fixing plate, 3. depth camera, 4. infrared thermal camera, 5. transmission bracket, 6. bearing, 7. servo motor, 8. support wheel, 9. gear, 10. flexible deformable rack, 11. proximity switch I, 12. proximity switch II, 13. proximity switch III, 14. robot, 15. canopy. DETAILED DESCRIPTION

[0044] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0045] like Figure 1 As shown, the robot autonomous ultra-close panoramic detection and positioning system for crop diseases of the present invention includes a canopy ultra-close adaptive detection device and a canopy profiling panoramic capture device.

[0046] The canopy ultra-close-range adaptive detection device includes a servo lifting rod 1, a dual-camera fixing plate 2, a depth camera 3, an infrared thermal camera 4 and a transmission bracket 5; the servo lifting rod 1 is fixed to the transmission bracket 5 on one side along the forward direction of the robot, the dual-camera fixing plate 2 is installed at the lower end of the servo lifting rod 1, and the depth camera 3 and the infrared thermal camera 4 are installed in parallel on both sides of the dual-camera fixing plate 2 with a horizontal interval S. In this embodiment, S=5cm.

[0047] The canopy profiling panoramic capture device includes a bearing 6, a servo motor 7, a support wheel 8, a gear 9, a flexible deformable rack 10, a proximity switch I 11, a proximity switch II 12 and a proximity switch III 13. The bearing 6 is fixed to the inner side H below the transmission bracket 5. 1 The support wheel 8 is fixed to the inner side H below the transmission bracket 5. 2 The gear 9 is fixed to the inner side H below the transmission bracket 5. 3 At, and H 1 >H 2 >H 3 ;like Figure 2 As shown, the servo motor 7 is installed on the side of the transmission bracket 5 away from the forward direction of the robot, and the output shaft of the servo motor 7 is fixedly connected to the gear 9; the flexible deformable rack 10 is installed in the gap between the bearing 6 and the gear 9, and the flexible deformable rack 10 is meshed with the gear 9, and the proximity switch I 11 is installed on one side of the flexible deformable rack 10 1 The proximity switch III 13 is installed on the other side S of the flexible deformable rack 10. 1 The proximity switch II 12 is installed at the center of the flexible deformable rack 10, and the transmission bracket 5 is located at the proximity switch II 12; in this embodiment, H1 =4cm, H 2 =1.5cm, H 3 =1cm, S 1 =5.5cm. Both ends of the flexible deformable rack 10 can be detachably mounted on the robot bracket.

[0048] Among them, the servo lifting rod 1 has a motor integrated therein, and the motor communicates with the control system, and the depth camera 3, the infrared thermal camera 4, the servo motor 7, the proximity switch I 11, the proximity switch II 12 and the proximity switch III 13 all communicate with the control system.

[0049] The robot autonomous ultra-close panoramic detection and positioning method for crop diseases of the present invention includes an ultra-close canopy adaptive adjustment method, a canopy morphology matching and profiling method, a canopy panoramic capture and splicing method, and a disease pre-inspection-re-inspection step-by-step diagnosis method, which are specifically described as follows:

[0050] The canopy ultra-close-up adaptive adjustment method, the robot 14 moves to the top of the canopy 15, the control system controls the depth camera 3 to work, scans the canopy 15 information, and obtains a three-dimensional depth point cloud data set; the center B of the depth camera 3 field of view is calculated based on the three-dimensional depth point cloud data set 01 (0,0,D 0 (0,0)) and the canopy 15 point cloud coordinates B 02 (α,β,D 0 The coordinate difference H of (α,β)) along the Z direction 4 , the control system will H 4 The preset height △H between the depth camera and the canopy surface is compared online to determine the number of revolutions N and direction of the servo lifter 1 motor; where N is:

[0051]

[0052] In formula (1), C is the distance that the servo lifter 1 motor moves in one circle;

[0053] When △HH 4 >0: The servo lift rod 1 motor rotates forward to drive the depth camera 3 upward, so that △H=H 4 ; When △HH 4 <0: The motor of the servo lift rod 1 reverses and drives the depth camera 3 to move downward, so that △H=H 4 ; In this embodiment, △H=15cm.

[0054] Canopy morphology matching profiling methods, such as Figure 2 , 3As shown in 4, according to the plant canopy morphology, before the robot works, the flexible deformable rack 10 is manually bent into a corresponding shape of the plant canopy. Common plant canopy shapes include arc, straight shape, and inverted V shape.

[0055] like Figure 5 , 6 , 7, the canopy panoramic capture and stitching method, the specific steps are:

[0056] Step 1: The control system drives the servo motor 7 to move toward the direction of the proximity switch Ⅰ11, and the inner wall of the transmission bracket 5 triggers the proximity switch Ⅰ11, thereby controlling the depth camera 3 and the infrared thermal camera 4 to scan the canopy 15 information at the first viewpoint;

[0057] Step 2: During the field of view scanning, the depth camera 3 uses the horizontal viewing angle [-α, α] and vertical viewing angle [-β 3 ,β 3 ] to capture color-depth image composite information; the infrared thermal camera 4 uses the horizontal viewing angle [-α, α] and vertical viewing angle [-β 1 ,β 2 ] To capture infrared thermal image information;

[0058] Among them, the horizontal viewing angle α and the vertical viewing angle β of the depth camera 3 are 3 for:

[0059]

[0060] The horizontal viewing angle α of the infrared thermal camera 4 and the depth camera 3 are the same, and the vertical viewing angle β 1 and β 2 for:

[0061]

[0062] In formula (2) and (3), W 1 is the lateral width of the canopy 15, W 2 is the horizontal and vertical spread of the canopy 15 to be counted, S is the horizontal spacing between the depth camera 3 and the infrared thermal camera 4;

[0063] Step 3: After completing the canopy 15 information scanning operation at the first viewpoint, the servo motor 7 moves toward the proximity switch II 12 and the proximity switch III 13 in sequence, and controls the depth camera 3 and the infrared thermal camera 4 to complete the canopy 15 information scanning operation at the second and third viewpoints at the same field of view scanning angle in step 2;

[0064] Step 4: The control system uses the RGB image taken by the second viewpoint depth camera 3 as the base image, and the first viewpoint and third viewpoint RGB images as auxiliary images. The first viewpoint and third viewpoint RGB images are used to detect similar feature points with the second viewpoint RGB image, and the RGB images of different viewpoints are quickly spliced ​​to establish a panoramic RGB image A (α, β 3 ); According to the color-depth and infrared thermal camera 4 coordinate alignment relationship, the canopy 15 panoramic three-dimensional depth point cloud dataset B (α, β 3 ,D(α,β 3 )) and panoramic infrared thermal image C(α,β,T(α,β))(β represents β 1 and β 2 ). In this embodiment, S=5cm, W 1 =60cm, W 2 =25cm.

[0065] The disease pre-inspection-re-inspection step-by-step diagnosis method includes a disease multi-layer feature model training sub-method and a disease pre-inspection-re-inspection fusion diagnosis sub-method.

[0066] The specific process of the disease multi-layer feature model training sub-method is as follows:

[0067] Step 1: Use the color and morphological features of the canopy 15 panoramic RGB image to establish a dual semantic context feature label set of primary and secondary features, and use the temperature difference between the diseased and healthy areas of the panoramic infrared thermal image to establish a single semantic context feature label set of tertiary features. Use Labeling software to label the RGB color-morphology and infrared thermal image temperature difference one by one, and construct a context fusion matching crop canopy multi-class disease feature training set to determine the disease type, as shown in Table 1:

[0068] Table 1

[0069]

[0070] Step 2: According to the order of RGB color, RGB morphology and thermal image temperature difference features corresponding to the disease types, the images are implanted into the deep learning network of Mask R_CNN semantic segmentation with ResNet-50 as the backbone for successive training of multi-level features. The feature maps (including color, morphology and thermal image temperature difference) obtained from the successive training are weighted and aggregated to obtain a canopy multi-disease multi-feature solidified comparison model, which includes the color and morphological features of RGB images and the temperature difference features of infrared thermal images.

[0071] The specific process of the disease pre-inspection-re-inspection fusion diagnosis sub-method is as follows:

[0072] Step 1: The control system inputs the canopy 15 panoramic RGB image A into the canopy multi-disease multi-feature solidification comparison model, and sequentially performs online comparison of the RGB color feature map and the RGB morphological feature map to determine the disease category with the highest comprehensive similarity between the panoramic RGB image A and the color-morphological feature;

[0073] Step 2: The control system inputs the panoramic infrared thermal image C into the canopy multi-disease multi-feature solidification comparison model to perform online comparison of the thermal temperature difference characteristics between the diseased and healthy areas. If the image C is detected to have a disease temperature feature, the canopy 15 is initially inspected to have a disease and the suspicious area C is located. 1 , otherwise there is no disease;

[0074] Step 3: Based on the suspicious area C initially inspected in step 2 1 The control system performs coordinate matrix conversion and alignment of the infrared thermal camera 4 and the depth camera 3 to locate the suspicious area B in the three-dimensional depth point cloud data set. 1 ; According to the camera-world coordinate system transformation of the field of view center of the depth camera 3 and the suspicious area B1, the positional relationship between the suspicious area and the depth camera 3 is calculated, and the control system drives the servo motor 7 to move directly above the suspicious area B1;

[0075] Step 4: The depth camera 3 and the infrared thermal camera 4 automatically focus on the suspicious area to obtain RGB images A and B of the suspicious area. 2 (α,β 3 ) and infrared thermal image C 2 (α,β,T 2 (α,β))(β represents β 1 and β 2 );The control system then converts the RGB image A 2 and infrared thermal image C 2 Input into the canopy multi-disease multi-feature solidification comparison model, repeat steps one and two to complete the re-inspection of the suspicious area to determine the type of disease.

[0076] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.

Claims

1. A robot autonomous ultra-close panoramic detection and positioning method for crop diseases, characterized by: The RGB image taken by the depth camera at the center of the flexible deformable rack is used as the base image, and the RGB image taken by the depth camera at both ends of the flexible deformable rack is used as the auxiliary image. Similar feature points are detected between the auxiliary image and the base image to achieve the splicing of RGB images from different viewpoints and establish the canopy panoramic RGB image A; According to the coordinate alignment relationship between color-depth and infrared thermal cameras, the canopy panoramic 3D depth point cloud dataset B and the panoramic infrared thermal image C are stitched together; According to the order of RGB color features, RGB morphological features and thermal image temperature difference features corresponding to the disease types, they are successively implanted into the deep learning network for successive training of multi-level features to obtain feature maps, and the features are weighted and aggregated to obtain a canopy multi-disease multi-feature solidified comparison model. The panoramic RGB image A is input into the canopy multi-disease multi-feature solidification comparison model, and the RGB color and RGB morphological feature maps are compared online in sequence to determine the disease area with the highest comprehensive similarity between the panoramic RGB image A and the color-morphological features; The panoramic infrared thermal image C is input into the canopy multi-disease multi-feature solidification comparison model to perform online comparison of the thermal temperature difference characteristics between the diseased and healthy areas. If the panoramic infrared thermal image C is detected to have the disease temperature characteristics, the initial inspection of the canopy will show the disease and locate the suspicious area C1. Otherwise, there is no disease. According to the suspicious area C1 of the initial inspection, the control system performs coordinate matrix conversion and alignment of the infrared thermal camera and the depth camera, locates the suspicious area B1 in the canopy panoramic three-dimensional depth point cloud data set B, and the control system drives the servo motor to move directly above the suspicious area B1. The depth camera and the infrared thermal camera automatically focus and shoot the suspicious area to obtain the RGB image A2 and the infrared thermal image C2 of the suspicious area respectively; the control system then inputs the A2 and C2 into the canopy multi-disease multi-feature solidification comparison model to complete the re-inspection of the suspicious area to determine the type of disease; The RGB color features and RGB morphological features are acquired from a panoramic RGB image, and the thermal image temperature difference features are acquired from a panoramic infrared thermal image.

2. The robot autonomous ultra-close panoramic detection and positioning method according to claim 1 is characterized in that: Also includes: Compare the coordinate difference H4 between the center of the depth camera's field of view and the canopy point cloud coordinates directly below along the Z direction with the preset height △H between the depth camera and the canopy surface online, adjust the depth camera height, and adapt to the canopy super close view; The flexible deformable rack matches the plant canopy morphology.

3. The robot autonomous ultra-close panoramic detection and positioning method according to claim 2, characterized in that: The specific process of adjusting the height of the depth camera is as follows: The coordinate difference H4 is compared with the preset height △H online to determine the number of revolutions N and direction of the servo lift motor, and: Where C is the distance that the servo lift motor moves in one circle; When △H-H4>0, the servo lift motor rotates forward to drive the depth camera upward, so that △H=H4; when △H-H4<0, the servo lift motor rotates reversely to drive the depth camera downward, so that △H=H4.

4. The robot autonomous ultra-close panoramic detection and positioning method according to claim 2, characterized in that: The flexible deformable tooth bar matches and imitates the plant canopy morphology in the following specific steps: before the robot works, the flexible deformable tooth bar is bent into a shape corresponding to the plant canopy.

5. The robot autonomous ultra-close panoramic detection and positioning method according to claim 1, characterized in that: The disease type is determined as follows: The color and morphological features of the panoramic RGB images are used to establish a dual-semantic contextual feature label set of primary and secondary features. The temperature difference between the diseased and healthy areas of the panoramic infrared thermal images is used to establish a single-semantic contextual feature label set of tertiary features. The RGB color-morphology and infrared thermal image temperature difference are marked one by one, and a context-fused matching training set of multi-category crop canopy disease features is constructed to determine the disease type.

6. The robot autonomous ultra-close panoramic detection and positioning method according to claim 1, characterized in that: When the base image and the auxiliary image are acquired, the depth camera collects information at the set horizontal viewing angle [-α, α] and vertical viewing angle [-β3, β3], and the infrared thermal camera collects information at the set horizontal viewing angle [-α, α] and vertical viewing angle [-β1, β2], and: Where: W1 is the statistical horizontal extension of the canopy, W2 is the statistical horizontal and vertical extension of the canopy, S is the horizontal distance between the depth camera and the infrared thermal camera, and △H is the preset height.

7. A system for implementing the robot autonomous ultra-close panoramic detection and positioning method according to any one of claims 1 to 6, characterized in that: include: The canopy ultra-close-range adaptive detection device includes a depth camera and a servo lifting rod. Based on the preset height between the depth camera and the canopy surface, the servo lifting rod drives the depth camera and the infrared thermal camera to move; A canopy profiling panoramic capture device includes a servo motor, a gear and a flexible deformable rack. The servo motor is connected to the gear, and the gear is meshed with the flexible deformable rack. The canopy ultra-close-range adaptive detection device and the servo motor are installed as a whole through a bracket to achieve the movement of the canopy ultra-close-range adaptive detection device between the center position and the two end positions of the flexible deformable rack.

8. The system according to claim 7, characterized in that The flexible deformable rack is located in the middle gap between the bearing and the gear, the gear is fixed at H3 on the inner side below the transmission bracket, the bearing is fixed at H1 on the inner side below the transmission bracket, and H1>H3.

9. The system according to claim 7, characterized in that A proximity switch II is installed at the center of the flexible deformable rack, and a proximity switch I and a proximity switch III are installed at both ends of the flexible deformable rack respectively.

10. The system according to claim 7, characterized in that The two ends of the flexible deformable rack can be detachably mounted on the robot bracket.

Images