A method for measuring distance of stubble covered ground based on deep learning of multi-source images

Abstract

A method for measuring the distance of stubble-covered ground based on multi-source image deep learning relates to the technical field of agricultural machinery surface measurement. Use image data sets to train neural networks, and obtain a neural network model that can segment stubble-covered areas in plane images; obtain a depth image and a plane image of the stubble-covered surface, and align the plane image with the depth image through registration; input the plane image into the neural network model to obtain a binary image of the stubble-covered area; perform morphological operations to improve robustness; obtain a depth image with stubble coverage removed, calculate the average pixel value of the area without cover influence and convert it into distance, that is, obtain the real ground distance. By acquiring ground information from multiple sources in conjunction with neural network image segmentation and morphological processing, it is possible to obtain ground distance information without cover interference more accurately in different environments, improve the accuracy and robustness of distance measurement, and have the advantages of strong adaptability and high precision.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural machinery surface measurement, and in particular to a method for measuring the distance of stubble-covered ground based on multi-source image deep learning. Background Art

[0002] Traditional depth control mainly targets the seedbed after tillage, using mechanical limit, pressure control, laser / ultrasonic sensors to measure or control the depth of machine operation. Under the condition of surface stubble coverage, the above methods cannot accurately obtain the ground distance.

[0003] In the depth control of operations such as fixed-depth sowing, the pressure control type aims at the prepared seedbed and achieves the effect of consistent distance by maintaining a constant pressure. However, under the condition of surface cover, the pressure feedback and depth are not in a linear relationship, and the distance cannot be accurately obtained to maintain the fixed depth operation. The single-point ranging type uses ultrasonic / laser sensors. Under the condition of surface cover, the measured distance is the distance to the cover, not the distance to the ground. Therefore, the ground distance information under the cover cannot be obtained, and the depth cannot be maintained constant.

[0004] In view of the above reasons, the present invention obtains ground information over a large range through the plane image obtained by the 2D camera and the depth image obtained by the TOF camera, and obtains ground information by identifying the covered area and removing the influence of the covered area, so as to achieve the purpose of accurately obtaining the ground distance. Summary of the invention

[0005] In order to address the shortcomings of the background technology, the present invention provides a stubble-covered ground distance measurement method based on multi-source image deep learning, which obtains ground information from multiple sources and cooperates with neural network image segmentation and morphological processing to more accurately obtain ground distance information without the interference of covering materials, thereby improving the accuracy and robustness of ranging.

[0006] To achieve the above object, the present invention adopts the following technical solution: a method for measuring the distance of stubble covered ground based on multi-source image deep learning, comprising the following steps:

[0007] Step 1: using an image data set to train a neural network, wherein the image data set includes a plane image of the ground surface and a binary image corresponding to the calibrated stubble covered area, and after the neural network training, a neural network model capable of segmenting the stubble covered area in the plane image is obtained;

[0008] Step 2: Obtain the depth image and plane image of the stubble-covered surface, and align the plane image with the depth image through registration so that the pixels under the same pixel coordinates (x, y) correspond to the same physical object, and the object distance corresponding to the pixel under the depth image (x, y) is z(x, y);

[0009] Step 3: Input the plane image processed in step 2 into the neural network model trained in step 1 to obtain the binary image mask1 of the stubble covered area, which represents the stubble covered area S′ C and the area without stubble cover G′ C , the pixel values ​​m′(x,y) at different (x,y) coordinates in the binary image mask1 of the stubble covered area are:

[0010]

[0011] Step 4: Perform morphological operations on the binary image mask1 of the stubble covered area obtained in step 3 to improve robustness and obtain the binary image mask2 of the stubble covered area, which represents the stubble covered area S. C and no stubble covered area G C , the pixel values ​​m(x,y) at different (x,y) coordinates in the binary image mask2 of the stubble covered area are:

[0012]

[0013] Step 5: Let D be the set of all pixel coordinates (x, y) in the depth image, S D is the set of pixel coordinates of the stubble covered area, G D is the set of pixel coordinates of the area without stubble coverage. From step 2, we can know that:

[0014] S D =S C

[0015] According to the binary image mask2 of the stubble covered area, the stubble covered area S in the depth image is C The pixel values ​​of the corresponding points are set to 0, and the pixel values ​​of the other points remain unchanged, so as to obtain the depth image dst after removing the stubble coverage. The pixel value at (x, y) in the depth image is d(x, y). Then the pixel value dst(x, y) at (x, y) in the depth image dst after removing the stubble coverage is as follows:

[0016]

[0017] Find the average value avr of non-zero dst(x,y), and convert avr to distance z through the unit conversion factor g, where z is the actual ground distance.

[0018] Compared with the prior art, the present invention has the following beneficial effects: for ground distance measurement under complex surface conditions with stubble covering, the present invention obtains plane images and depth images of the ground through multiple sources of a 2D camera and a TOF camera, uses a neural network to learn the stubble image data, obtains a neural network model that can segment the stubble area in the plane image and segments the plane image, and uses the segmentation result to remove the covering area of ​​the depth image after morphological processing, so that the ground distance information without the interference of the covering can be obtained more accurately, and the problem that the traditional fixed-depth ranging method cannot accurately measure the distance on the stubble-covered surface is solved. The present invention can be used for obtaining information on fixed-depth control in operations such as tillage, sowing, and transplanting. Compared with the single-point ranging method, the interference of the covering can be eliminated. The multi-source acquisition of ground information combined with neural network image segmentation and morphological processing can more comprehensively acquire and process ground information, improve the accuracy and robustness of ranging, and use the neural network for stubble image segmentation with more flexibility, strong adaptability to different soils, different stubble types and different light, and has the advantages of strong adaptability and high precision. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a system block diagram of the present invention;

[0020] Figure 2 is the depth image collected in Example 1;

[0021] Figure 3 is the color plane image collected in Example 1;

[0022] Figure 4 is the color plane image aligned with the depth image in Example 1;

[0023] Figure 5 is a binary image of the stubble coverage area obtained by inference of the neural network model in Example 1;

[0024] Figure 6 Yes Figure 5 A more robust binary image is obtained after dilation operation;

[0025] Figure 7 is the depth image obtained by removing the area affected by the covering in the depth image in Example 1;

[0026] Figure 8 is the color plane image aligned with the depth image in Example 2;

[0027] Fig. 9 It is a binary image of the stubble covered area obtained after the dilation operation in Example 2. DETAILED DESCRIPTION

[0028] The technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0029] like Figure 1 As shown, a method for measuring the distance of stubble covered ground based on multi-source image deep learning includes the following steps:

[0030] Step 1: Use the image data set to train the neural network. The image data set contains a plane image of the ground surface and a binary image corresponding to the calibrated stubble covered area. After the neural network training, a neural network model capable of segmenting the stubble covered area in the plane image is obtained;

[0031] Step 2: Use the TOF camera and 2D camera to obtain the depth image and plane image of the stubble-covered surface respectively, and align the plane image with the depth image through registration so that the pixels under the same pixel coordinates (x, y) correspond to the same physical object, and the object distance corresponding to the pixel under the depth image (x, y) is z(x, y);

[0032] Step 3: Input the plane image processed in step 2 into the neural network model trained in step 1 to obtain the binary image mask1 of the stubble covered area, which represents the stubble covered area S′ C and the area without stubble cover G′ C , the pixel values ​​m′(x,y) at different (x,y) coordinates in the binary image mask1 of the stubble covered area are:

[0033]

[0034] Step 4: Perform morphological operations on the binary image mask1 of the stubble covered area obtained in step 3 to improve robustness and obtain the binary image mask2 of the stubble covered area, which represents the stubble covered area S. C and no stubble covered area G C , the pixel values ​​m(x,y) at different (x,y) coordinates in the binary image mask2 of the stubble covered area are:

[0035]

[0036] Step 5: Let D be the set of all pixel coordinates (x, y) in the depth image, S D is the set of pixel coordinates of the stubble covered area, G D is the set of pixel coordinates of the area without stubble coverage. From step 2, we can know that:

[0037] S D =S C

[0038] Therefore, the depth information of the area affected by the covering in the depth image can be removed by using the binary image mask2 of the stubble covered area in step 4, that is, the stubble covered area S in the depth image C The pixel values ​​of the corresponding points are set to 0, and the pixel values ​​of the other points remain unchanged, so as to obtain the depth image dst after removing the stubble coverage. The pixel value at (x, y) in the depth image is d(x, y). Then the pixel value dst(x, y) at (x, y) in the depth image dst after removing the stubble coverage is as follows:

[0039]

[0040] Find the average value avr of non-zero dst(x,y), and convert avr to distance z through the unit conversion factor g, where z is the actual ground distance.

[0041] Example 1

[0042] This embodiment uses the method of the present invention to achieve distance measurement of the stubble-covered surface without interference from the covering. The information sources used are the TOF camera and the color 2D camera. The color plane image is segmented through the neural network model to obtain the stubble covering range, thereby removing the stubble covering influence area in the depth image and obtaining the real ground distance, which is as follows:

[0043] Step 1: Use the image data set to train the neural network. The image data set of this embodiment includes a color plane image of the surface and a binary image corresponding to the calibrated stubble-covered area. This embodiment adopts a U-Net neural network. The input is a 3-channel 8-bit color plane image with a resolution of 640×480, and the output is a single-channel 8-bit binary image with a resolution of 640×480. The neural network is trained using a data set containing color plane images of stubble-covered areas and corresponding stubble mask images. The data set contains 1,000 groups of images. After the neural network training, a neural network model that can segment the stubble-covered area in the color plane image is obtained.

[0044] Step 2: Use a TOF camera and a color 2D camera to obtain a depth image and a color plane image of the stubble-covered surface respectively. The depth image collected in this embodiment is combined with Figure 2 The resolution of the 16-bit single-channel grayscale image is 640×480, and the collected color plane image is combined with Figure 3The 3-channel 8-bit RGB color image with a resolution of 1280×1024 is shown, in which the color plane image has radial distortion. The radial distortion of the color plane image is corrected, and then the color plane image and the depth image are registered by reverse mapping using the intrinsic parameter matrix of the color 2D camera and the TOF camera, and the extrinsic parameter matrix of the TOF camera with the color 2D camera coordinate system as the world coordinate system. The obtained color plane image aligned with the depth image is combined Figure 4 As shown in the figure, the aligned color plane image has the same resolution as the depth image, both of which are 640×480. The pixels with the same coordinates (x, y) are consistent with the corresponding positions of the target object. The coordinates (x, y) in the aligned color plane image can be directly applied to the operation of the depth image.

[0045] Step 3: Input the color plane image processed in step 2 into the U-Net neural network model trained in step 1 for inference. The binary image mask1 of the stubble coverage area obtained by inference is combined with Figure 5 The single-channel 8-bit binary image shown in the figure shows the pixel values ​​of each point in the binary image mask1 of the stubble covered area as follows:

[0046]

[0047] Figure 5 Given the segmented binary image, Figure 4 By comparison, it can be seen that the use of neural networks has a good segmentation effect on the stubble covered area.

[0048] Step 4: Dilate the binary image mask1 of the stubble coverage area obtained in step 3 to obtain the combined Figure 6 The binary image mask2 of the stubble covered area with stronger robustness is shown. The pixel values ​​of each point in the binary image mask2 of the stubble covered area are as follows:

[0049]

[0050] Step 5: Let D be the set of all pixel coordinates (x, y) in the depth image, S D is the set of pixel coordinates of the stubble coverage area, G D is the set of pixel coordinates of the area without stubble coverage. From step 2, we can know that:

[0051] S D =S C

[0052] Therefore, the depth information of the area affected by the covering in the depth image can be removed by using the binary image mask2 of the stubble covered area in step 4, that is, the stubble covered area S in the depth image CThe pixel values ​​of the corresponding points are set to 0, and the pixel values ​​of the other points remain unchanged, thereby obtaining a depth image dst without stubble coverage. In this embodiment, the binary image mask2 after the dilation operation is an 8-bit image, which cannot be directly operated with the depth image obtained by the TOF camera and needs to be converted into a 16-bit image. The pixel values ​​of each point in the binary image mask2 after conversion are as follows:

[0053]

[0054] Use the pixel value d(x,y) of the depth image at the coordinate (x,y) to subtract the pixel value m(x,y) of the 16-bit image of mask2 at the coordinate (x,y) to obtain the depth image pixel value dst(x,y) at the coordinate (x,y) after removing the stubble coverage as follows:

[0055]

[0056] Depth image combination after removing the influence of stubble coverage Figure 7 As shown, the average value avr of non-zero dst(x,y) is calculated, and avr is multiplied by the unit conversion coefficient g from pixels to millimeters (g = 6.5536 in this embodiment) to obtain the distance z in millimeters, which is the ground distance after the stubble is removed. In this embodiment, the z value obtained is 512mm, the average distance obtained by using the depth image without removing the stubble is 506mm, and the average distance obtained by using the depth image after the stubble on the ground is completely cleaned is 513mm. The results show that this method has a significant effect on removing stubble interference.

[0057] Example 2

[0058] The difference between this embodiment and embodiment 1 is that in step 1, images of soils with different lighting conditions and stubble covering the ground in different forms are collected, and the aligned color plane images are combined with Figure 8 As shown in the figure, the binary image of the stubble coverage area obtained after the dilation operation is combined with Fig. 9 As shown, from Fig. 9 It can be seen that the segmentation effect is very good under different lighting environments, soil colors, and stubbles of different shapes, which shows that this method has strong adaptability.

[0059] The above embodiments verify that the method of the present invention can solve the problem of traditional distance measurement being interfered by covering objects by acquiring distance information over a large range. The multi-source perception combined with the feature segmentation of the color plane image has good effect and high precision in segmenting the stubble area, and can effectively eliminate the interference of surface stubble and accurately obtain the ground distance.

[0060] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other forms of assembly without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations within the meaning and range of equivalents of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0061] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for measuring the distance of stubble covered ground based on multi-source image deep learning, characterized in that: The following steps are involved: Step 1: using an image data set to train a neural network, wherein the image data set includes a plane image of the ground surface and a binary image corresponding to the calibrated stubble covered area, and after the neural network training, a neural network model capable of segmenting the stubble covered area in the plane image is obtained; Step 2: Obtain the depth image and plane image of the stubble-covered surface, and align the plane image with the depth image through registration so that the pixels under the same pixel coordinates (x, y) correspond to the same physical object, and the object distance corresponding to the pixel under the depth image (x, y) is z(x, y); Step 3: Input the plane image processed in step 2 into the neural network model trained in step 1 to obtain the binary image mask1 of the stubble covered area, which represents the stubble covered area S′ C and the area without stubble cover G′ C , the pixel values ​​m′(x,y) at different (x,y) coordinates in the binary image mask1 of the stubble covered area are: Step 4: Perform morphological operations on the binary image mask1 of the stubble covered area obtained in step 3 to improve robustness and obtain the binary image mask2 of the stubble covered area, which represents the stubble covered area S. C and no stubble covered area G C , the pixel values ​​m(x,y) at different (x,y) coordinates in the binary image mask2 of the stubble covered area are: Step 5: Let D be the set of all pixel coordinates (x, y) in the depth image, S D is the set of pixel coordinates of the stubble coverage area, G D is the set of pixel coordinates of the area without stubble coverage. From step 2, we can know that: S D =S C According to the binary image mask2 of the stubble covered area, the stubble covered area S in the depth image is C The pixel values ​​of the corresponding points are set to 0, and the pixel values ​​of the other points remain unchanged, so as to obtain the depth image dst after removing the stubble coverage. The pixel value at (x, y) in the depth image is d(x, y). Then the pixel value dst(x, y) at (x, y) in the depth image dst after removing the stubble coverage is as follows: Find the average value avr of non-zero dst(x,y), and convert avr to distance z through the unit conversion factor g, where z is the actual ground distance.

Images