A random sampling method for bulk grain based on binocular vision and image segmentation technology

Abstract

The present invention relates to a random sampling method for bulk grain based on binocular vision and image segmentation technology. First, a binocular camera is used to obtain the left image and right image information of the grain loading area of ​​a bulk grain transport vehicle and correct them respectively. The corrected left image is segmented by an image segmentation model, and the pixel coordinates of the four corner points of the grain surface area boundary of the grain loading vehicle are identified and calculated. The grain surface area is divided into multiple sampling areas according to the sampling rule, and several sampling areas are randomly selected and their center points are used as sampling points. The pixel coordinates of the sampling points are calculated, and the three-dimensional space world coordinates of the sampling points are obtained by using a binocular vision stereo matching algorithm, so as to control the grain sampling machine to move the sampling rod to each sampling point to complete the random sampling operation. The method has a good recognition effect for the grain loading area and can realize the random selection of the sampling point position.

Description

Technical Field

[0001] The invention relates to a random sampling method for bulk grain based on binocular vision and image segmentation technology. Background Art

[0002] At present, in the process of grain procurement, most of the grain is transported in bulk by vehicles and ships, which increases the difficulty of grain inspection and sampling. Although electric sampling machines can be used to obtain grain samples from transport vehicles, the overall operation is too complicated and the labor intensity is high, so the sampling efficiency is low. During the peak period of grain collection, there are long queues of grain-selling vehicles and "difficulty in selling grain". In recent years, some grain depots have had problems with false samples in the process of grain procurement. Some grain inspection personnel have been too one-sided in the selection of areas during the sampling process, which can easily lead to problems such as stealing and replacing pillars, and passing inferior goods as good ones. This not only causes economic losses to grain purchasers, but is also a serious fraud. In order to ensure the authenticity and representativeness of grain sampling samples, it is necessary to strictly regulate the sampling procedures and operations, reasonably determine the sampling points according to the size of the vehicle; realize unattended automatic sampling to reduce interference from human factors; and achieve random sampling to avoid the formation of fixed sampling points and patterns, which will be exploited by bad vendors and cause losses to grain procurement companies. Summary of the invention

[0003] The purpose of the present invention is to provide a random sampling method for bulk grain based on binocular vision and image segmentation technology, so as to solve the problems of overly complicated operation, high labor intensity and human fraud in the existing manual sampling process.

[0004] To achieve the above object, the solution of the present invention includes:

[0005] A random sampling method for bulk grain based on binocular vision and image segmentation technology comprises the following steps:

[0006] (1) using a binocular camera to collect binocular images of the bulk grain transport vehicle and the grain loading area on the vehicle, respectively, wherein the binocular images include a left image and a right image;

[0007] (2) First, the binocular camera is calibrated and the binocular image is corrected; second, the grain surface area of ​​the collected vehicle-mounted grain loading area image is annotated, and the image segmentation model is used to perform target segmentation on the corrected left image to obtain a mask image; third, the edge coordinates of multiple target areas in the mask image are calculated; finally, the pixel coordinates of the four corner points of the grain surface area boundary of the grain loading vehicle are identified and calculated;

[0008] (3) Dividing the grain surface area into multiple sampling areas according to the sampling rules, randomly selecting a number of sampling areas and taking their center points as sampling points, and calculating the pixel coordinates of the sampling points;

[0009] (4) Using binocular vision stereo matching algorithm to complete the three-dimensional spatial positioning of the sampling point, obtain the three-dimensional spatial world coordinates of the sampling point, and thus control the grain sampling machine to move the sampling rod to each sampling point to complete the random sampling operation;

[0010] Step (2) of processing the image of the vehicle loading area includes the following steps:

[0011] S1: Calibrate the binocular camera, and use the calibration parameters to perform distortion correction and binocular correction on the left and right images of the grain loaded on the vehicle captured by the binocular camera;

[0012] S2: annotating the grain surface area of ​​the collected vehicle-mounted grain loading area image to obtain a training image set, using the training image set to train an image segmentation network model, and using the trained image segmentation network model to segment the corrected left image to obtain a mask image;

[0013] S3: Use Opencv to calculate the edge coordinates of multiple target areas in the mask image, set threshold parameters to filter the loading area, and generate the minimum bounding rectangle;

[0014] S4: Determine the coordinates of the four corner points of the grain loading area, calculate the distances between the edge of the grain loading area and the four sides and four corners of the minimum circumscribed rectangle, and adjust the weight parameters to determine the final pixel coordinates of the four corner points;

[0015] Furthermore, the calibration parameters include: camera intrinsic parameters, extrinsic parameters, and distortion coefficients;

[0016] Further, the distortion correction includes: radial distortion and tangential distortion correction;

[0017] Furthermore, the image segmentation network model is a U-Net model;

[0018] Furthermore, step (4) adopts a binocular vision stereo matching algorithm and includes the following steps:

[0019] S1: Matching cost calculation, using Census transformation method, respectively, the left and right images use the local grayscale difference in the pixel neighborhood to convert the pixel grayscale into a bit string, compare the pixel grayscale value in the neighborhood window with the grayscale value of the window center pixel, calculate the matching cost based on the comparison result, and then measure the correlation between the candidate pixels to be matched;

[0020] S2: Cost aggregation, using the SGM algorithm, globally optimizes the cost matrix obtained by calculating the matching cost by establishing connections between adjacent pixels, and performs one-dimensional aggregation of the matching costs of all paths around the pixel under all disparities to obtain the path cost values ​​under each path. Finally, all path cost values ​​are added together to obtain the matching cost value of the pixel after aggregation;

[0021] S3: Disparity calculation, using the winner-takes-all algorithm to process the cost matrix after cost aggregation, and selecting the disparity value corresponding to the minimum aggregated cost value for each pixel as the optimal disparity, and finally obtaining a disparity map of the same size as the left image;

[0022] S4: Disparity optimization, using the left-right consistency check algorithm to eliminate erroneous disparity caused by occlusion and noise, swapping the left and right images for stereo matching to obtain a new disparity map. According to the uniqueness constraint of the disparity, the disparity map of the left image is used to find the same-name point pixel of each pixel in the right image and the disparity value corresponding to the pixel. If the difference between the two disparity values ​​is less than a certain threshold, the uniqueness constraint is met and it is retained. Otherwise, the uniqueness constraint is not met and it is eliminated.

[0023] Furthermore, the process of completing the three-dimensional spatial positioning of the sampling points by using the binocular vision stereo matching algorithm in step (4) includes: using the stereo matching algorithm to find the same-name points corresponding to the sampling points identified in the left image in the right image, and obtaining the actual three-dimensional spatial coordinates of each point based on the binocular stereo vision positioning;

[0024] Beneficial effects of the present invention:

[0025] The random sampling method of bulk grain provided by the present invention combines binocular vision positioning and image segmentation technology, realizes the random selection of sampling point positions, ensures the scientificity and rationality of sampling point selection, improves operating efficiency, and enables the extracted grain samples to fully reflect the overall quality level of the purchased grain; and does not sample excessively, which is of great significance to protecting the interests of grain farmers and reducing grain loss and waste. In addition, the method has a good recognition effect for the grain loading area, can realize the random selection of sampling point positions, and provides a new technical route and solution for the problem that the sampling point selection is unscientific and unreasonable in the grain purchase process, resulting in insufficient representativeness of the sampling samples and the risk of human fraud, and provides technical support for the automation and intelligent development of the grain sampling link. The image of bulk grain on the vehicle is processed by the binocular vision method, which improves the reliability of the recognition model of bulk grain on the vehicle; the three-dimensional spatial positioning of the sampling point is completed by the binocular vision stereo matching algorithm, which improves the reliability of the three-dimensional spatial positioning of the sampling point. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flowchart of a random sampling method for bulk grain based on binocular vision and image segmentation technology.

[0027] Figure 2 The invention is a method for random sampling of bulk grain based on binocular vision and image segmentation technology to divide the sampling area and select the sampling points. DETAILED DESCRIPTION

[0028] like Figure 1As shown in the figure, the image acquisition module uses a binocular camera to collect images of the bulk grain area on the vehicle to obtain left and right images. Then, the binocular images are distorted and binocularly calibrated. The lens distortion is mainly divided into radial distortion and tangential distortion. OpenCV provides a correction algorithm that can be used directly. It can be completed at one time using the undistort() function or combined with initUndistortRectifyMap() and remap(). The purpose of binocular correction is to correct the two images that are not aligned in the same plane after the distortion is eliminated into coplanar alignment. Correct the actual binocular system into an ideal binocular system.

[0029] Then, a network model is constructed to achieve target segmentation of the corrected left image and calculate the coordinates of the corner points in the target area. There are many ways to construct a bulk grain image segmentation model, such as the classic semantic segmentation model FCN, Mask-RCNN, U-Net, etc. Here, a specific implementation process is given using U-Net as an example, but the present invention is not limited to the following specific process:

[0030] U-Net is generally an encoder-decoder structure, that is, an encoder and decoder structure. The left half of the network is an encoder structure for feature extraction, which mainly includes convolution and pooling operations, consisting of two 3×3 convolution layers and a 2×2 pooling layer, and the Relu activation function is used after each convolution operation. The right half is a decoder structure for upsampling, which is mainly composed of a 2×2 deconvolution layer, feature concatenation operations, and two 3×3 convolution layers, and the Relu activation function is used after each convolution operation.

[0031] First, the original image size is normalized to 512×512, and the image is enhanced by brightness enhancement, horizontal and vertical mirroring, diagonal inversion, translation and other operations. Data enhancement technology can speed up the construction of data sets, reduce model overfitting, and enhance the generalization ability of the model. After data enhancement, a data set of about 1,000 bulk grain images is obtained. The image annotation tool is used to manually annotate the data set, generate the corresponding json file, and obtain the corresponding label image. The U-Net network is trained according to the training image set to obtain the image recognition model of grain storage in the warehouse.

[0032] Furthermore, the parameters of the model are set, where the batch size is set to 5, that is, 5 images are processed each time; the learning rate is 0.0001; the running cycle is 10 times, and each cycle is iterated 2000 times;

[0033] Further, the path of the training image and the corresponding labeling result is set for the deep model;

[0034] Furthermore, the image input model is trained for forward calculation, and the prediction result is obtained through the softmax of the last layer. The loss function is calculated based on the manual labeling result, and the parameters are updated according to the parameter iteration formula obtained by gradient descent based on the value in the current network.

[0035] Finally, when the network reaches the maximum number of iterations or the preset stopping condition, the training ends and the bulk grain image recognition model is obtained.

[0036] Then, if Figure 2 As shown, the target area obtained by segmentation is divided into multiple sampling areas according to the sampling rule, and sampling points are randomly generated in each area. The stereo matching algorithm is used to perform stereo matching on the left and right images to construct a disparity map;

[0037] The stereo matching is also called disparity estimation, or binocular depth estimation. The input is a pair of left and right images captured at the same time and corrected by the epipolar line. The output is a disparity map composed of the disparity values ​​corresponding to each pixel in the reference image (usually the left image is used as the reference object). There are many kinds of stereo matching algorithms, such as the common local matching algorithm BM, semi-global matching algorithm SGBM, global matching algorithm GC, etc. Its calculation process mainly includes the following steps:

[0038] (1) Matching cost calculation, the purpose of which is to measure the correlation between the pixel to be matched and the candidate pixel. In computer vision, mutual information MI method, Census transformation method, Rank transformation method, BT method, etc. are often used as matching cost calculation methods. Here, the Census transformation method is used as an example to describe the specific calculation process, but the present invention is not limited to the following specific process:

[0039] Census transformation uses the local grayscale difference in the pixel neighborhood to convert the pixel grayscale into a bit string. By comparing the grayscale value of the pixel in the neighborhood window with the grayscale value of the pixel in the center of the window, the Boolean value obtained by comparison is mapped to a bit string, and finally the value of the bit string is used as the Census transformation value C of the center pixel. s , the specific calculation formula is as follows:

[0040]

[0041] The n′ and m′ are the largest integers not greater than half of n and m, respectively. ⊕ is a bit-by-bit concatenation operation. The ξ operation is defined by the following formula:

[0042]

[0043] The matching cost calculation method based on Census transformation is to calculate the Hamming distance of the Census transformation values ​​of the two pixels corresponding to the left and right images, that is:

[0044] C(u,v,d):=Hamming(C sl (u,v),C sr (ud,v)

[0045] The Hamming distance is the number of different corresponding bits of two bit strings, and is calculated by performing an XOR operation on the two bit strings and then counting the number of bits of the XOR operation result that are not 1.

[0046] (2) Cost aggregation, the purpose of which is to allow the cost value to accurately reflect the correlation between pixels. Common cost aggregation methods include line drawing method, dynamic programming method, path aggregation method in SGM algorithm, etc. The following takes the path aggregation method in SGM algorithm as an example to describe the specific calculation process, but the present invention is not limited to the following specific process:

[0047] The SGM algorithm adopts the idea of ​​global stereo matching algorithm, that is, the global energy optimization strategy. In simple terms, it is to find the optimal disparity of each pixel so that the global energy function of the entire image is minimized. The definition of the global energy function is as follows:

[0048]

[0049] Among them, C is the matching cost. The first term of the formula is the data term, which means the accumulation of the matching cost of all pixels when the disparity map is D. The second and third terms are smoothing terms, which means the N matching cost of pixel P. p All pixels q in the area are penalized, where the second penalty is smaller and the third penalty is larger.

[0050] In order to achieve the global energy optimization strategy, SGM proposes a path cost aggregation idea, that is, the matching cost of all disparities of the pixel is aggregated in one dimension on all paths around the pixel to obtain the path cost value under each path, and then all path cost values ​​are added to obtain the matching cost value of the pixel after aggregation. The path cost calculation method of pixel p along a path r is shown in the following formula:

[0051]

[0052] Among them, the first item is the data item, which is the matching cost value C; the second item is the smoothing item; the third item is to ensure that the new path cost value Lr does not exceed a certain upper limit, that is:

[0053] L≤C max +P2

[0054] Thus, the total path cost S of the pixel is obtained:

[0055]

[0056] (3) Disparity calculation: The optimal disparity value of each pixel is determined by the cost matrix after cost aggregation. The winner-takes-all algorithm (WTA) is often used for calculation.

[0057] The winner-takes-all algorithm calculates the following: Among all the cost values ​​of a pixel under all disparities, the disparity corresponding to the minimum cost value is selected as the optimal disparity. The result of the disparity calculation is a disparity map of the same size as the left image, which stores the disparity value of each pixel. When the internal and external parameters of the image are known, the disparity map can be converted into a depth map to indicate the position of each pixel in space.

[0058] (4) Disparity optimization, the purpose of which is to further optimize the disparity map obtained in the previous step, improve the quality of the disparity map, improve the accuracy of the disparity map, and make the disparity value more reliable and accurate. Disparity optimization generally uses a left-right consistency check algorithm to eliminate erroneous disparity caused by occlusion and noise;

[0059] Furthermore, the left-right consistency check is based on the uniqueness constraint of disparity, that is, there is at most one correct disparity for each pixel. The specific steps are to swap the left and right images, perform stereo matching again, and obtain another disparity map. Because each value in the disparity map reflects the correspondence between two pixels, based on the uniqueness constraint of the disparity, the disparity map of the left image is used to find the same-name point pixel of each pixel in the right image and the disparity value corresponding to the pixel. If the difference between the two disparity values ​​is less than a certain threshold (usually 1 pixel), it satisfies the uniqueness constraint and is retained. Otherwise, it does not satisfy the uniqueness constraint and is eliminated. The specific calculation formula is as follows:

[0060]

[0061] Finally, the disparity map is calculated based on the binocular vision stereo matching algorithm, the three-dimensional coordinates of each sampling point are calculated, and the grain sampling machine is controlled to move to each sampling point to complete the sampling operation.

[0062] Specific implementation methods are given above, but the present invention is not limited to the described implementation methods. The basic idea of ​​the present invention lies in the above basic scheme. For ordinary technicians in this field, it does not take creative work to design various deformed models, formulas, and parameters according to the teachings of the present invention. Changes, modifications, substitutions, and variations of the implementation methods without departing from the principles and spirit of the present invention still fall within the scope of protection of the present invention.

Claims

1. A random sampling method for bulk grain based on binocular vision and image segmentation technology, characterized in that: The following steps are involved: (1) using a binocular camera to collect binocular images of the bulk grain transport vehicle and the grain loading area on the vehicle, respectively, wherein the binocular images include a left image and a right image; (2) Processing the images of the grain loading area on the vehicle: First, calibrate the binocular camera and correct the binocular image; second, annotate the grain surface area of ​​the collected images of the grain loading area on the vehicle, and use the image segmentation model to perform target segmentation on the corrected left image to obtain a mask image; third, calculate the edge coordinates of multiple target areas in the mask image; finally, identify and calculate the pixel coordinates of the four corner points of the grain surface area boundary of the grain loading vehicle; (3) Divide the grain surface area into multiple sampling areas according to the sampling rules, randomly select several sampling areas and use their center points as sampling points, and calculate the pixel coordinates of the sampling points; (4) A binocular vision stereo matching algorithm is used to complete the three-dimensional spatial positioning of the sampling point and obtain the three-dimensional spatial world coordinates of the sampling point, thereby controlling the grain sampling machine to move the sampling rod to each sampling point to complete the random sampling operation.

2. According to claim 1, a random sampling method for bulk grain based on binocular vision and image segmentation technology is characterized in that: Step (2) of processing the image of the vehicle loading area includes the following steps: S1: Calibrate the binocular camera, and use the calibration parameters to perform distortion correction and binocular correction on the left and right images of the grain loaded on the vehicle captured by the binocular camera; S2: annotating the grain surface area of ​​the collected vehicle-mounted grain loading area image to obtain a training image set, using the training image set to train an image segmentation network model, and using the trained image segmentation network model to segment the corrected left image to obtain a mask image; S3: Use Opencv to calculate the edge coordinates of multiple target areas in the mask image, set threshold parameters to filter the loading area, and generate the minimum bounding rectangle; S4: Determine the coordinates of the four corner points of the grain loading area, calculate the distances between the edge of the grain loading area and the four sides and four corners of the minimum circumscribed rectangle, and adjust the weight parameters to determine the final pixel coordinates of the four corner points.

3. The method for random sampling of bulk grain based on binocular vision and image segmentation technology according to claim 2 is characterized in that: The calibration parameters include: camera intrinsic parameters, extrinsic parameters, and distortion coefficients.

4. The method for random sampling of bulk grain based on binocular vision and image segmentation technology according to claim 2 is characterized in that: The distortion correction includes radial distortion and tangential distortion correction.

5. The method for random sampling of bulk grain based on binocular vision and image segmentation technology according to claim 2 is characterized in that: The image segmentation network model is a U-Net model.

6. The method for random sampling of bulk grain based on binocular vision and image segmentation technology according to claim 1, characterized in that: Step (4) using binocular vision stereo matching algorithm includes the following steps: S1: Matching cost calculation, using Census transformation method, respectively, the left and right images use the local grayscale difference in the pixel neighborhood to convert the pixel grayscale into a bit string, compare the pixel grayscale value in the neighborhood window with the grayscale value of the window center pixel, calculate the matching cost based on the comparison result, and then measure the correlation between the candidate pixels to be matched; S2: Cost aggregation, using the SGM algorithm, globally optimizes the cost matrix obtained by calculating the matching cost by establishing connections between adjacent pixels, and performs one-dimensional aggregation of the matching costs of all paths around the pixel under all disparities to obtain the path cost values ​​under each path. Finally, all path cost values ​​are added together to obtain the matching cost value of the pixel after aggregation; S3: Disparity calculation, using the winner-takes-all algorithm to process the cost matrix after cost aggregation, and selecting the disparity value corresponding to the minimum aggregated cost value for each pixel as the optimal disparity, and finally obtaining a disparity map of the same size as the left image; S4: Disparity optimization, using the left-right consistency check algorithm to eliminate erroneous disparity caused by occlusion and noise, swapping the left and right images for stereo matching to obtain a new disparity map. According to the uniqueness constraint of the disparity, the disparity map of the left image is used to find the same-name point pixel of each pixel in the right image and the disparity value corresponding to the pixel. If the difference between the two disparity values ​​is less than a certain threshold, the uniqueness constraint is met and it is retained. Otherwise, it does not meet the uniqueness constraint and is eliminated.

7. The method for random sampling of bulk grain based on binocular vision and image segmentation technology according to claim 1, characterized in that: The process of completing the three-dimensional spatial positioning of the sampling points by using the binocular vision stereo matching algorithm in step (4) includes: using the stereo matching algorithm to find the corresponding points of the same name in the right image of the sampling points identified in the left image, and obtaining the actual three-dimensional spatial coordinates of each point based on the binocular stereo vision positioning.

Images