Grain sampling point automatic generation method and device based on computer vision

Abstract

According to the automatic grain sampling point generation method and device based on computer vision, automatic identification and three-dimensional positioning of a vehicle-mounted grain surface are achieved through a double-branch deep neural network fusing stereo matching and semantic segmentation, sampling points are generated randomly according to the automatic identification and the three-dimensional positioning, the sampling points are transmitted to sampling equipment, and full automation of grain sampling in and out of a warehouse is achieved. The method comprises the following steps: firstly, calibrating a binocular camera to obtain internal and external parameters, collecting a sampling vehicle image, then training a model by adopting a method of combining domain adaptive learning and semi-supervised learning, then obtaining a vehicle-mounted grain surface and three-dimensional coordinates thereof by utilizing the trained model, and randomly sampling; and mapping the coordinates of the sampling points from a camera coordinate system to a sampler coordinate system, and sending the coordinates to sampling equipment to finally complete automatic sampling. According to the device, deep learning and computer vision technologies are combined, automatic generation of grain in-out warehouse sampling points is achieved, grain depot management automation and sampling efficiency can be improved, and grain depot management informatization and intelligent technology development is further promoted.

Description

Technical Field

[0001] The invention relates to a method and device for automatically generating grain sampling points based on computer perception. Background Art

[0002] At present, grain sampling usually adopts manual or manual remote monitoring methods, which is not only time-consuming and labor-intensive, but also easily interfered by human factors, with monitoring blind spots, making it difficult to ensure the objectivity and representativeness of the sampling process. In recent years, researchers and grain-related companies have proposed some advanced automated sampling methods to improve the accuracy and intelligence level of grain sampling, such as automatic sampling methods based on robotic arms, automatic sampling methods based on lidar, automatic sampling methods based on drones, and automatic sampling methods based on image processing.

[0003] Compared with the traditional manual sampling method, the above-mentioned automated sampling method improves the efficiency and accuracy of sampling to a certain extent, and reduces the errors and interference caused by manual operation. However, these methods also have some shortcomings. For example:

[0004] (1) The automatic sampling method based on a robotic arm requires precise mechanical control and positioning technology. The system is complex, the cost is high, and the adaptability is poor in complex environments.

[0005] (2) The automatic sampling method based on LiDAR relies on the accuracy and stability of LiDAR, which is limited by the equipment cost and laser life. In addition, the detection reliability is affected in the dusty granary environment.

[0006] (3) The automatic sampling method based on drones is suitable for sampling large-scale granaries, but it is difficult to operate in closed or narrow spaces, and the battery life and flight stability of drones are limiting factors.

[0007] (4) The automatic sampling method based on image processing relies on the accuracy of the camera and image processing algorithm, but the recognition effect may not be ideal when the lighting changes and the grain surface is uneven.

[0008] In order to solve the above problems, the present invention proposes a method and device for automatically generating grain sampling points based on computer vision. By combining binocular cameras, deep learning and computer vision technology, the method automatically identifies and locates the grain surface of the vehicle, obtains the three-dimensional coordinates of the grain surface, and randomly generates sampling point information, thereby achieving full automation of the sampling process and ensuring the representativeness and objectivity of the sampling. The system not only improves the sampling efficiency and accuracy, reduces the risk of human intervention, but also can adapt to different types of granary environments, and has high practical value and promotion prospects. Summary of the invention

[0009] The purpose of the present invention is to provide a method and device for automatically generating grain sampling points based on computer perception, so as to solve the problems of low efficiency, susceptibility to human interference and monitoring blind spots existing in the existing grain sampling methods.

[0010] To achieve the above object, the technical solution adopted by the present invention is specifically to propose a method and device for automatically generating grain sampling points based on stereo matching combined with semantic segmentation, comprising the following steps:

[0011] (1) Use a calibrated binocular camera to capture a sampled vehicle image, split the image into left and right images according to the resolution requirement, and use the binocular camera calibration parameters to correct the left and right images;

[0012] (2) Inputting the corrected left and right images into a deep neural network model that integrates stereo matching and semantic segmentation, using the semantic segmentation network branch to identify and extract the grain surface and the stereo matching network branch to obtain the disparity map of the left image;

[0013] (3) Calculate the depth information of the left image based on the calibration parameters and the disparity map to generate three-dimensional coordinate information;

[0014] (4) Based on the grain surface extracted by the semantic segmentation network branch, randomly generate sampling points and obtain their corresponding three-dimensional coordinate information;

[0015] (5) The generated three-dimensional coordinate information of the sampling point is converted into the coordinate system of the sampling device, and the three-dimensional coordinate information of the sampling point and the identified grain information are sent to the sampling device according to the set interface, and the sampling device samples the grain surface according to the sampling point.

[0016] Furthermore, the dual-branch deep neural network model for fusion of stereo matching and semantic segmentation is trained using a domain adaptive learning method and a semi-supervised method, specifically including:

[0017] (1) Use domain adaptive learning method to train the stereo matching branch network;

[0018] (2) Based on the pre-training of the public dataset, the semantic segmentation branch network is trained using manually labeled training samples of grain on board.

[0019] (3) Use the semantic segmentation branch network to guide the training of the stereo matching branch network.

[0020] Furthermore, in order to further improve the accuracy of model training, a semi-supervised learning strategy is adopted in the model training process, specifically including:

[0021] (1) While collecting grain surface samples on the vehicle, use a laser rangefinder to measure the distance from the left camera of the binocular camera to the grain surface at the specified position;

[0022] (2) Obtain the depth map of the left image using the obtained disparity map and the calibration parameters of the binocular camera;

[0023] (3) The error between the distance of the specified pixel in the depth map and the actual distance is calculated as the loss to guide the training model to improve the robustness of the model.

[0024] Furthermore, the stereo matching module uses a stereo matching algorithm to calculate a disparity map, thereby obtaining depth information of the grain surface, wherein the relationship between the disparity map D and the depth Z is: f is the focal length of the camera, and B is the baseline distance between the two cameras.

[0025] The stereo matching algorithm compares the left view image I L and right view image I R The disparity map D is generated by the pixel difference in the left and right view images. The disparity map D represents the pixel offset of the same object in the left and right view images. According to the disparity map D and the camera parameters f and B, the depth Z of each pixel can be calculated to obtain the depth information of the entire grain surface.

[0026] Furthermore, the semantic segmentation module uses a pre-trained convolutional neural network model to perform semantic segmentation on the captured image, identify the grain surface area, and generate a segmentation mask S, where:

[0027]

[0028] The semantic segmentation module first inputs the captured image into a pre-trained deep neural network model. The model extracts features from the image through a series of convolution and pooling operations, and finally generates a classification result for each pixel. By processing the classification results, a segmentation mask S of the grain surface area can be obtained, where a pixel with a value of 1 indicates that it belongs to the grain surface, and a pixel with a value of 0 indicates that it does not belong to the grain surface.

[0029] Furthermore, the data processing module calculates the three-dimensional coordinates (X, Y, Z) of each pixel based on the stereo matching and semantic segmentation results, where: Among them, (c x ,c y ) is the optical center coordinate of the camera.

[0030] The data processing module first calculates the depth value Z of each pixel based on the stereo matching results. Then, it combines the image coordinates (X, Y) and the camera parameters f and the optical center coordinates (c x ,c y ), calculate the three-dimensional coordinates (X, Y, Z) of each pixel. These three-dimensional coordinates can be used to generate a three-dimensional model of the grain surface and provide data support for subsequent sampling point generation.

[0031] Furthermore, the sampling point generation module uses a random algorithm to generate sampling points in the grain surface area, and the generated sampling points (X s ,Y s ,Z s )Satisfies: X s ∈[X min ,X max ],Y s ∈[Y min ,Y max ],Z s ∈Z surface (X s ,Y s ).

[0032] The sampling point generation module first determines the boundary of the sampling area [X min ,X max ] and [Y min ,Y max Then, a random algorithm is used to generate multiple sampling points (X s ,Y s ), and obtain the corresponding depth value Zs according to the grain surface model to ensure that the generated sampling points are evenly distributed on the grain surface. If the random point is not on the grain surface, remove the remaining part of the normal sampling point at the normal grain surface point and resample.

[0033] Furthermore, the sampling device generates the sampling point information (X i ,Y i ,Z i ) Samples are taken from the grain surface.

[0034] Furthermore, the specific process of step (2) is as follows:

[0035] (1) inputting the corrected left and right images in step (1) into a deep neural network integrating stereo matching and semantic segmentation functions;

[0036] (2) Process the image through the semantic segmentation network branch, identify the grain surface areas in the image, and extract the feature information of these areas;

[0037] (3) The stereo matching network branch processes the left and right images and calculates the disparity map of the left image. The disparity map is the key to the subsequent depth information calculation.

[0038] Furthermore, the specific process of step (3) is as follows:

[0039] (1) Calculate the grain surface depth information Z based on the disparity map D and camera parameters, where

[0040]

[0041] (2) Generate three-dimensional coordinate information of each pixel through depth information.

[0042] Furthermore, the specific process of step (4) is as follows:

[0043] (1) using the grain surface information extracted by the semantic segmentation network branch in step (2), randomly generating sampling points;

[0044] (2) Obtain the three-dimensional coordinate information of these sampling points relative to the grain surface on the vehicle.

[0045] Furthermore, the specific process of step (5) is as follows:

[0046] (1) converting the three-dimensional coordinates of the sampling point calculated in step (4) from the coordinate system of the grain surface on the vehicle to the coordinate system used by the sampling equipment, which involves geometric transformations such as coordinate rotation and translation;

[0047] (2) When performing coordinate transformation, the actual working range and mechanical limitations of the sampling equipment should be taken into account to ensure that each sampling point is within the reach of the equipment.

[0048] (3) The converted three-dimensional coordinate information of the sampling point is combined with the identified grain information and sent to the sampling device through a preset interface;

[0049] (4) The sampling device performs accurate sampling of the grain surface according to the received information. Compared with the traditional grain storage sampling method, the beneficial effects of the present invention are:

[0050] 1. Operational improvement - fully automated sampling process: The present invention realizes fully automated sampling process. The sampling point generation module uses a random algorithm to generate uniformly distributed sampling points (X s ,Y s ,Z s ). The sampling equipment receives the information of these sampling points and accurately locates the grain surface for sample extraction. The fully automated sampling process not only greatly improves the sampling efficiency, but also significantly reduces the labor intensity and avoids the errors and safety hazards that may be caused by manual operation. This improvement in automated operation makes the present invention more efficient and reliable in practical applications, and is suitable for quality inspection during large-scale grain transportation and storage.

[0051] 2. Technological breakthrough - deep learning model training and high-precision acquisition of three-dimensional coordinates of grain surface: The present invention uses a binocular camera combined with stereo matching and semantic segmentation technology to achieve accurate identification and positioning of grain surface; the proposed model is trained through domain adaptive learning and semi-supervised learning methods, avoiding time-consuming and labor-intensive manual labeling while ensuring the accuracy of the model. Based on the segmented grain surface and the obtained disparity map and other information, the data processing module calculates the three-dimensional coordinates (X, Y, Z) of the grain surface. This method has the characteristics of low cost and high precision, and provides reliable data support for subsequent automatic sampling and testing. This technological breakthrough has significantly improved the automation level of the sampling process and the accuracy of the sampling results. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of the automatic grain sampling system provided by the present invention;

[0053] Figure 2 It is a schematic diagram of a deep neural network model for the fusion of stereo matching and semantic segmentation provided by the present invention;

[0054] Figure 3 It is a schematic diagram of a sampling image domain adaptive stereo matching model based on image feature conversion provided by the present invention. DETAILED DESCRIPTION

[0055] In order to elaborate on the technical solutions and advantages of the present invention, the present invention is further described in detail below in conjunction with specific implementation steps and drawings.

[0056] (1) The binocular camera can be fixed on the arm of the sample machine or fixed separately to a position where it can capture the entire vehicle;

[0057] (2) Use the calibration board and the built-in stereo calibration program of MATLAB to calibrate the binocular camera and obtain the internal and external parameters of the camera, which are used to correct the left and right images and obtain the three-dimensional information of the object;

[0058] (3) Using a calibrated binocular camera to capture a sampled vehicle image, the image is split into left and right images according to the resolution requirement, and the left and right images are corrected using the binocular camera calibration parameters;

[0059] (4) Inputting the corrected left and right images into the stereo matching and semantic segmentation fusion deep neural network model, using the semantic segmentation network branch to identify and extract the grain surface, and the stereo matching network branch to obtain the disparity map of the left image;

[0060] (5) Use the collected images to train the model. Since stereo matching training samples are not easy to label, domain adaptive learning methods and semi-supervised learning methods are used to train the fusion model. This patent does not limit the selection of training strategies, and only selects one strategy for illustration. The details are as follows:

[0061] Compared with stereo matching, semantic segmentation samples are easier to annotate. Therefore, an annotation tool (such as Labelme) is used to manually annotate the collected left image to construct a semantic segmentation annotation set of the sampled image, which is used to train the semantic segmentation branch and guide stereo matching training. Figure 2 A schematic diagram of a deep neural network model that integrates stereo matching and semantic segmentation is shown. The model is divided into a backbone network, a semantic segmentation branch, a stereo matching branch, and a domain adaptation module. The backbone network can adopt the popular backbone network of deep learning, the semantic segmentation branch adopts a model similar to DeepLab v3, and stereo matching uses a lightweight network based on correlation coefficients.

[0062] During the training process, several domain adaptation and semi-supervised learning strategies are adopted.

[0063] Semantic-Guided Stereo Matching:

[0064] First, the backbone network and stereo matching branch are trained using a public stereo matching dataset. Then, the backbone network and stereo matching network molecules are frozen. The semantic segmentation branch is trained using the sampled image semantic segmentation samples. The cross entropy loss L is used. ce Optimize the semantic segmentation branch parameters; freeze the semantic segmentation branch parameters, release the backbone network, and use the semantic segmentation sample training model to update the backbone network again; then freeze the backbone network, train the stereo matching branch network, use the predicted disparity map to deform the semantic segmentation of the right image, and compare it with the semantic segmentation of the left image, and select the smooth L1 loss function L sm Optimize parameters and use multiple unsupervised loss functions L at the same time u Guide the training of the stereo matching network. In order to align the stereo matching network features and the semantic segmentation features, the semantic segmentation Decoder features are fused into the stereo matching features, or a fusion-and-separation strategy can be adopted.

[0065] Semi-supervised stereo matching:

[0066] In order to further improve the accuracy of model training, a semi-supervised learning strategy is adopted in the model training process. While collecting grain surface samples on the vehicle, a laser rangefinder is used to measure the distance from the left camera of the binocular camera to the grain surface at the specified position; the depth map of the left image is obtained using the obtained disparity map and the calibration parameters of the binocular camera; the error between the distance of the specified pixel in the depth map and the actual distance is calculated as the loss to guide the training model to improve the robustness of the model.

[0067] Stereo matching based on color distribution transformation:

[0068] like Figure 3As shown in Figure 1, in order to train the stereo matching network branch more directly, during the training process, the color distribution conversion between the existing public dataset and the sampled images is used to directly use the sampled images to train the stereo matching branch. Figure 3 The specific operation is as follows: training reads the public dataset image I contained in each batch s and the sampled image I t , using momentum update to calculate the mean μ of the sampled image set t and variance σ t , and calculate the mean μ of the public dataset images s and variance σ s , and update the mean and variance of the currently calculated sampling image set to the public dataset image Then input the updated public dataset images into the model for training.

[0069] (6) During the sampling process, the images captured by the binocular camera are segmented and rectified by the left and right images and input into the fusion model to obtain the disparity map and semantic segmentation results. The depth information Z and X and Y coordinates of the left image are calculated based on the calibration parameters and the disparity map. The relationship between the disparity map D and the depth Z is: f is the focal length of the camera, B is the baseline distance between the two cameras, and three-dimensional coordinate information is generated;

[0070] The stereo matching algorithm compares the left view image I L and right view image I R The disparity map D is generated by the pixel difference in the left and right view images. The disparity map D represents the pixel offset of the same object in the left and right view images. According to the disparity map D and the camera parameters f and B, the depth Z of each pixel can be calculated to obtain the depth information of the entire grain surface.

[0071] The semantic segmentation module uses a pre-trained convolutional neural network model to perform semantic segmentation on the captured image, identify the grain surface area, and generate a segmentation mask S, where:

[0072]

[0073] The semantic segmentation module first inputs the captured image into a pre-trained convolutional neural network model. The model extracts features from the image through a series of convolution and pooling operations, and finally generates a classification result for each pixel. By processing the classification results, a segmentation mask S of the grain surface area can be obtained, where a pixel with a value of 1 indicates that it belongs to the grain surface, and a pixel with a value of 0 indicates that it does not belong to the grain surface.

[0074] The data processing module calculates the three-dimensional coordinates (X, Y, Z) of each pixel based on the stereo matching and semantic segmentation results, where: Among them, (c x ,c y ) is the optical center coordinate of the camera.

[0075] The data processing module first calculates the depth value Z of each pixel based on the stereo matching results. Then, it combines the image coordinates (X, Y) and the camera parameters f and the optical center coordinates (c x ,c y ), calculate the three-dimensional coordinates (X, Y, Z) of each pixel. These three-dimensional coordinates can be used to generate a three-dimensional model of the grain surface and provide data support for subsequent sampling point generation.

[0076] (7) Based on the grain surface extracted by the semantic segmentation network branch, randomly generate sampling points and obtain their corresponding three-dimensional coordinate information. The sampling point generation module uses a random algorithm to generate sampling points in the grain surface area. The generated sampling points (X s ,Y s ,Z s )Satisfies: X s ∈[X min ,X max ],Y s ∈[Y min ,Y max ],Z s ∈Z surface (X s ,Y s ). The random sampling strategy is: according to the size of the grain surface area, the number of sampling points is decomposed into rows and columns, and the grain surface area is divided into corresponding equal-sized areas. Coordinate information is randomly generated in each area. If the coordinates of the selected sampling points are not in the grain surface area, they are reselected until the number of points required for sampling is met. In addition, the sampling points near the boundary are removed to ensure the normal operation of the sampling machine.

[0077] (8) Convert the generated three-dimensional coordinate information of the sampling point into the coordinate system of the sampling device;

[0078] (9) The three-dimensional coordinate information of the sampling point and the identified grain information are sent to the sampling device according to the set interface, and the sampling device samples the grain surface according to the sampling point.

Claims

1. A method and device for automatically generating grain sampling points based on computer vision, as shown in FIG1, the steps for generating sampling points are as follows: (1) Use a calibrated binocular camera to capture a sampled vehicle image, split the image into left and right images according to the resolution requirement, and use the binocular camera calibration parameters to correct the left and right images; (2) Inputting the corrected left and right images into a deep neural network model that integrates stereo matching and semantic segmentation, using the semantic segmentation network branch to identify and extract the grain surface and the stereo matching network branch to obtain the disparity map of the left image; (3) Calculate the depth information of the left image based on the calibration parameters and the disparity map to generate three-dimensional coordinate information; (4) Based on the grain surface extracted by the semantic segmentation network branch, randomly generate sampling points and obtain their corresponding three-dimensional coordinate information; (5) The generated three-dimensional coordinate information of the sampling point is converted into the coordinate system of the sampling device, and the three-dimensional coordinate information of the sampling point and the identified grain information are sent to the sampling device according to the set interface, and the sampling device samples the grain surface according to the sampling point.

2. The method and device for automatically generating grain sampling points according to claim 1, characterized in that: The dual-branch deep neural network model for fusion of stereo matching and semantic segmentation is trained using a domain adaptive learning method and a semi-supervised method, specifically including: (1) Use domain adaptive learning method to train the stereo matching branch network; (2) Based on the pre-training of the public dataset, the semantic segmentation branch network is trained using manually labeled on-board grain samples; (3) Use the semantic segmentation branch network to guide the training of the stereo matching branch network.

3. The method and device for automatically generating grain sampling points according to claim 1, characterized in that: In order to further improve the accuracy of model training, a semi-supervised learning strategy is used in the model training process to further train the stereo matching network branch, specifically including: (1) While collecting grain surface samples on the vehicle, use a laser rangefinder to measure the distance from the left camera of the binocular camera to the grain surface at the specified position; (2) Obtain the depth map of the left image using the obtained disparity map and the calibration parameters of the binocular camera; (3) The error between the distance of the specified pixel in the depth map and the actual distance is calculated as the loss to guide the training model to improve the robustness of the model.

4. The method and device for automatically generating grain sampling points according to claims 1 and 2, characterized in that: The stereo matching module uses a stereo matching algorithm to calculate the disparity map to obtain the depth information of the grain surface; the semantic segmentation module uses a pre-trained deep neural network model to perform semantic segmentation on the captured image, identify the grain surface area, and generate a segmentation mask. The model can further identify the grain category.

5. The method and device for automatically generating grain sampling points according to claim 1, characterized in that: The data processing module calculates the three-dimensional coordinates (X, Y, Z) of each pixel based on the stereo matching and semantic segmentation results, where Z is the depth information of the object relative to the left camera.

6. The method and device for automatically generating grain sampling points according to claim 1, characterized in that: The sampling point generation module uses a random algorithm to generate sampling points in the grain surface area. s ,Y s ,Z s )Satisfies: X s ∈[X min ,X max ],Y s ∈[Y min ,Y max ],Z s ∈Z surface (X s ,Y s ). The sampling point generation module first determines the boundary of the sampling area based on the three-dimensional coordinates of the grain surface [X min ,X max ] and [Y min ,Y max Then, a random algorithm is used to generate multiple sampling points (X s ,Y s ), and obtain the corresponding depth value Z according to the grain surface model s , ensuring that the generated sampling points are evenly distributed on the grain surface. If the random point is not on the grain surface, the normal sampling point is removed from the determined grain surface area, and the remaining part is resampled until the sampling point number requirement is met. The sampling device generates a sampling point information (X i ,Y i ,Z i ) Samples are taken from the grain surface.

7. The method and device for automatically generating grain sampling points according to claim 1, characterized in that: The specific process of step (2) is as follows: (1) inputting the corrected left and right images in step (1) into a deep neural network integrating stereo matching and semantic segmentation functions; (2) Processing the image through the semantic segmentation network branch, identifying the grain surface area and grain category in the image, and extracting the feature information of these areas; (3) The stereo matching network branch processes the left and right images and calculates the disparity map of the left image.

8. The method and device for automatically generating grain sampling points according to claim 1, characterized in that: The specific process of step (3) is as follows: (1) Calculate the grain surface depth information Z based on the disparity map D and camera parameters, where (2) Generate three-dimensional coordinate information of each pixel through depth information.

9. The method and device for automatically generating grain sampling points according to claim 1, characterized in that: The specific process of step (4) is as follows: (1) Using the grain surface information extracted by the semantic segmentation network branch in step (2), randomly generate sampling points, decompose the number of sampling points into the number of rows and columns, split the grain surface into small areas according to the number of rows and columns, and randomly select row and column coordinate points in each small area as sampling points; (2) Obtain the three-dimensional coordinate information of these sampling points relative to the left camera.

10. The automated method for grain sampling according to claim 1, characterized in that: The specific process of step (5) is as follows: (1) converting the three-dimensional coordinates of the sampling point calculated in step (4) from the coordinate system of the grain surface on the vehicle to the coordinate system used by the sampling equipment, using on-site calibration and measurement combined with geometric transformations such as coordinate rotation and translation; (2) When performing coordinate transformation, the actual working range and mechanical limitations of the sampling equipment should be taken into account to ensure that each sampling point is within the reach of the equipment. (3) The converted three-dimensional coordinate information of the sampling point is combined with the identified grain information and sent to the sampling device through a preset interface; (4) The sampling equipment performs accurate sampling operations on the grain surface based on the received information.

Images