A method for judging changes in stored grain based on image processing

Abstract

The application provides a method for judging the change of stored grain based on image processing. The application uses the local grain depression information of different time periods to compare the depression profile range, thereby avoiding the misjudgment caused by the change of stored grain to a certain extent. The method can achieve the purpose of automatically detecting the stored grain condition regularly. The application comprises a grain loading line detection module and a local grain surface detection module. First, the grain loading line detection module performs image preprocessing on the collected grain warehouse image, and then extracts the grain loading line by using Hough transformation, and cuts the image based on the line. Then, the local grain surface detection module performs edge detection on the cut image, and extracts the profile of the grain surface depression. Finally, the stored grain change detection module compares the extracted depression profile with the previous profile to judge whether the grain loading amount of the grain warehouse changes. The application can improve the accuracy of judging the change of stored grain.

Description

TECHNICAL FIELD

[0001] The present application relates to a method for judging the change of stored grain based on image processing, belonging to the field of computer vision. BACKGROUND

[0002] With the development of intelligent grain depot, in order to master the grain condition in the warehouse, the Internet of Things technology such as infrared laser scanner and pressure sensor is often used to detect the storage condition of the grain depot. However, the above method has high requirements for hardware and high installation and maintenance cost. Therefore, the present application adopts a detection method based on computer vision, which only needs an edge device to capture grain surface images at regular intervals and send them to the server. The server further pre-processes the images, and then finds the range of the grain surface through edge detection and contour extraction, and compares it with the previously recorded grain surface range, so as to complete the function of detecting the change of the grain surface, thereby realizing the alarm function of the change of stored grain.

[0003] The existing grain surface recognition method based on computer vision is to calculate the overall height of the grain surface, and compare it with the overall height of the grain surface calculated once or previously, so as to judge whether the storage capacity of the grain depot changes. L. Li, Z. Dong et al. [1] proposed a method for judging the change of grain surface, which first extracts the grain loading line and the grain surface of the image according to the depth semantic segmentation model, and finally calculates the distance and area between the grain loading line and the grain surface, and compares them with the previous measurement to judge whether the grain loading capacity of the grain depot changes. [2] Y. Lin and Y. Sun et al. [3] Li L, Fei X et al. Figure 1 As shown in the two pictures, the height of the grain changes obviously, so the area between the grain surface and the grain loading line can be calculated, and then compared with the area measured last time to judge whether the grain surface changes.

[0004] However, the existing grain depot image shows that when the grain is transported away, the overall height of the grain surface changes slightly, and the grain surface is locally concave, so the storage condition of the grain cannot be judged according to the overall height. Figure 2 As shown in the figure, calculating the height of the grain surface and the grain loading line cannot accurately judge the change of the grain depot.

[0005] Based on the characteristics of the grain warehouse image data, the present application carries out edge detection and contour extraction for the concave situation of the grain surface to judge the local grain surface change. A method of grain warehouse image segmentation is designed to detect the loading line, and according to the loading line, the distance change between the overall height of the grain surface and the loading line is judged by cutting only the loading line and the grain surface, so as to improve the accuracy of judging the change of the stored grain.

[0006] References:

[0007] [1] L. Li, Z. Dong, T. Yang and H. Cao, "Deep Learning-Based Automatic Monitoring Method for Grain Quantity Change in Warehouse Using Semantic Segmentation," in IEEE Transactions on Instrumentation and Measurement, vol. 70, pp. 1-10, 2021, Art no. 5007110, doi: 10.1109 / TIM.2021.3056743.

[0008] [2] Lin, Ying, and Yueheng Sun. "Study for packaged granary grain quantity intelligent reckoning." 2008 International Symposium on Computer Science and Computational Technology. Vol. 2. IEEE, 2008.

[0009] [3] Li L, Fei X, Dong Z, et al. Computer vision-based method for monitoring grain quantity change in warehouses [J]. Grain & Oil Science and Technology, 2020, 3(3): 87-99. SUMMARY

[0010] The existing research on judging the change of stored grain based on computer vision is directed to the overall height of the grain surface, which ignores the problem of local collapse of the grain surface during a small amount of transportation, and is easy to cause misjudgment. In addition, since the performance of the deep learning model is seriously dependent on a large number of data samples, the method based on the semantic segmentation model to judge the loading line is difficult to implement due to the limitation of the data set. The present application adopts a computer vision-based grain surface detection method, compares the recessed profile range of the local grain food in different time periods, avoids the misjudgment of the change of stored grain to a certain extent, and achieves the purpose of automatically detecting the storage of grain.

[0011] The technical scheme adopted by the present application mainly consists of two parts: a grain loading line detection module and a local grain surface detection module. The grain loading line detection module is used to extract the grain loading line by using Hough transform after image preprocessing of the collected grain warehouse image, and the image is cut based on the line. The local grain surface detection module is used to extract the profile of the grain surface recess by performing edge detection on the cut image. Finally, the stored grain change detection module compares the extracted recess profile with the previous profile to determine whether the grain loading capacity of the grain warehouse has changed.

[0012] The specific scheme of the present application is shown in the accompanying Figure 3 .

[0013] Step 1: Grain warehouse image acquisition

[0014] The purpose of this step is to obtain grain warehouse pictures in different time periods through edge devices, and to make information notes for the pictures, so as to judge the change of the grain surface.

[0015] Step 1.1: The edge device collects grain warehouse information at regular intervals: the present application needs to judge the change of the grain surface according to the grain warehouse pictures in different time periods obtained from the edge device.

[0016] Step 2.2: Store the grain warehouse image and number: the edge device uploads the pictures collected at regular intervals to the server, the server numbers the pictures according to the time stamp information, and stores them, preparing for the input of the pictures to the self-built network.

[0017] Step 2: Grain warehouse image preprocessing

[0018] The purpose of this module is to perform morphological operation on the collected grain warehouse pictures, and to obtain the more essential morphology of the object through certain operation of the interaction between the object and the structural element, so as to prepare for the subsequent loading line detection and grain surface detection module.

[0019] Step 2.1: Image erosion operation: a 3*3 convolution kernel is used to perform an operation with iterations of 3, that is, a mathematical morphology classical operator is used for erosion to solve the problem of burr and isolated small pits on the grain surface edge.

[0020] Step 2.2 image inflation operation: a 3*3 all-1 matrix convolution kernel is used, wherein the anchor point position of the structure element is (-1, -1) by default, and 1 iteration operation is performed.

[0021] Step 2.3 image to grayscale: converting the granary image to a grayscale image can reduce the amount of calculation and provide a basis for subsequent feature point detection.

[0022] Step 3: loading line detection

[0023] The purpose of this step is to use an edge detection algorithm to obtain the position and length information of the loading line, so that the granary picture can be cropped to obtain an image containing the loading line and the grain surface condition.

[0024] Step 3.1 granary image edge detection: using cvCanny to detect edges can increase contrast and eliminate image noise, and converting the image matrix to a Boolean value to facilitate Hough line extraction.

[0025] Step 3.2 locate the loading line: the invention needs to find the loading line of the granary image to crop the picture, so it is necessary to find the position of the loading line in the granary image.

[0026] Step 4: grain surface change detection

[0027] The purpose of this step is to perform contour detection on the granary image processed in step 3, find the maximum contour and area, and then compare the maximum contour area of the input before and after two pictures to obtain the change of the stored grain.

[0028] Compared with the prior art, the present application has the following advantages:

[0029] 1. The method for judging the change of stored grain based on image processing adopted by the present application can compare the local grain depression information in different time periods, avoid the problem that the overall height of the grain surface changes slightly during loading, the grain surface is locally depressed, and the stored grain condition cannot be judged according to the overall height, and achieve the purpose of automatically detecting the stored grain condition regularly.

[0030] 2. The method for judging the change of stored grain based on image processing adopted by the present application detects the loading line through computer vision technology, segments the granary image according to the position of the loading line, then according to the characteristics of the granary image data, judges the local grain surface change condition in combination with edge detection and contour extraction technology, and judges the distance change between the overall height of the grain surface and the loading line, so as to improve the accuracy of judging the change of stored grain, and improve the efficiency and accuracy of granary management. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1The method is generally aimed at the situation of the existing detection method.

[0032] (Left: original warehouse; right: after the grain is transported)

[0033] Figure 2 The image collected by the warehouse camera. (Left: original warehouse; right: after the grain is transported)

[0034] Figure 3 The overall model diagram of the method proposed by the present application. DETAILED DESCRIPTION

[0035] The embodiments of the present application are described in detail below in combination with the accompanying drawings of the specification:

[0036] The present application is a method for judging the change of stored grain based on image processing. The method acquires pictures from the edge device at regular intervals, and then performs morphological operation, edge detection, and contour extraction operation to identify whether there is any change in the grain in the grain depot, thereby achieving the purpose of automatic detection. The specific process is as follows: the first stage is that the edge device collects grain depot pictures at regular intervals, uploads the grain depot pictures to the server, numbers the pictures and stores them. The second stage is the grain depot image preprocessing stage, which converts the image into a gray image by performing erosion and expansion morphological operation on the image. The third stage uses Canny operator to perform edge detection, uses Gaussian filter to smooth the pixels in the image field, and then uses image gradient to represent the size of the edge gray scale transformation, finds the maximum value of the gradient in the divided gradient direction, and finally performs double threshold value detection to detect the edge. Finally, the Hough transform is used to extract straight lines, and the length and position information is used to obtain the grain loading line. The fourth stage performs cropping according to the grain loading line to avoid interference of redundant information. Then the image is binarized, the contour is extracted and the mesh structure is reconstructed to find the maximum contour and area, and the maximum contour areas in the input before and after two pictures are compared to determine the change of the grain surface.

[0037] Specifically, the method comprises the following steps:

[0038] Step 1: Grain depot image acquisition

[0039] The purpose of this step is to obtain grain depot pictures of different time periods through the edge device and make information notes for the pictures.

[0040] Step 1.1: The edge device collects grain depot information at regular intervals: the present application needs to judge the change of the grain surface according to the grain depot pictures of different time periods obtained from the edge device.

[0041] Step 1.2: Store the grain depot image and number it: the edge device uploads the pictures collected at regular intervals to the server, the server numbers the pictures according to the time stamp information, and stores them, which prepares for the input of the pictures to the self-built network.

[0042] Step 2: Granary image preprocessing

[0043] The purpose of this module is to perform morphological operation on the collected granary picture, to obtain the more essential morphology of the object through certain operation of the interaction between the object and the structural element, to prepare for the subsequent grain loading line detection and grain surface detection module.

[0044] Step 2.1 Image erosion operation: a 3*3 all-1 matrix convolution kernel is selected, wherein the anchor point position of the structural element is (-1, -1) by default, indicating that it is located at the center position of the structural element, and then 3 iteration operation operations are performed, that is, the classical operator erosion of mathematical morphology is used to solve the problem of burr and isolated small pit of the grain surface edge.

[0045] Step 2.2 Image dilation operation: a 3*3 all-1 matrix convolution kernel is used, and the anchor point position is selected as (-1, -1). One iteration operation is performed, and the new pixel value obtained after convolution is the maximum value in the filter, that is, the brightest point replaces the anchor point in the filter. In turn, the bright part of the image can be enhanced, and the boundary features can be enlarged, which provides a basis for the subsequent detection of the grain loading line and the contour.

[0046] Step 2.3 Image conversion to grayscale: converting the granary image to a grayscale image can reduce the amount of calculation and provide a basis for subsequent feature point detection.

[0047] Step 3: Grain loading line detection

[0048] The purpose of this step is to obtain the position and length information of the grain loading line using edge detection algorithm, so that the granary picture can be cropped to obtain an image containing the grain loading line and the grain surface condition.

[0049] Step 3.1 Granary image edge detection: using cvCanny to detect the edge can increase the contrast and eliminate the image noise, and the image matrix is converted to a Boolean value to facilitate the Hough line extraction.

[0050] Step 3.1.1 Gaussian smoothing filter: when smoothing the pixels in the image field using Gaussian filter, different positions of the pixels in the field are assigned different weights, and the weighted average gray value of the pixels in the template is used to replace the value of the pixel at the center of the template. In this application, a 3*3 Gaussian kernel is used, and the Gaussian filter calculates the weighted average of the nine numbers in the 3*3 neighborhood. Each point in the neighborhood is assigned a weight value according to its distance from the center point, and the closer to the center point, the greater the weight value. That is, it satisfies the Gaussian distribution (normal distribution), and the obtained 3*3 convolution kernel is used to perform Gaussian smoothing filter operation on the input image.

[0051] Step 3.1.2 Sobel calculation gradient: After Gaussian smoothing filtering is performed on the image, the partial Gaussian noise of the image is reduced, and the sharpness is also weakened, but at this time the edge decomposition of the image is not obvious, and the change of the gray scale is not strong, and it is not easy to capture. Therefore, in order to define the size of the edge gray scale change, the image gradient is proposed, and the size of the gradient is used to represent the size of the edge gray scale change.

[0052] Step 3.1.3 Non-maximum suppression: In the divided gradient direction, the local gradient maximum value is sought, and the gradient intensity of the current pixel is compared with the two pixels along the positive and negative gradient directions. If the gradient intensity of the current pixel is the maximum compared with the other two pixels, the pixel point is retained as an edge point, otherwise the pixel point will be suppressed.

[0053] Step 3.1.4 Double threshold detection: Double threshold detection is performed according to the following formula (1), wherein x and y are the horizontal and vertical coordinates of the point in the image respectively, src(x, y) is the point value of the original gray scale image (x, y), and dst(x, y) is the value of the (x, y) point of the output image.

[0054]

[0055] Double thresholding refers to setting the gray scale values greater than thresh1 and less than thresh2 as maxval, and setting the rest of the gray scale values as 0. By calculating the average gray scale value and the standard deviation of the image, the average value minus three times the standard deviation is selected as the low threshold thresh1, and the average value plus three times the standard deviation is selected as the high threshold thresh2. At the same time, the hysteresis threshold algorithm is used: according to the high threshold, an edge image is obtained, so that the image contains few false edges, but due to the high threshold, the generated image edge may not be closed, and to solve this problem, another low threshold is used. In the high threshold image, the edges are linked into contours, and when the end point of the contour is reached, the algorithm will search for a point that meets the low threshold in the 8-neighborhood points of the breakpoint, and then collect new edges according to the point until the entire image edge is closed.

[0056] Step 3.2 Locate the grain loading line: The present application needs to find the grain loading line of the grain depot image to crop the picture, so it is necessary to find the position of the grain loading line in the grain depot image.

[0057] Step 3.2.1 Hough transform to extract straight lines: Hough transform can be used for detection of regular geometric shapes. The principle is to convert each point in the Cartesian coordinate system to the polar coordinate space, forming a curve by curve, and the position of the intersection point of the curves in the image is considered as a point on the same straight line, and then the polar coordinates of these points are converted back to the Cartesian coordinate space by the formula.

[0058] Step 3.2.2: Obtain the grain loading line by position, length information: Since the grain loading line should be through the x-axis of the image and almost horizontal, after using the Hough transform to obtain all lines in the image, the following evaluation criteria is designed to determine whether it is a grain loading line:

[0059] S χ = L x θ (2)

[0060] S x The score represents the score of each line of the table, L represents the length of the line segment, and θ represents the angle between the line and the x-axis, that is, the greater the length L of the straight line, the smaller the angle with the x-axis, and the more characteristic it fits the grain loading line, and the higher the score obtained. The line with the highest score is the selected grain loading line.

[0061] Step 4: Change detection of grain surface

[0062] The purpose of this step is to perform contour detection on the grain storage image processed in step 3, find the maximum contour and area, and then compare the maximum contour areas of the input before and after two images to obtain the change of the stored grain.

[0063] Step 4.1: Cropping according to the grain loading line

[0064] After cropping the grain storage image, only the image information containing the grain loading line and the stored grain is obtained, avoiding the interference of unnecessary information such as windows and ceilings.

[0065] Step 4.2: Image binarization

[0066] Using the OTSU method, for a given image, when the threshold value of foreground and background is t, the proportion of foreground points in the image is W0, and the mean is U0. The proportion of background points in the image is W1, and the mean is U1. The number of pixels in each gray level in the image is counted to obtain a gray histogram, and the total number of pixels N of the image is calculated. Traverse each gray level i, calculate the foreground pixel number f i and the background pixel number b i . Calculate the foreground pixel proportion W0 and the background pixel proportion W1, the calculation formula of W0 is:

[0067]

[0068] The calculation formula of W1 is:

[0069]

[0070] The mean U of the entire image is:

[0071] U = W0 x U0 + W1 x U1 (5)

[0072] Establish the objective function

[0073] G(t) = W0(U0-U) 2 + W1(U1-U) 2 (6)

[0074] G(t) is the expression of the inter-class variance when the segmentation threshold is t. OTSU algorithm makes G(t) to obtain the global maximum value, when G(t) is maximum, the corresponding t is called the best threshold.

[0075] Step 4.3 contour detection

[0076] Extract all contours and re-establish the network structure, get each pixel of each contour, the difference of the pixel position of two adjacent points is not more than 1.

[0077] Step 4.4 find the largest contour and area: traverse all the area surrounded by the contour, find the largest area.

[0078] Step 4.5 judge the change of grain surface

[0079] After the input of the front and rear two pictures are processed by the network, each outputs its own largest contour area. Set the contour area of the first picture as A1, and the contour area of the second picture as A2, then consider that the grain area value of the picture with larger contour area is lower. That is:

[0080] A1>A2=>the grain surface of picture 1 is lower than that of picture 2.

[0081] A1<A2=>the grain surface of picture 2 is lower than that of picture 1.

Claims

1. A method for judging changes in stored grain based on image processing, characterized in that... Includes the following steps: Step 1: Grain warehouse image acquisition By acquiring images of grain silos at different time periods through edge devices and adding information annotations to the images, changes in the grain surface can be determined. Step 1.1 Edge devices periodically collect grain warehouse information: Based on grain warehouse images obtained from edge devices at different time periods, changes in grain surface are determined; Step 2.2 Store and number the grain warehouse images: The edge device uploads the periodically collected images to the server. The server numbers and stores the images according to the timestamp information, preparing them for input into the self-built network. Step 2: Grain warehouse image preprocessing details are as follows Step 2.1 Image erosion operation: Use a 3*3 convolution kernel to perform an operation with 3 iterations, that is, first use the classical mathematical morphology operator erosion to solve the problem of burrs and isolated small pits on the grain surface. Step 2.2 Image dilation operation: A 3*3 all-1 matrix convolution kernel is used, where the anchor point position of the structuring element is (-1, -1) by default, and one iteration operation is performed; Step 2.3 Convert the image to grayscale: Step 3.2 Positioning the grain loading line is as follows: Step 3.2.1 Extracting straight lines using Hough transform: Step 3.2.2 Obtain the loading line using position and length information: Since the loading line should run through the x-axis of the image and be almost horizontal, after obtaining all lines in the image using Hough transform, the following evaluation criteria are designed to determine whether it is a grain loading line: （2） The value represents the score for each line, L represents the length of the line segment, and θ represents the angle between the line and the x-axis. In other words, the larger the line length L, the smaller the angle between the line and the x-axis. The closer its features are to the grain loading line, the higher the score will be; the line with the highest score is the selected grain loading line. Step 4: Grain surface change detection: The grain warehouse image processed in step 3 is subjected to contour detection to find the maximum contour and area. The maximum contour area of ​​the two input images is then compared to obtain the changes in grain storage. Step 4.1 Cutting according to the grain loading line is as follows: Step 4.2 Image Binarization: Using the OTSU method, for an image, let t be the segmentation threshold between foreground and background, and let the proportion of foreground elements in the image be... The mean is The background points occupy a certain percentage of the image. The mean is Count the number of pixels at each gray level in the image to obtain a gray-level histogram, calculate the total number of pixels N in the image, and iterate through each gray level i to calculate the number of foreground pixels. and the number of background pixels ; Calculate the proportion of foreground pixels and background pixel ratio The calculation formula is: （3） The calculation formula is: （4） The mean U of the entire image is: （5） Establish objective function (6) G(t) is the expression for the inter-class variance when the segmentation threshold is t; the OTSU algorithm makes G(t) reach the global maximum value, and the t corresponding to the maximum value of G(t) is called the optimal threshold; Step 4.3 Contour Detection: Extract all contours and rebuild the mesh structure, obtain each pixel of each contour, and ensure that the pixel position difference between two adjacent points does not exceed 1. Step 4.4 Find the maximum contour and area Iterate through the areas enclosed by all contours and find the largest area; Step 4.5 Determine changes in grain surface area After processing the two input images using the aforementioned network, each image outputs its maximum contour area. Let the contour area of ​​the first image be A1 and the contour area of ​​the second image be A2. The image with the larger contour area is considered to have a lower contour area value. That is: A1>A2 => The grain level in image 1 is lower than the grain level in image 2; A1<A2 => The beam surface in Image 2 is lower than the grain surface in Image 1.

2. The method according to claim 1, characterized in that... Includes the following steps: Step 1: Grain warehouse image acquisition Grain warehouse images at different time periods are acquired through edge devices, and information is added to the images. Step 1.1 Edge devices periodically collect grain warehouse information: Based on grain warehouse images obtained from edge devices at different time periods, changes in grain surface are determined; Step 1.2 Store the granary images and number them: The edge device uploads the pictures collected at regular intervals to the server. The server numbers and stores the pictures according to the timestamp information, preparing for inputting the pictures into the self-built network; Step 2: The preprocessing of the granary images is as follows: Step 2.1 Image erosion operation: Select a 3*3 all-1 matrix convolution kernel, where the anchor position of the structural element is defaulted to (-1, -1), indicating that it is located at the center of the structural element, and then perform 3 iterative operation operations, that is, use the classic operator of mathematical morphology to erode; Step 2.2 Image dilation operation: Adopt a 3*3 all-1 matrix convolution kernel, where the anchor position of the structural element is defaulted to (-1, -1), and perform 1 iterative operation; Step 2.3 Convert the image to grayscale: Step 3: The detection of the grain loading line is as follows: Step 3.1 Detect the edges of the granary image: Using cvCanny to detect edges can increase the contrast and eliminate image noise, and convert the image matrix into a boolean value for easy extraction of Hough lines; Step 3.1.1 Gaussian smoothing filter: When using Gaussian filtering to smooth the pixels in the image domain, pixels at different positions in the domain are assigned different weights, and the weighted average gray value of the pixels in the domain is used to replace the value of the pixel at the center of the template; Use a 3*3 Gaussian kernel, and Gaussian filtering calculates the weighted average of the 9 numbers in the 3*3 neighborhood. Each point in the neighborhood is set with a weight according to the distance from the center point. The closer to the center point, the greater the weight; that is, it satisfies the Gaussian distribution, and the obtained 3*3 convolution kernel is used to perform Gaussian smoothing filtering operation on the input image; Step 3.1.2 Calculate the sober gradient: In order to define the magnitude of this edge gray change, the image gradient is proposed, and the magnitude of the gradient is used to represent the magnitude of the edge gray change; Step 3.1.3 Non-maximum suppression: Seek the local gradient maximum in the divided gradient direction, and compare the gradient intensity of the current pixel with the two pixels along the positive and negative gradient directions; If the gradient intensity of the current pixel is the largest compared with the other two pixels, then this pixel point is retained as an edge point, otherwise this pixel point will be suppressed; Step 3.1.4 Double-threshold detection: Perform double-threshold detection according to the following formula (1), where x and y are the abscissa and ordinate of the point in the image respectively, src(x, y) is the value of the point (x, y) in the original grayscale image, and dst(x, y) is the value of the point (x, y) in the output image; （1） Double-thresholding means that for two thresholds thresh1 < thresh2, the gray values greater than thresh1 and less than thresh2 are set to maxval, and the rest of the gray values are set to 0; By calculating the average gray value and standard deviation of the image, select the average value minus three times the standard deviation as the low threshold thresh1, and the average value plus three times the standard deviation as the high threshold thresh2; At the same time, use the hysteresis threshold algorithm: Connect the edges into contours in the high-threshold image. When reaching the end point of the contour, search for points that satisfy the low threshold among the 8-neighborhood points of the break point, and then collect new edges based on this point until the entire image edge is closed.

Images