A method and system for detecting sorghum opening rate based on machine vision

Abstract

The present invention provides a machine vision-based method and system for detecting the opening rate of sorghum. The method comprises: capturing images of moving sorghum on a conveyor belt; detecting scattered sorghum using a convolutional neural network method; removing overlapping regions from the detected sorghum grains using a machine vision method; classifying the openings of the sorghum grains after removing overlapping regions using a convolutional neural network method; and detecting the opening size of the sorghum grains using a machine vision method. This application enables detection of the sorghum opening rate during the liquor brewing process and can accurately assess the sorghum opening rate.

Description

Technical Field

[0001] The present invention belongs to the field of winemaking, and in particular, relates to a sorghum opening rate detection method and system based on machine vision. Background Art

[0002] The application of machine vision artificial intelligence in the liquor brewing industry, particularly within production, is still in its infancy. Currently mature industrial machine vision solutions cannot be directly adapted to the liquor brewing process. The brewing process is subject to heavy fog, high humidity and moisture on the conveyor belts for the objects being measured, and severe accumulation and adhesion of objects. Furthermore, traditional methods for evaluating objects do not rely on a single visual evaluation method, and there are no comprehensive, quantitative visual evaluation standards. Summary of the Invention

[0003] The embodiment of the present application provides a method and system for detecting the opening rate of sorghum based on machine vision, which realizes the detection of the opening rate of sorghum during the liquor brewing process and can accurately evaluate the opening rate of sorghum.

[0004] In a first aspect, an embodiment of the present application provides a method for detecting the opening rate of sorghum based on machine vision, comprising:

[0005] Step S1, collecting images of moving sorghum on a conveyor belt;

[0006] Step S2, detecting scattered sorghum based on a convolutional neural network method;

[0007] Step S3, removing overlapping areas of the detected sorghum grains based on a machine vision method;

[0008] Step S4, performing opening classification on the sorghum grains after removing the overlapping areas based on a convolutional neural network method;

[0009] Step S5: detecting the opening size of the sorghum grains based on a machine vision method.

[0010] Wherein, the step S1 includes:

[0011] High-definition image data is acquired through a high-speed camera, and the RGB image in the original data is converted to grayscale while retaining the original RGB image. The grayscale image is used for scattered sorghum grain detection.

[0012] Wherein, the step S2 includes:

[0013] A sorghum scattered grain dataset was created, which includes image data of multiple overlapping scattered grains, single overlapping scattered grains, and blurred scattered grains. Each image contains a scene category label.

[0014] Design a convolutional neural network model, using the Resnet-50 structure as the skeleton model;

[0015] Training convolutional neural networks;

[0016] The trained network model is called to detect scattered particles.

[0017] Wherein, the step S3 includes:

[0018] The image containing sorghum grains is divided into four equal parts, and then a Gaussian function with a radius of 5 is used to calculate the average brightness of each equally divided window to obtain the segmentation threshold of each window. The original image is segmented using this threshold to obtain a binary image containing sorghum grains to remove the background interference of the conveyor belt and the overlapping sorghum grain areas.

[0019] Wherein, the step S4 includes:

[0020] The classic classification network MobileNet is used to perform binary classification on the segmented sorghum grain images, and the output results are opened and unopened.

[0021] Wherein, the step S5 includes:

[0022] The watershed algorithm is used to sort the gray level of each pixel from low to high by taking advantage of the difference between the open area and the unopen area of ​​sorghum;

[0023] In the process of flooding from low to high, each local minimum value is judged and marked using a first-in-first-out structure to obtain a marked gradient image;

[0024] Use the custom Laplace operator to calculate the opening part;

[0025] The adaptive segmentation threshold obtained in step S3 is used to perform denoising on the opening portion to obtain opening data.

[0026] Wherein, the step S5 includes:

[0027] Assume that there is a matrix function f(x, y) containing a single sorghum image. Its gradient at the point (x, y) is a vector with magnitude and direction. The gradients in the x-direction and y-direction are represented by Gx and Gy, respectively. The gradient expression is defined as follows:

[0028]

[0029] in,

[0030]

[0031]

[0032] Since Laplace is a differential operator that is used to significantly express the sudden changes in pixels in an image, the Laplace operator is defined as follows based on the image characteristics between the open and closed parts of sorghum grains:

[0033]

[0034] The gradient information after segmentation by the watershed algorithm was calculated using the Laplace operator, and only the opening part information of the opening sorghum was retained for the final statistical analysis.

[0035] In a second aspect, the present application provides a sorghum opening rate detection system based on machine vision, comprising:

[0036] a collection unit for collecting images of moving sorghum on a conveyor belt;

[0037] a first detection unit, configured to detect scattered sorghum using a convolutional neural network-based method;

[0038] a removal unit, used for removing overlapping areas of detected sorghum grains based on a machine vision method;

[0039] a classification unit for classifying the openings of sorghum grains after removing overlapping areas based on a convolutional neural network method;

[0040] The second detection unit is used to detect the opening size of sorghum grains based on a machine vision method.

[0041] In a third aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the above methods when executed by a processor.

[0042] In a fourth aspect, an embodiment of the present application provides a computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the program.

[0043] The sorghum opening rate detection method and system based on machine vision in the embodiment of the present application have the following beneficial effects:

[0044] This application proposes a machine vision-based method for detecting the opening rate of sorghum, including: capturing images of moving sorghum on a conveyor belt; detecting scattered sorghum using a convolutional neural network; removing overlapping regions from the detected sorghum grains using a machine vision method; classifying the openings of the sorghum grains after removing overlapping regions using a convolutional neural network method; and detecting the opening size of the sorghum grains using a machine vision method. This application enables detection of the sorghum opening rate during the liquor brewing process and can accurately assess the sorghum opening rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of a method for detecting sorghum opening rate based on machine vision according to an embodiment of the present application;

[0046] Figure 2 Schematic diagram of the pyramid network structure for detecting sorghum grains of different sizes in the embodiment of the present application;

[0047] Figure 3 This is a schematic diagram of the structure of the sorghum opening rate detection system based on machine vision in an embodiment of the present application;

[0048] Figure 4 This is a schematic diagram of the structure of the computer system according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] The present application will be further described below with reference to the accompanying drawings and embodiments.

[0050] In the following description, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The following description provides multiple embodiments of the present invention, and different embodiments can be replaced or combined, so this application can also be considered to include all possible combinations of the same and / or different embodiments described. Therefore, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then this application should also be considered to include embodiments containing one or more of all other possible combinations of A, B, C, and D, even though such embodiments may not be explicitly described in the following text.

[0051] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made to the function and arrangement of the elements described without departing from the scope of the present application. Various examples may appropriately omit, replace, or add various processes or components. For example, the described method may be performed in an order different from the order described, and various steps may be added, omitted, or combined. In addition, features described in some examples may be combined in other examples.

[0052] The present invention aims to provide a machine vision-based sorghum opening rate detection system that uses four detection stages to evaluate sorghum opening. The system uses an object detection algorithm to detect scattered sorghum grains, an adaptive threshold boundary segmentation algorithm to remove interference from overlapping grains, an object classification algorithm to detect whether sorghum is open, and a watershed segmentation algorithm to detect the opening size. This system is used to evaluate the steaming and brewing process in the sorghum winemaking industry.

[0053] like Figure 1As shown, the present invention's machine vision-based sorghum opening rate detection method includes: step S1, capturing images of moving sorghum on a conveyor belt; step S2, detecting scattered sorghum using a convolutional neural network method; step S3, removing overlapping regions from the detected sorghum grains using a machine vision method; step S4, classifying the openings of the sorghum grains after removing overlapping regions using a convolutional neural network method; and step S5, detecting the opening size of the sorghum grains using a machine vision method. Finally, statistical analysis of the sorghum opening results is performed.

[0054] Step S1 involves acquiring high-definition image data using a high-speed camera and converting the original RGB image into grayscale. The original RGB image is retained, and the grayscale image is used for scattered sorghum grain detection to reduce computation time. The image resolution is 1920 x 1280.

[0055] Step S2 includes: establishing a scattered sorghum grain dataset, which includes image data of multiple overlapping scattered grains, single overlapping scattered grains, and blurred scattered grains, each image containing a scene category label; designing a convolutional neural network model, in this example using the Resnet-50 structure as the backbone model of the algorithm; training the convolutional neural network; and calling the trained network model to detect scattered grains.

[0056] The convolutional neural network is designed based on the Resnet-50 basic network. Considering the common sorghum grain detection needs, three pyramid networks are designed to detect sorghum grains of different sizes. Figure 2 For specific network structures, such as Figure 2 As shown, the network structure is divided into three layers of pyramids. ResNet50 is divided into three ResUnit units, each with 16, 16, and 18 layers. The first layer of the pyramid passes directly through all ResNet units to obtain a 16x16 scale, which is used to detect small objects. After the second ResUnit unit (32 layers), the features are fused with the upsampled features to obtain a 32x32 scale, which is used to detect medium-sized objects, namely medium-sized sorghum grains. After the shallow ResUnit unit (16 layers), the features are upsampled and fused with the higher-level features to obtain 64x64 scale features for detecting large sorghum grains.

[0057] Step S3 includes: using an adaptive threshold method to segment the grains. The idea is: divide the picture containing the sorghum grains into four equal parts, and then use a Gaussian function with a radius of 5 to calculate the average brightness of each equally divided window to obtain the segmentation threshold of each window. Use this threshold to segment the original image to obtain a binary picture containing the sorghum grains. At this time, the background interference of the conveyor belt and the overlapping sorghum grain areas can be removed.

[0058] The calculation formula of Gaussian window function is shown as follows:

[0059]

[0060] Where x and y represent the pixel locations in the image, and σ is the standard deviation of the Gaussian window. Larger values ​​indicate a smoother Gaussian window, meaning that when calculating the threshold for an image block, points surrounding the center are given greater weight. Smaller values ​​indicate a more pronounced Gaussian window shape, resulting in a greater weighting of the center point. This adaptive threshold sliding window calculation method can reduce instabilities caused by the global threshold while preserving detailed information within each image block.

[0061] Step S4 includes: using the classic classification network MobileNet to perform binary classification on the segmented sorghum grain image, and outputting the result as opened and unopened.

[0062] Step S5 includes: detecting the opening size of the opening image output by step S4. The main idea is to use the watershed algorithm, taking advantage of the obvious difference between the open area and the unopened area of ​​sorghum, to sort the grayscale of each pixel from low to high; in the process of flooding from low to high, each local minimum is judged and marked using a first-in-first-out (FIFO) structure to obtain a marked gradient image; the opening part is calculated using a custom Laplace operator; and the opening part is denoised using the adaptive segmentation threshold obtained in step S3 to obtain accurate opening data.

[0063] Step S5 includes: assuming that there is a matrix function f(x, y) containing a single sorghum image, its gradient at the point (x, y) is a vector with magnitude and direction, where the gradients in the x direction and the y direction are represented by G x With G y , define the gradient expression as follows:

[0064]

[0065] in,

[0066]

[0067]

[0068] Since Laplace is a differential operator that is used to significantly express the sudden changes in pixels in an image, the Laplace operator is defined as follows based on the image characteristics between the open and closed parts of sorghum grains:

[0069]

[0070] The gradient information after segmentation by the watershed algorithm was calculated using the Laplace operator, and only the opening part information of the opening sorghum was retained for the final statistical analysis.

[0071] This application uses high-speed, high-definition cameras to capture images of sorghum production lines and detect individual, scattered sorghum kernels. This method detects as complete a sorghum kernel as possible, preparing data for detecting whether it is open. The scattered kernels are segmented to remove background interference from other overlapping kernels or the conveyor belt on the production line. The segmented kernels are then classified to determine whether they are open. If they are open, the open portion is further segmented to obtain data on the size of the opening.

[0072] This application realizes the detection of sorghum opening rate during the liquor brewing process and can accurately evaluate the sorghum opening rate.

[0073] like Figure 3 As shown, the sorghum opening rate detection system based on machine vision of the present application includes: an acquisition unit 201, used to acquire images of moving sorghum on a conveyor belt; a first detection unit 202, used to detect scattered sorghum based on a convolutional neural network method; a removal unit 203, used to remove overlapping areas of the detected sorghum grains based on a machine vision method; a classification unit 204, used to classify the openings of the sorghum grains after removing the overlapping areas based on a convolutional neural network method; and a second detection unit 205, used to detect the opening size of the sorghum grains based on a machine vision method.

[0074] In this application, the embodiment of the sorghum opening rate detection system based on machine vision is basically similar to the embodiment of the sorghum opening rate detection method based on machine vision. For relevant details, please refer to the introduction of the embodiment of the sorghum opening rate detection method based on machine vision.

[0075] Those skilled in the art will clearly understand that the technical solutions of the embodiments of the present invention can be implemented with the help of software and / or hardware. "Unit" and "module" in this specification refer to software and / or hardware that can independently perform or cooperate with other components to perform specific functions, where the hardware can be, for example, an FPGA (Field-Programmable Gate Array) or an IC (Integrated Circuit).

[0076] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer-readable storage medium implements the steps of the above-mentioned machine vision-based sorghum aperture ratio detection method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), or any other type of medium or device suitable for storing instructions and / or data.

[0077] Figure 4 This is a schematic diagram of the structure of the computer system according to the embodiment of the present application. Figure 4 As shown, the computer system of the present application includes a processor 401, a memory 402, an input device 403, and an output device 404. The processor 401, the memory 402, the input device 403, and the output device 404 may be connected via a bus 405 or other means. The memory 402 stores a computer program that can be run on the processor 401. When the processor 401 executes the program, the steps of the above-mentioned machine vision-based sorghum opening rate detection method are implemented.

[0078] Input device 403 may be, for example, a touch screen, a keypad, a mouse, a trackpad, a touch pad, a pointer, one or more mouse buttons, a trackball, a joystick, or the like. Output device 404 may include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor). Display devices may include, but are not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, plasma displays, and touch screens.

[0079] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0080] In each embodiment of the present invention, all functional units can be integrated into one processing unit, or each unit can be separately used as a unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0081] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for detecting the opening rate of sorghum based on machine vision, characterized in that: include: Step S1, collecting images of moving sorghum on a conveyor belt; Step S2, detecting scattered sorghum based on a convolutional neural network method; Step S3, removing overlapping areas of the detected sorghum grains based on a machine vision method; Step S4, performing opening classification on the sorghum grains after removing the overlapping areas based on a convolutional neural network method; Step S5, detecting the opening size of sorghum grains based on a machine vision method, comprising: The watershed algorithm is used to sort the gray level of each pixel from low to high by taking advantage of the difference between the open area and the unopen area of ​​sorghum; In the process of flooding from low to high, each local minimum value is judged and marked using a first-in-first-out structure to obtain a marked gradient image; Use the custom Laplace operator to calculate the opening part; Using the adaptive segmentation threshold obtained in step S3, denoising is performed on the opening portion to obtain opening data; Step S5 includes: Assume that there is a matrix function f(x,y) containing a single sorghum image, and its gradient at point (x,y) is a vector with magnitude and direction, where the gradients in the x and y directions are represented by G x With G y , define the gradient expression as follows: in, Since Laplace is a differential operator that is used to significantly express the sudden changes in pixels in an image, the Laplace operator is defined as follows based on the image characteristics between the open and closed parts of sorghum grains: The gradient information after segmentation by the watershed algorithm was calculated using the Laplace operator, and only the opening part information of the opening sorghum was retained for the final statistical analysis.

2. The method for detecting sorghum opening rate based on machine vision according to claim 1, characterized in that: The step S1 comprises: High-definition image data is acquired through a high-speed camera, and the RGB image in the original data is converted to grayscale while retaining the original RGB image. The grayscale image is used for scattered sorghum grain detection.

3. The method for detecting sorghum opening rate based on machine vision according to claim 2, characterized in that: The step S2 comprises: A sorghum scattered grain dataset was created, which includes image data of multiple overlapping scattered grains, single overlapping scattered grains, and blurred scattered grains. Each image contains a scene category label. Design a convolutional neural network model, using the Resnet-50 structure as the skeleton model; Training convolutional neural networks; The trained network model is called to detect scattered particles.

4. The method for detecting sorghum opening rate based on machine vision according to any one of claims 1 to 3, characterized in that: The step S3 comprises: The image containing sorghum grains is divided into four equal parts, and then a Gaussian function with a radius of 5 is used to calculate the average brightness of each equally divided window to obtain the segmentation threshold of each window. The original image is segmented using this threshold to obtain a binary image containing sorghum grains to remove the background interference of the conveyor belt and the overlapping sorghum grain areas.

5. The method for detecting sorghum opening rate based on machine vision according to any one of claims 1 to 3, characterized in that: The step S4 comprises: The classic classification network MobileNet is used to perform binary classification on the segmented sorghum grain images, and the output results are opened and unopened.

6. A sorghum opening rate detection system based on machine vision, characterized in that: include: a collection unit for collecting images of moving sorghum on a conveyor belt; a first detection unit, configured to detect scattered sorghum using a convolutional neural network-based method; a removal unit, used for removing overlapping areas of detected sorghum grains based on a machine vision method; a classification unit for classifying the openings of sorghum grains after removing overlapping areas based on a convolutional neural network method; The second detection unit is used to detect the opening size of the sorghum grains based on a machine vision method; The watershed algorithm is used to sort the gray level of each pixel from low to high by taking advantage of the difference between the open area and the unopen area of ​​sorghum; In the process of flooding from low to high, each local minimum value is judged and marked using a first-in-first-out structure to obtain a marked gradient image; Use the custom Laplace operator to calculate the opening part; Using the adaptive segmentation threshold obtained in step S3, denoising is performed on the opening portion to obtain opening data; Assume that there is a matrix function f(x,y) containing a single sorghum image, and its gradient at point (x,y) is a vector with magnitude and direction, where the gradients in the x and y directions are represented by G x With G y , define the gradient expression as follows: in, Since Laplace is a differential operator that is used to significantly express the sudden changes in pixels in an image, the Laplace operator is defined as follows based on the image characteristics between the open and closed parts of sorghum grains: The gradient information after segmentation by the watershed algorithm was calculated using the Laplace operator, and only the opening part information of the opening sorghum was retained for the final statistical analysis.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 5 are implemented.

Images