Semantic segmentation-based corn mildew degree online detection system and method

Abstract

The invention belongs to the technical field of agricultural material detection, and particularly relates to a corn mildew degree online detection system and method based on semantic segmentation. The system comprises a material conveying mechanism, a material rotating mechanism, a control mechanism and an image acquisition and detection mechanism, the material conveying mechanism is used for conveying to-be-detected materials; the material rotating mechanism is used for controlling the to-be-detected material to rotate by one circle; the control mechanism is used for detecting whether the to-be-detected material reaches a detection station or not and controlling the material conveying mechanism and the material rotating mechanism to switch working states; and the image acquisition and detection mechanism is used for acquiring an image of the to-be-detected material at an interval of 120 degrees when the to-be-detected material rotates for a circle, inputting the acquired three images of the to-be-detected material into the corn image semantic segmentation network model for online detection of the corn mildew degree, and calculating the grade of the to-be-detected material. The problems that manual operation is large in labor amount, long in consumed time, low in efficiency and the like are solved, and good social benefits and economic benefits are achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of agricultural material detection, and in particular relates to an online detection system and method for corn mildew degree based on semantic segmentation. Background Art

[0002] Corn crops were introduced into my country and widely planted due to their strong adaptability, simple planting methods, high yield and excellent quality. The planting areas are mainly distributed in Southwest, North China and Northeast my country. Its yield has ranked first in China's grain output for several consecutive years. Corn is not only an important food crop for humans, but also widely used in starch production, fermentation processing, oil extraction, animal energy feed and other aspects. It has a wide range of uses and is of great significance to food security and production and life. However, due to the complex planting environment of corn, the weather is rainy and humid during the harvest season, which will cause some corn to mold and deteriorate, and hidden deterioration occurs before harvesting. Since the embryo of corn accounts for almost one-third of the volume of the whole grain, it is rich in nutrients and sweet, with a large soluble sugar content, high respiration intensity, strong hygroscopicity, and a large amount of bacteria. In addition, the original moisture content of newly harvested corn is large and it is difficult to fully dry, resulting in strong physiological activity of corn. In a closed environment, respiration will cause the storage temperature to rise. Under high humidity conditions, mold will grow rapidly. Therefore, the nutrients that decompose and consume grains can easily lead to secondary mycotoxin contamination during drying, storage, and transportation. Corn contaminated by mycotoxins can cause serious harm, so corn mildew detection is particularly important. The existing method for corn mildew detection is mainly manual screening, and the mechanization and automation level of corn screening operations is low. Therefore, there is an urgent need for an online detection system and method for corn mildew based on semantic segmentation to solve the problems of large manual labor, long time consumption, and low efficiency. Summary of the invention

[0003] The purpose of the present invention is to provide an online detection system for the degree of mildew of corn based on semantic segmentation, which is characterized by comprising: a material conveying mechanism, a material rotating mechanism, a control mechanism and an image acquisition and detection mechanism; the material conveying mechanism comprises: a frame and a conveying drive mechanism, the conveying drive mechanism comprises a first stepper motor; the material rotating mechanism comprises: a transmission crawler and a rotating drive mechanism, the rotating drive mechanism comprises a second stepper motor; the control mechanism comprises: an embedded controller and a beam-type photoelectric switch group; the image acquisition and detection mechanism comprises: a detection and calculation device based on semantic segmentation, a camera bracket and an industrial camera;

[0004] The transmission drive mechanism is arranged on the top of the frame, and two packaging side panels are arranged in parallel on the two side walls of the frame; a rotating drive mechanism is fixed between the two packaging side panels, and the rotating drive mechanism drives the transmission crawler to move, and the upper surface of the transmission crawler is in contact with the lower surface of the transmission drive mechanism; a camera bracket is arranged along the center line of the positive projection of the upper surface of the transmission crawler on the top of the frame, and the center line is perpendicular to the moving direction of the material to be measured; an industrial camera is arranged at the center of the top of the camera bracket, and a beam-type photoelectric switch group is arranged at two intersections of the center line and the top of the frame, and the beam-type photoelectric switch group and the industrial camera form a detection plane, and the detection plane is perpendicular to the upper surface of the transmission drive mechanism; the first stepper motor, the second stepper motor, and the beam-type photoelectric switch group are respectively connected to the embedded controller; the detection calculation device based on semantic segmentation is respectively connected to the embedded controller and the industrial camera;

[0005] The material conveying mechanism is used to convey the material to be tested; the material rotating mechanism is used to control the material to be tested to rotate one circle; the control mechanism is used to detect whether the material to be tested has reached the testing station, and control the material conveying mechanism and the material rotating mechanism to switch the working state; the image acquisition and detection mechanism is used to capture an image of the material to be tested at an interval of 120° when the material to be tested rotates one circle, and the three acquired images of the material to be tested are input into the corn image semantic segmentation network model for online detection of the degree of corn mildew, and calculate the grade of the material to be tested.

[0006] The transmission drive mechanism also includes: a first synchronous pulley, a first synchronous belt, a first transmission sprocket, a second transmission sprocket, a first transmission chain, a third transmission sprocket, a fourth transmission sprocket, a second transmission chain and a chain tray; the first stepper motor drives the first synchronous pulley to rotate, the first synchronous pulley drives the fourth synchronous pulley through the first synchronous belt, and the fourth synchronous pulley drives the first transmission sprocket to rotate; the first transmission sprocket drives the second transmission sprocket to rotate through the first transmission chain; the first transmission sprocket is coaxial with the third transmission sprocket, and the third transmission sprocket drives the fourth transmission sprocket to rotate through the second transmission chain; a plurality of roller mounting seats are arranged at equal intervals on the outer surfaces of the first transmission chain and the second transmission chain arranged in parallel, and bearing terminals are embedded at both ends of each rotating roller, and the bearing terminals are fixedly connected to the roller mounting seats at relative positions; the spacing between two adjacent rotating rollers is less than the width of a single material to be measured; the chain tray is used to support the first transmission chain and the second transmission chain.

[0007] The rotary drive mechanism also includes: a second synchronous pulley, a second synchronous belt, a third synchronous pulley, a first tensioning roller, a second tensioning roller, and a track tray; the second stepper motor drives the second synchronous pulley to rotate; the second synchronous pulley drives the third synchronous pulley to rotate through the second synchronous belt; the third synchronous pulley is coupled to the first tensioning roller and drives the first tensioning roller to rotate; the first tensioning roller drives the second tensioning roller to rotate through the transmission track; the track tray is used to support the transmission track so that the upper surface of the transmission track contacts the lower surface of the rotating roller.

[0008] The control mechanism also includes: a first stepper driver and a second stepper driver, and the opposing photoelectric switch group includes: a first opposing photoelectric switch and a second opposing photoelectric switch; the first stepper motor is connected to the embedded controller through the first stepper driver, the second stepper motor is connected to the embedded controller through the second stepper driver, and the first opposing photoelectric switch and the second opposing photoelectric switch are connected to the embedded controller.

[0009] A material discharge mechanism is provided at one end of the material conveying mechanism, which is used to enable a single material to be tested to vertically drop through the bottom opening of the discharge funnel to between two adjacent rotating rollers of the material conveying mechanism, and ensure that when the material conveying mechanism stops moving, the next material to be tested cannot fall; a material discharge buffer box is provided at the other end of the material conveying mechanism, which is used to store the tested material after the test is completed;

[0010] The unloading mechanism includes: an unloading funnel and a funnel bracket, the width of the funnel bracket is greater than the width of the material conveying mechanism; the bottom of the unloading funnel vertically passes through the opening on the upper surface of the funnel bracket, the unloading funnel is an inverted trapezoidal funnel, the bottom of the unloading funnel is a rectangular opening, and the distance between the bottom opening of the unloading funnel and the rotating roller is 50 mm.

[0011] The image acquisition and detection mechanism is also provided with a scanner, which is connected to the detection and calculation device based on semantic segmentation to scan the batch identification code of the material to be tested, so as to confirm the type and batch of the material to be tested.

[0012] The image acquisition and detection mechanism is also provided with a display device, which is connected to the detection and calculation device based on semantic segmentation to realize real-time output of detection results. The detection results include: batch, quantity, degree of corn mildew, degree of corn missing kernels and grade of the material to be tested.

[0013] The materials to be tested are classified into:

[0014] When the classification parameter K satisfies: 0≤K<0.1, the classification of the material to be tested is defined as Class I;

[0015] When the classification parameter K satisfies: 0.1≤K<0.3, the material to be tested is defined as Class II;

[0016] When the classification parameter K satisfies: 0.3≤K<0.5, the classification of the material to be tested is defined as Class III;

[0017] When the classification parameter K satisfies: 0.5≤K≤1.0, the material to be tested is classified as Class IV;

[0018] The classification parameter K=Q+M / 2;

[0019] Where Q is the degree of corn kernel deficiency, M is the degree of corn mildew, and both Q and M are percentages.

[0020] The corn image semantic segmentation network model includes: a feature extraction part and an upsampling part, the feature extraction part includes 2 convolutional layers and 1 maximum pooling layer, and the upsampling part includes 1 feature fusion layer and 2 convolutional layers;

[0021] The convolution layer uses a 3x3 convolution kernel and does not include a padding operation to extract local features of the image;

[0022] The maximum pooling layer uses a 2x2 window with a step size of 2, which halves the size of the feature map but retains the most important feature information;

[0023] The feature fusion layer fuses the feature layer obtained by convolution with the feature layer obtained by upsampling, and gradually restores the spatial dimension and detail information of the image;

[0024] Add CBAM attention mechanism between convolutional layer and feature fusion layer;

[0025] The input of the corn image semantic segmentation network model is the surface image of the material to be tested;

[0026] The output of the corn image semantic segmentation network model is to segment the input surface image of the material to be tested to obtain a detection result image; the detection result image uses different colors to represent the moldy area, the missing grain area, the healthy area and the irrelevant background area;

[0027] The surface image of the material to be tested is obtained by splicing three images of the material to be tested collected at intervals of 120°;

[0028] The size of the surface image of the material to be tested is 512×512, the unit is pixel, and the color channel is 3 channels;

[0029] The comprehensive data set used by the corn image semantic segmentation network model includes: an original data set and an enhanced data set; the original data set includes a number of surface images of the material to be tested; the comprehensive data set is an enhanced image obtained by performing enhancement processing based on the surface images of the material to be tested in the original data set; the enhancement processing includes: randomly adding noise and flipping operations to the original surface images of the material to be tested; the noise adding operation is to randomly add salt and pepper noise; the flipping operation is to randomly rotate 45° or 90°;

[0030] The number of images in the comprehensive data set is greater than 500, and the comprehensive data set is divided into a training set and a test set, and the ratio of the training set to the test set is 9:1;

[0031] The corn image semantic segmentation network model uses a training set for feature extraction; the Adam optimizer is selected, the initial learning rate is defined to be less than or equal to 0.0001, the learning decay rate is 0.5 times the original, and the number of iterations is set to 200 times.

[0032] Another object of the present invention is a detection method of the online detection system for corn mildew degree based on semantic segmentation according to the present invention, characterized in that it comprises the following steps:

[0033] The materials to be tested fall sequentially between the adjacent first rotating rollers 809 and the second rotating rollers 810 on the material conveying mechanism;

[0034] The control mechanism controls the first stepper motor 704 to rotate, and the first stepper motor 704 drives a group of transmission sprockets 706 to rotate through the first synchronous pulley 705, and the first transmission chain 701 then drives the rotating rollers to move at a uniform speed in sequence; the material to be tested is lifted by the adjacent first rotating rollers 809 and second rotating rollers 810 to achieve horizontal movement;

[0035] When the material to be tested passes through the detection plane, the first and second photoelectric switches are triggered, and the control mechanism determines that the material to be tested has reached the detection station, and controls the first stepper motor to stop rotating, and the second stepper motor starts to rotate. Under the action of the second synchronous pulley, the track tensioning roller is driven to move, and the tensioned transmission track rotates accordingly under the action of friction. Under the action of the friction between the upper surface of the transmission track and the lower surfaces of the first rotating roller and the second rotating roller, the first rotating roller and the second rotating roller rotate axially in the same direction. Under the action of friction, the material to be tested begins to rotate 360° axially at the detection station. During this process, the image acquisition and detection mechanism takes a surface image of the material to be tested every 120°, and takes a total of 3 surface images of the material to be tested after rotating 360°;

[0036] The surface image of the material to be tested is input into the corn image semantic segmentation network model for processing, and the three surface images of the material to be tested are merged into a merged image with a size of 512×512, a unit of pixel, and a color channel of 3 channels, and the merged image is segmented to obtain a detection result image; the detection result image uses different colors to represent the moldy area, the missing kernel area, the healthy area, and the irrelevant background area; the image acquisition and detection mechanism calculates the corn missing kernel degree Q and the corn mildew degree M based on the detection result image, calculates the grading parameter K based on the corn missing kernel degree Q and the corn mildew degree M, and calculates the grading of the material to be tested based on the grading parameter K;

[0037] The image acquisition and detection mechanism sends a signal to the control mechanism, and the control mechanism controls the second stepper motor 803 to stop rotating, while the first stepper motor 704 continues to rotate, moving the material to be tested out of the detection station until the next material to be tested triggers the first opposing photoelectric switch 507 and the second opposing photoelectric switch 510 again, and the cycle continues until all materials to be tested have completed detection.

[0038] The beneficial effects of the present invention are:

[0039] The invention discloses an online detection system and method for the degree of mildew of corn based on semantic segmentation. The detection system includes a packaging side plate, a feeding funnel, a funnel bracket, a synchronous pulley, two sets of transmission sprockets, a transmission chain, a first stepper motor, a second stepper motor, an industrial camera, a camera bracket, a fill light belt, two stepper drivers, an Arduino single-chip computer, and a feeding buffer box. The system realizes the automation of the detection of the degree of mildew of corn, has the technical advantages of high efficiency and accuracy, and has good social and economic benefits.

[0040] The present invention discloses a semantic segmentation network model for corn images, which realizes pixel-level segmentation for the moldy, healthy and missing kernel areas of the input corn image, can obtain a higher segmentation accuracy, and the image output after detection is clear and distinct, dark green represents the moldy area, light yellow represents the missing kernel area, dark yellow represents the healthy area, and the black area represents the irrelevant background. Compared with the prior art that only segments the defective area, the present model processes the input image, removes redundant background information, makes the detection result image clearer, and outputs and saves the missing kernel degree of corn at the same time. The present model can detect corn more accurately, and the classification index and basis are more comprehensive.

[0041] The corn image semantic segmentation network model disclosed by the present invention has the characteristics of high efficiency and simple construction, and good results can be obtained without the need for training of a large number of data sets. The first half of the detection model of the present invention is feature extraction, and the second half is upsampling. The feature extraction part consists of a convolution operation and a downsampling operation, and the convolution structure used is unified as a 3x3 convolution kernel. The upsampling part is used to restore the original resolution of the feature map, and is commonly implemented by transposed convolution and interpolation. High gradient descent is used in the process of model training, so that the subsequent learning rate of the training model is in a self-regulating process.

[0042] The detection method disclosed in the present invention uses semantic segmentation technology to replace manual detection of the degree of corn mildew, thereby solving the problems of labor waste and inability to make accurate judgments in manual detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a structural diagram of an online detection system for corn mildew degree based on semantic segmentation of the present invention;

[0044] Figure 2 It is a schematic diagram of the relationship between the subsystems of the online detection system for the degree of mildew of corn based on semantic segmentation of the present invention;

[0045] Figure 3 A schematic diagram of the construction process of the corn image semantic segmentation network model of the present invention;

[0046] Figure 4 This is a schematic diagram of the operation interface of the online detection system for corn mildew degree;

[0047] Figure 5 This is a flow chart of the online detection method for corn mildew degree based on semantic segmentation of the present invention;

[0048] Figure 6 It is a right view of the feeding mechanism in the embodiment;

[0049] Figure 7 It is a front view of the material conveying mechanism in the embodiment;

[0050] Figure 8 This is a front view of the detection station in the embodiment;

[0051] Fig. 9 It is a right view of the image acquisition and detection mechanism in the embodiment;

[0052] Fig.10 It is a top view of the online detection system of corn mildew degree based on semantic segmentation in the embodiment;

[0053] Fig.11 It is a right view of the online detection system of corn mildew degree based on semantic segmentation in the embodiment;

[0054] Fig.12 This is a structural diagram of a network model for semantic segmentation of corn images in an embodiment;

[0055] Fig.13 for Fig.12 The enlarged structure diagram of the CBAM model;

[0056] Among them: 501-industrial camera, 502-unloading funnel, 503-funnel bracket, 504-packaging side plate, 505-rotating roller, 506-photoelectric switch mounting base, 507-first photoelectric switch, 508-unloading buffer box, 509-transmission crawler, 510-second photoelectric switch, 511-camera bracket, 512-fill light belt, 601-fourth synchronous pulley, 701-first transmission chain, 702-roller mounting seat, 703-first synchronous belt, 704-first step Motor, 705-first synchronous pulley, 706-first transmission sprocket, 707 chain tray, 708-track tray, 710-second transmission sprocket, 711-bearing terminal, 801-first tensioning roller, 802-second synchronous belt, 803-second stepping motor, 804-second synchronous pulley, 805-third synchronous pulley, 808-second tensioning roller, 809-first rotating roller, 810-second rotating roller, 901-first fill light belt fixing plate, 907-second fill light belt fixing plate. DETAILED DESCRIPTION

[0057] The present invention provides a system and method for online detection of corn mildew degree based on semantic segmentation, and the present invention is further described in detail below in conjunction with the accompanying drawings.

[0058] like Figure 1 The embodiment of the present invention shown discloses an online detection system for the degree of corn mildew based on semantic segmentation, including: a material conveying mechanism, a material rotating mechanism, a control mechanism, and an image acquisition and detection mechanism;

[0059] like Figure 2 As shown, the material conveying mechanism is used to convey the material to be tested; the material rotating mechanism is used to control the material to be tested to rotate one circle; the control mechanism is used to detect whether the material to be tested has reached the detection station, and control the material conveying mechanism and the material rotating mechanism to switch the working state; the image acquisition and detection mechanism is used to collect an image of the material to be tested at an interval of 120° when the material to be tested rotates one circle, and the three images of the material to be tested are input into the corn image semantic segmentation network model to perform online detection of the degree of corn mildew, and calculate the grade of the material to be tested.

[0060] In this embodiment, the material to be tested is corn, which has a cylindrical axisymmetric shape. Other similar shapes include carrots, cucumbers, long strips of eggplant, and sweet potatoes.

[0061] In this embodiment, the material conveying mechanism is used to convey the material to be tested;

[0062] The material conveying mechanism comprises: a frame and a conveying drive mechanism, wherein the conveying drive mechanism comprises a first stepping motor 704;

[0063] The transmission drive mechanism also includes: a first synchronous pulley 705, a first synchronous belt 703, a first transmission sprocket 706, a second transmission sprocket 710, a first transmission chain 701, a third transmission sprocket, a fourth transmission sprocket, a second transmission chain and a chain tray 707; the first stepper motor 704 drives the first synchronous pulley 705 to rotate, the first synchronous pulley 705 drives the fourth synchronous pulley 601 through the first synchronous belt 703, and the fourth synchronous pulley 601 drives the first transmission sprocket 706 to rotate; the first transmission sprocket 706 drives the second transmission sprocket 710 through the first transmission chain 701 The moving sprocket 710 rotates; the first transmission sprocket 706 is coaxial with the third transmission sprocket, and the third transmission sprocket drives the fourth transmission sprocket to rotate through the second transmission chain; a plurality of roller mounting seats 702 are evenly spaced on the outer surfaces of the first transmission chain 701 and the second transmission chain arranged in parallel, and bearing terminals 711 are embedded at both ends of each rotating roller, and the bearing terminals 711 are fixedly connected to the roller mounting seats 702 at the relative position; the spacing between two adjacent rotating rollers is less than the width of a single material to be tested; the chain tray 707 is used to support the first transmission chain 701 and the second transmission chain.

[0064] In this embodiment, if Figure 7 As shown, the material conveying mechanism is composed of a first stepper motor, a synchronous pulley, two sets of transmission sprockets, two parallel transmission chains and 33 rotating rollers evenly spaced between the two transmission chains. Two adjacent rotating rollers hold up a corn.

[0065] In this embodiment, the chain tray 707 is located directly below the chain, which helps the lower surface of the rotating roller to be in close contact with the upper surface of the transmission track 509.

[0066] In this embodiment, the material rotating mechanism is used to control the material to be tested to rotate one circle;

[0067] The material rotating mechanism includes: a transmission crawler 509 and a rotating driving mechanism, and the rotating driving mechanism includes a second stepping motor 803;

[0068] The rotary drive mechanism also includes: a second synchronous pulley 804, a second synchronous belt 802, a third synchronous pulley 805, a first tensioning roller 801, a second tensioning roller 808, and a track tray 708; the second stepper motor 803 drives the second synchronous pulley 804 to rotate; the second synchronous pulley 804 drives the third synchronous pulley 805 to rotate through the second synchronous belt 802; the third synchronous pulley 805 is coupled to the first tensioning roller 801 and drives the first tensioning roller 801 to rotate; the first tensioning roller 801 drives the second tensioning roller 808 to rotate through the transmission belt 509; the track tray 708 is used to support the transmission belt 509, so that the upper surface of the transmission belt 509 contacts the lower surface of the rotating roller.

[0069] In this embodiment, the transmission crawler 509 is located directly below the first rotating roller 809 and the second rotating roller 810, and the upper surface of the transmission crawler 509 contacts the lower surface of the first rotating roller 809 and the second rotating roller 810. The crawler tray 708 is located directly below the transmission crawler 509, and the upper surface of the crawler tray 708 contacts the lower surface of the transmission crawler 509. The transmission crawler 509 is tensioned by the first tensioning roller 801 and the second tensioning roller 808. The bearing terminals 711 are embedded at both ends of the rotating roller, and the bearing terminals at both ends are sequentially fixed above the two parallel transmission chains through the roller mounting seat 702. In this embodiment, the upper surface of the crawler needs to fit with the lower surface of the rotating roller. When the transmission crawler 509 moves, the rotating roller is driven to rotate by friction, and the crawler tray 708 can hold up the transmission crawler 509 to ensure that the upper surface of the transmission crawler 509 fits with the lower surface of the rotating roller.

[0070] In this embodiment, the control mechanism is used to detect whether the material to be tested has reached the testing station, and control the material conveying mechanism and the material rotating mechanism to switch the working state;

[0071] The control mechanism includes: an embedded controller and a beam-type photoelectric switch group;

[0072] The control mechanism also includes: a first stepper driver and a second stepper driver, and the opposing photoelectric switch group includes: a first opposing photoelectric switch 507 and a second opposing photoelectric switch 510; the first stepper motor 704 is connected to the embedded controller through the first stepper driver, the second stepper motor 803 is connected to the embedded controller through the second stepper driver, and the first opposing photoelectric switch 507 and the second opposing photoelectric switch 510 are connected to the embedded controller.

[0073] In this embodiment, the embedded controller is an Arduino single-chip microcomputer, and the Arduino single-chip microcomputer communicates with the detection computing device based on semantic segmentation of the detection system through serial port technology. The first stepper motor and the second stepper motor are controlled by the first stepper driver and the second stepper driver. The control system is used to control the coordinated movement of the material conveying mechanism, the detection station, the image acquisition mechanism and the detection system, and uses serial port technology for communication. The first beam photoelectric switch 507 and the second beam photoelectric switch 510 are installed on both sides of the detection station. The embedded controller obtains the signal sent by the beam photoelectric switch to detect whether the material to be tested has reached the detection position.

[0074] In this embodiment, the image acquisition and detection mechanism is used to capture an image of the material to be tested at an interval of 120° when the material to be tested rotates one circle, and the three acquired images of the material to be tested are input into the corn image semantic segmentation network model to perform online detection of the degree of corn mildew and calculate the grade of the material to be tested.

[0075] The image acquisition and detection mechanism includes: a detection and calculation device based on semantic segmentation, a camera bracket 511 and an industrial camera 501;

[0076] like Fig.10 and Fig.11 As shown, the transmission drive mechanism is arranged on the top of the frame, and two packaging side plates 504 are arranged in parallel on the two side walls of the frame; a rotating drive mechanism is fixed between the two packaging side plates 504, and the rotating drive mechanism drives the transmission crawler 509 to move, and the upper surface of the transmission crawler 509 contacts the lower surface of the transmission drive mechanism; a camera bracket 511 is arranged along the center line of the positive projection of the upper surface of the transmission crawler 509 on the top of the frame, and the center line is perpendicular to the moving direction of the material to be measured; an industrial camera 501 is arranged at the center of the top of the camera bracket 511, and a beam-type photoelectric switch group is arranged at two intersections of the center line and the top of the frame, and the beam-type photoelectric switch group and the industrial camera 501 form a detection plane, and the detection plane is perpendicular to the upper surface of the transmission drive mechanism; the first stepper motor 704, the second stepper motor 803, and the beam-type photoelectric switch group are respectively connected to the embedded controller; the detection calculation device based on semantic segmentation is respectively connected to the embedded controller and the industrial camera 501;

[0077] In this embodiment, the camera bracket 511 and the industrial camera 501 form an image acquisition mechanism; the camera bracket 511 fixes the industrial camera 501 above the detection station so that the shooting range of the industrial camera 501 can cover the entire detection station; Fig. 9As shown, in the specific implementation process, the industrial camera 501 is fixed to the half position of the middle crossbeam on the top of the camera bracket 511 through the camera fixing base, and the industrial camera 501 is fixed directly above the detection station through the bracket. The industrial camera 501 is also located directly above the middle position of the straight line formed by the first opposite-beam photoelectric switch 507 and the second opposite-beam photoelectric switch 510. The detection plane formed by the first opposite-beam photoelectric switch 507, the second opposite-beam photoelectric switch 510 and the industrial camera 501 is perpendicular to the horizontal direction of the conveying mechanism.

[0078] In this embodiment, the material to be tested is lifted by the adjacent first rotating roller 809 and the second rotating roller 810 to achieve horizontal movement; when the material to be tested passes through the detection plane, the first and second photoelectric switches 507 and 510 are triggered, and the space between the first and second rotating rollers 809 and 810 is defined as the detection station. Figure 8 As shown, the first rotating roller 809 and the second rotating roller 810 on the detection station hold the material to be tested, the upper surface of the transmission track contacts the lower surface of the rotating roller, and the corn rotates at a uniform speed along the axial direction under the friction between the transmission track, the rotating roller and the corn. In this embodiment, the third synchronous pulley 805 is coaxially connected to the first tensioning roller shaft 801, and the first tensioning roller shaft 801 is coaxially fixed to the third synchronous pulley 805 through a convex shaft.

[0079] In this embodiment, the control mechanism controls the state switching of the material conveying mechanism and the material rotating mechanism. After the material to be tested is conveyed to the detection station by the material conveying mechanism, the material rotating mechanism controls the material to be tested to rotate 360° along the axial direction. During the rotation process, the image acquisition and detection mechanism takes a surface image of the material to be tested every 120°, and takes a total of 3 surface images of the material to be tested after rotating 360°. The surface image of the material to be tested is input into the corn image semantic segmentation network model for processing, and the three surface images of the material to be tested are merged into a merged image with a size of 512×512, a unit of pixel, and a color channel of 3 channels, and the merged image is segmented to obtain a detection result image; the detection result image uses different colors to represent the moldy area, the missing kernel area, the healthy area and the irrelevant background area; the image acquisition and detection mechanism calculates the corn missing kernel degree Q and the corn mildew degree M based on the detection result image, calculates the grading parameter K based on the corn missing kernel degree Q and the corn mildew degree M, and calculates the grading of the material to be tested based on the grading parameter K; the control mechanism controls the state switching of the material conveying mechanism and the material rotating mechanism again, stops the rotation of the material to be tested and moves it out of the detection station until the next material to be tested triggers the first opposing photoelectric switch 507 and the second opposing photoelectric switch 510 again, and the cycle is repeated until all the materials to be tested are tested.

[0080] In this embodiment, the detection and calculation device based on semantic segmentation is a computer, the detection and calculation device based on semantic segmentation serves as a server, and the embedded controller serves as a client, and realizes information interaction between terminals through a serial communication protocol. Specifically, it includes: the detection and calculation device is connected to the embedded controller to realize the control of the material conveying mechanism and the detection station to be tested, the detection and calculation device is connected to the image acquisition mechanism to realize the acquisition of image data of the material to be tested; the detection and calculation device is connected to the detection result output device to realize the output of the detection result;

[0081] The image acquisition and detection mechanism calculates and stores the classification of the material to be tested;

[0082] The materials to be tested are classified into:

[0083] When the classification parameter K satisfies: 0≤K<0.1, the classification of the material to be tested is defined as Class I;

[0084] When the classification parameter K satisfies: 0.1≤K<0.3, the material to be tested is defined as Class II;

[0085] When the classification parameter K satisfies: 0.3≤K<0.5, the classification of the material to be tested is defined as Class III;

[0086] When the classification parameter K satisfies: 0.5≤K≤1.0, the material to be tested is defined as Class IV;

[0087] The classification parameter K=Q+M / 2;

[0088] Where Q is the degree of corn kernel deficiency, M is the degree of corn mildew, and both Q and M are percentages.

[0089] The corn image semantic segmentation network model includes: a feature extraction part and an upsampling part, the feature extraction part includes 2 convolutional layers and 1 maximum pooling layer, and the upsampling part includes 1 feature fusion layer and 2 convolutional layers;

[0090] The convolution layer uses a 3x3 convolution kernel and does not include a padding operation to extract local features of the image;

[0091] The maximum pooling layer uses a 2x2 window with a step size of 2, which halves the size of the feature map but retains the most important feature information;

[0092] The feature fusion layer fuses the feature layer obtained by convolution with the feature layer obtained by upsampling, and gradually restores the spatial dimension and detail information of the image;

[0093] Add CBAM attention mechanism between convolutional layer and feature fusion layer;

[0094] The input of the corn image semantic segmentation network model is the surface image of the material to be tested;

[0095] The output of the corn image semantic segmentation network model is to segment the input surface image of the material to be tested to obtain a detection result image; the detection result image uses different colors to represent the moldy area, the missing grain area, the healthy area and the irrelevant background area;

[0096] The surface image of the material to be tested is obtained by splicing three images of the material to be tested collected at intervals of 120°;

[0097] The size of the surface image of the material to be tested is 512×512, the unit is pixel, and the color channel is 3 channels;

[0098] The comprehensive data set used by the corn image semantic segmentation network model includes: an original data set and an enhanced data set; the original data set includes a number of surface images of the material to be tested; the comprehensive data set is an enhanced image obtained by performing enhancement processing based on the surface images of the material to be tested in the original data set; the enhancement processing includes: randomly adding noise and flipping operations to the original surface images of the material to be tested; the noise adding operation is to randomly add salt and pepper noise; the flipping operation is to randomly rotate 45° or 90°;

[0099] The number of images in the comprehensive data set is greater than 500, and the comprehensive data set is divided into a training set and a test set, and the ratio of the training set to the test set is 9:1;

[0100] The corn image semantic segmentation network model uses a training set for feature extraction; the Adam optimizer is selected, the initial learning rate is defined to be less than or equal to 0.0001, the learning decay rate is 0.5 times the original, and the number of iterations is set to 200 times.

[0101] In an optional embodiment, a material discharge mechanism is provided at one end of the material conveying mechanism, which is used to enable a single material to be tested to vertically drop through the bottom opening of the discharge hopper 502 to between two adjacent rotating rollers of the material conveying mechanism, and ensure that when the material conveying mechanism stops moving, the next material to be tested cannot fall; a material discharge buffer box 508 is provided at the other end of the material conveying mechanism for storing the tested material;

[0102] The unloading mechanism includes: an unloading funnel 502 and a funnel bracket 503, the width of the funnel bracket 503 is greater than the width of the material conveying mechanism; the bottom of the unloading funnel 502 vertically passes through the opening on the upper surface of the funnel bracket 503, the unloading funnel 502 is an inverted trapezoidal funnel, the bottom of the unloading funnel 502 is a rectangular opening, and the distance between the bottom opening of the unloading funnel 502 and the rotating roller is 50 mm.

[0103] In this optional embodiment, if Figure 6As shown, the corn feeding mechanism consists of a funnel bracket and an inverted trapezoidal feeding funnel. The funnel bracket is used to support the funnel so that it is suspended above the material conveying mechanism. The inverted trapezoidal funnel is used to rectify the corn. The bottom of the funnel is a rectangle with a size of 60mm×250mm, and the distance from the material conveying mechanism below is 50mm, ensuring that when the conveying device does not move, when one corn falls, the other corn is blocked and cannot fall.

[0104] The unloading mechanism is vertically fixed to one end of the material conveying mechanism, and a single material to be tested falls vertically through the bottom opening of the unloading funnel 502 to between two adjacent rotating rollers 505 of the material conveying mechanism; the bottom opening of the unloading funnel 502 is set to allow only a single corn to fall, and the distance between the bottom opening position and the two rollers of the conveying device is fixed. After the first corn falls between the two rollers, if the conveying device does not move, the next corn cannot fall, and the next corn can only fall between the two rollers after the previous corn moves.

[0105] In this optional embodiment, the material conveying mechanism and the inspection station are both arranged in the packaging side panel, the unloading mechanism is located directly above the front end of the material conveying mechanism, the image acquisition mechanism is located directly above the inspection station, and the unloading buffer box 508 is located at the rear end of the material conveying mechanism.

[0106] In an optional embodiment, the image acquisition mechanism further includes two fill light strips, which are fixed to the bottom of the two side beams on the top of the bracket through fill light strip fixing plates. The fill light strip 512 is fixed to the lower surface of the two side beams above the camera bracket 511 through the first fill light strip fixing plate 901 and the second fill light strip fixing plate 907. The clarity of the captured image is ensured by fill light.

[0107] In an optional embodiment, the image acquisition and detection mechanism is further provided with a scanner, which is connected to a detection and calculation device based on semantic segmentation to scan a batch identification code of the material to be tested, so as to confirm the type and batch of the material to be tested.

[0108] In an optional embodiment, the image acquisition and detection mechanism is also provided with a display device, which is connected to the detection and calculation device based on semantic segmentation to realize real-time output of detection results. The detection results include: batch, quantity, degree of corn mildew, degree of corn missing kernels and grade of the material to be tested.

[0109] In this optional embodiment, the display device is an external display, which is convenient for real-time observation of the test results.

[0110] The semantic segmentation-based online detection system for corn mildew degree aims to provide a platform architecture for the online detection method, and combines mechatronic automatic control to realize automatic unloading, transportation and detection of corn. It also develops functional detection system software that can realize visualization of corn mildew detection, calculation and display of mildew degree, etc.

[0111] In a specific embodiment, after the online detection system for the degree of mildew of corn based on semantic segmentation is powered on, the corn batch identification code is scanned and identified, and the corn is put into the feeding hopper 502. The corn to be detected falls into the space between two adjacent rotating rollers of the material conveying mechanism in turn. The control system controls the first stepper motor 704 to rotate. The first stepper motor 704 drives a group of transmission sprockets 706 to rotate through the first synchronous pulley 705. The first transmission chain 701 then drives the rotating rollers to move at a uniform speed in turn. After the corn to be detected falls from the corn feeding hopper 502 between the first rotating roller 809 and the second rotating roller 810, it reaches the detection station under the action of the transmission sprocket 706, triggering the first photoelectric switch 507 and the second. The control system controls the first stepper motor 704 to stop moving, and the second stepper motor 803 to start rotating. Under the action of the second synchronous pulley 804, the track tensioning roller 801 is driven to move, and the tensioned transmission track 509 rotates accordingly under the action of friction. Under the action of friction between the upper surface of the transmission track 509 and the lower surface of the first rotating roller 809 and the second rotating roller 810, the first rotating roller 809 and the second rotating roller 810 will rotate axially in the same direction. The corn to be detected will also start to rotate 360° axially at the detection station under the action of friction. During this process, a corn surface image is taken every 120°, and a total of 3 corn surface images are taken after rotating 360°. While the control system performs the above control, it interacts with the detection system in real time through the serial port. The detection system controls the industrial camera 501 to start collecting corn surface images and inputs the detection model for detection. After the detection is completed, the control system controls the second stepper motor 803 to stop rotating, and the first stepper motor 704 continues to rotate until the next opposing photoelectric switches 507 and 510 are triggered.

[0112] The detection system controls the industrial camera 501 to collect a surface image every time the corn rotates 120° during the rotation of the corn. During the entire rotation process of the corn, a total of three corn surface images are collected for cutting and splicing as the complete surface image of the corn to be detected, and input into the detection model for detection. After the detection process is completed, the complete surface image and the complete result surface image after detection will be displayed on the detection software and saved, wherein the complete result surface image after detection, the dark yellow area represents the healthy corn area, the green area represents the moldy corn area, and the light yellow area represents the missing corn area. The detection and grading results are saved, and a statistical result table file will be generated in the root directory. The statistical result table file content includes the detected corn batch information, the total pixel area of ​​corn, the degree of corn missing kernels, the degree of corn mildew, and the corn grading level.

[0113] The corn grading standard is determined by two indicators: the degree of corn missing kernels and the degree of corn mildew. The degree of corn missing kernels is defined as Q, the degree of corn mildew is defined as M, and the grading parameter K = (Q + M) / 2, where Q and W are both percentages. K values ​​between 0 and 0.1 are defined as Grade I; K values ​​between 0.1 and 0.3 are defined as Grade II; K values ​​between 0.3 and 0.5 are defined as Grade III; K values ​​between 0.5 and 1 are defined as Grade IV.

[0114] like Figure 5 As shown, another embodiment of the present invention discloses a detection method of the online detection system for corn mildew degree based on semantic segmentation according to the present invention, comprising the following steps:

[0115] The materials to be tested fall sequentially between the adjacent first rotating rollers 809 and the second rotating rollers 810 on the material conveying mechanism;

[0116] The control mechanism controls the first stepper motor 704 to rotate, and the first stepper motor 704 drives a group of transmission sprockets 706 to rotate through the first synchronous pulley 705, and the first transmission chain 701 then drives the rotating rollers to move at a uniform speed in sequence; the material to be tested is lifted by the adjacent first rotating rollers 809 and second rotating rollers 810 to achieve horizontal movement;

[0117] When the material to be tested passes through the detection plane, the first photoelectric switch 507 and the second photoelectric switch 510 are triggered, and the control mechanism determines that the material to be tested has reached the detection station, and controls the first stepper motor 704 to stop rotating, and the second stepper motor 803 starts to rotate. Under the action of the second synchronous pulley 804, the track tensioning roller 801 is driven to move, and the tensioned transmission track 509 rotates accordingly under the action of friction. Under the action of the friction between the upper surface of the transmission track 509 and the lower surfaces of the first rotating roller 809 and the second rotating roller 810, the first rotating roller 809 and the second rotating roller 810 rotate axially in the same direction. Under the action of friction, the material to be tested begins to rotate 360° axially at the detection station. During this process, the image acquisition and detection mechanism takes a surface image of the material to be tested every 120°, and takes a total of 3 surface images of the material to be tested after rotating 360°;

[0118] The surface image of the material to be tested is input into the corn image semantic segmentation network model for processing, and the three surface images of the material to be tested are merged into a merged image with a size of 512×512, a unit of pixel, and a color channel of 3 channels, and the merged image is segmented to obtain a detection result image; the detection result image uses different colors to represent the moldy area, the missing kernel area, the healthy area, and the irrelevant background area; the image acquisition and detection mechanism calculates the corn missing kernel degree Q and the corn mildew degree M based on the detection result image, calculates the grading parameter K based on the corn missing kernel degree Q and the corn mildew degree M, and calculates the grading of the material to be tested based on the grading parameter K;

[0119] The image acquisition and detection mechanism sends a signal to the control mechanism, and the control mechanism controls the second stepper motor 803 to stop rotating, while the first stepper motor 704 continues to rotate, moving the material to be tested out of the detection station until the next material to be tested triggers the first opposing photoelectric switch 507 and the second opposing photoelectric switch 510 again, and the cycle continues until all materials to be tested have completed detection.

[0120] In this embodiment, the detection and calculation device based on semantic segmentation controls the first stepper driver through the embedded controller to drive the first stepper motor 704 to rotate; the first transmission chain 701, a plurality of rotating rollers and the second transmission chain form a conveying plane; under the drive of the first stepper motor 704, a single material to be tested moves in the horizontal direction of the conveying plane under the support of two adjacent rotating rollers;

[0121] The first and second photoelectric switches 507 and 510 are fixed on both sides of the detection station through a photoelectric switch mounting base 506. The first and second photoelectric switches 507 and 510 and the industrial camera 501 form a detection plane, which is perpendicular to the transmission plane.

[0122] When the material to be tested passes through the detection plane, the first photoelectric switch 507 and the second photoelectric switch 510 will start to transmit the detection signal to the detection calculation device through the embedded controller, and the embedded controller will stop the first stepper motor 704 through the first stepper driver, so as to park the single material to be tested within the shooting range of the industrial camera 501;

[0123] When the detection computing device receives the start detection signal, the detection computing device controls the second stepper driver through the embedded controller, so that the second stepper motor 803 connected to the second stepper driver drives the transmission crawler 509 to move in the horizontal direction to drive the rotating roller to rotate axially, and the rotating roller rotates axially to drive the single material to be tested to rotate axially, and the detection computing device controls the industrial camera 501 to shoot the material to be tested to obtain a number of image data; the detection computing device based on semantic segmentation detects the several image data to obtain the detection result and sends an end detection signal; the result output device displays the detection result;

[0124] In a specific embodiment, the detection method of the online detection system for corn mildew degree based on semantic segmentation is specifically implemented as follows:

[0125] Scan the batch identification code of the material to be tested to confirm the type and batch of the material to be tested;

[0126] After the material to be tested is automatically unloaded by the unloading mechanism, the single material to be tested is transported to the testing station by the material conveying mechanism;

[0127] After arriving at the inspection station, the industrial camera 501 starts to collect an image of the material to be inspected rotating at a constant speed along the axial direction every time it rotates 120°, and collects a total of three images of the surface of the material to be inspected, and inputs the collected three images of the surface of the material to be inspected into the image acquisition and inspection mechanism;

[0128] The image acquisition and detection mechanism cuts the three surface images of the material to be tested and splices them into one image along the axial direction, and inputs the image into the corn image semantic segmentation network model as the surface image of the material to be tested, and the corn image semantic segmentation network model performs detection of each area;

[0129] After the test is completed, the complete surface image of the material to be tested, the complete surface image after processing and the test results are displayed on the software interface;

[0130] The image acquisition and testing organization saves and compiles statistics on the testing results and grading results of the materials to be tested according to the grading indicators;

[0131] The material conveying mechanism transports the inspected materials to be inspected out of the inspection station.

[0132] The detection result image uses different colors to represent the moldy area, the missing kernel area, the healthy area and the irrelevant background area; the image acquisition and detection mechanism calculates the corn missing kernel degree Q and the corn mildew degree M based on the detection result image, calculates the grading parameter K based on the corn missing kernel degree Q and the corn mildew degree M, and calculates the grade of the material to be tested based on the grading parameter K;

[0133] In this embodiment, the steps for obtaining the corn kernel deficiency degree Q and the corn mildew degree M are as follows:

[0134] Define the number of corn missing pixels as q; the number of corn moldy pixels as m; the number of corn normal pixels as n; the total number of effective pixels t = q + m + n;

[0135] Traverse the color value of each pixel point of the detection result image; if the color value is equal to the preset corn kernel missing color value, then the value of the corn kernel missing pixel number q is increased by 1; if the color value is equal to the preset corn moldy color value, then the value of the corn moldy pixel number m is increased by 1; if the color value is equal to the preset corn normal color value, then the value of the corn normal pixel number n is increased by 1;

[0136] Calculate the degree of corn kernel deficiency Q = q ÷ t;

[0137] Calculate the degree of corn mildew: M = m÷t;

[0138] In this embodiment, if Figure 4 As shown, the software operation interface is designed using QtDesigner and written in Python to implement corresponding functions. It is designed to run in multiple threads to prevent infinite loop programs from blocking threads. It can realize the functions of counting, displaying and saving the batches, quantity, degree of mildew, degree of missing grains and corn grading of the tested corn.

[0139] The materials to be tested are classified into:

[0140] When the classification parameter K satisfies: 0≤K<0.1, the classification of the material to be tested is defined as Class I;

[0141] When the classification parameter K satisfies: 0.1≤K<0.3, the material to be tested is defined as Class II;

[0142] When the classification parameter K satisfies: 0.3≤K<0.5, the classification of the material to be tested is defined as Class III;

[0143] When the classification parameter K satisfies: 0.5≤K≤1.0, the material to be tested is defined as Class IV;

[0144] The classification parameter K=Q+M / 2;

[0145] Where Q is the degree of corn kernel deficiency, M is the degree of corn mildew, and both Q and M are percentages.

[0146] The corn image semantic segmentation network model includes: a feature extraction part and an upsampling part, the feature extraction part includes 2 convolutional layers and 1 maximum pooling layer, and the upsampling part includes 1 feature fusion layer and 2 convolutional layers;

[0147] The convolution layer uses a 3x3 convolution kernel and does not include a padding operation to extract local features of the image;

[0148] The maximum pooling layer uses a 2x2 window with a step size of 2, which halves the size of the feature map but retains the most important feature information;

[0149] The feature fusion layer fuses the feature layer obtained by convolution with the feature layer obtained by upsampling, and gradually restores the spatial dimension and detail information of the image;

[0150] Add CBAM attention mechanism between convolutional layer and feature fusion layer;

[0151] The input of the corn image semantic segmentation network model is the surface image of the material to be tested;

[0152] The output of the corn image semantic segmentation network model is to segment the input surface image of the material to be tested to obtain a detection result image; the detection result image uses different colors to represent the moldy area, the missing grain area, the healthy area and the irrelevant background area;

[0153] The surface image of the material to be tested is obtained by splicing three images of the material to be tested collected at intervals of 120°;

[0154] The size of the surface image of the material to be tested is 512×512, the unit is pixel, and the color channel is 3 channels;

[0155] The comprehensive data set used by the corn image semantic segmentation network model includes: an original data set and an enhanced data set; the original data set includes a number of surface images of the material to be tested; the comprehensive data set is an enhanced image obtained by performing enhancement processing based on the surface images of the material to be tested in the original data set; the enhancement processing includes: randomly adding noise and flipping operations to the original surface images of the material to be tested; the noise adding operation is to randomly add salt and pepper noise; the flipping operation is to randomly rotate 45° or 90°;

[0156] The number of images in the comprehensive data set is greater than 500, and the comprehensive data set is divided into a training set and a test set, and the ratio of the training set to the test set is 9:1;

[0157] The corn image semantic segmentation network model uses a training set for feature extraction; the Adam optimizer is selected, the initial learning rate is defined to be less than or equal to 0.0001, the learning decay rate is 0.5 times the original, and the number of iterations is set to 200 times.

[0158] In this embodiment, the specific process of constructing the corn image semantic segmentation network model is as follows: Figure 3 shown.

[0159] The corn image semantic segmentation network model includes: a feature extraction part and an upsampling part. The feature extraction part includes 2 convolutional layers and 1 maximum pooling layer, and the upsampling part includes 1 feature fusion layer and 2 convolutional layers. The corn image semantic segmentation network model has the features extracted in the first half and upsampling in the second half. The feature extraction part is composed of convolution operation and downsampling operation, and the convolution structure used is unified as a 3x3 convolution kernel. The upsampling part is used to restore the original resolution of the feature map, which is commonly implemented by transposed convolution and interpolation. High gradient descent is used in the process of model training, so that the subsequent learning rate of the training model is in a self-regulating process. The corn image semantic segmentation network model has the characteristics of simple structure and low construction difficulty, and good results can be achieved without a large amount of data set training.

[0160] In this embodiment, the trained network model is deployed to an online detection system for the degree of mildew of corn based on semantic segmentation. Through a hardware connection with the camera and relying on serial port communication to interact with the control system, the degree of mildew of corn is detected in conjunction with the mechanical structure of the device.

[0161] In this embodiment, if Fig.12 As shown in the figure, the corn image semantic segmentation network model is mainly divided into two parts, the left part is the feature extraction part, and the right part is the upsampling part.

[0162] Convolutional layer (Conv2d): The convolutional layer is used to extract local features of the image. A 3x3 convolution kernel is used, and no padding operation is included (padding = 0), resulting in a reduction in the size of the feature map after each convolution, while the number of channels increases, keeping the spatial dimension unchanged.

[0163] Max Pooling Layer: The max pooling layer uses a 2x2 window with a step size of 2, which halves the size of the feature map but retains the most important feature information.

[0164] Feature fusion layer (Concatenate): The feature fusion layer fuses the feature layer obtained by convolution with the feature layer obtained by upsampling, gradually restoring the spatial dimension and detail information of the image. The feature fusion layer is the key to connecting the two parts of the model, allowing feature information at different levels to interact and fuse, thereby making full use of the rich information in the image.

[0165] In this embodiment, if Fig.13 As shown, a CBAM attention mechanism is added between the convolution layer and the feature fusion layer; the CBAM attention mechanism resamples the input data in terms of both space and channels, and strengthens specific objects. In spatial attention, the output is obtained by multiplying the point-wise matrix between the differentiable weight mask and the convolution feature map, and in channel attention, the feature channels are weighted.

[0166] In this embodiment, the CBAM attention mechanism is added between the convolution layer and the feature fusion layer, which has the following advantages:

[0167] 1. Enhance feature expression capabilities: Through the attention mechanism in both channel and spatial dimensions, the input feature layer is automatically selected, and the features that are useful for the task are enhanced, while unimportant and irrelevant background features are suppressed.

[0168] 2. Improve model performance: Make the model more focused on key features and reduce noise interference, thereby improving the accuracy and robustness of the model.

[0169] 3. Optimize feature fusion: Enhance the feature map before feature fusion to make the fused features more effective and promote the interaction between features at different levels.

[0170] In an optional embodiment, without using the online corn mildew degree detection system based on semantic segmentation to collect corn pictures, the detection calculation device based on semantic segmentation can also directly analyze and detect the archived corn mildew pictures.

[0171] In this embodiment, the image of the material to be tested is a surface image of corn. To collect the surface image of corn, the corn is placed in the detection station, the transmission crawler and the rotating roller at the detection station are used to drive the corn to rotate at a constant speed along the axial direction, and the camera is used to collect the original image of the corn image. The number of various corn images collected is greater than 500, and the size of each image is 512×512, the unit is pixel, and the color channel is 3 channels.

[0172] The original image is processed, and the original data set is randomly added with noise, flipped, etc. in batches by an image processing method to obtain an enhanced data set, and the enhanced data set is merged with the original data set into a comprehensive data set. The enhancement method includes randomly rotating 45° and 90°. The random noise addition operation is to randomly add salt and pepper noise.

[0173] The salt and pepper noise, also known as impulse noise, is a black and white light and dark point noise generated by image sensors, transmission channels, decoding processing, etc. by randomly changing some pixel values.

[0174] The comprehensive data set is divided into a training set and a test set according to a ratio of 9:1.

[0175] During the data set preparation process, the data set was enhanced to improve the accuracy of identifying the moldy corn area; during the operation of the device, a camera was used to collect three images of the corn during its 360° rotation, and each three images were cropped and spliced ​​to form a complete surface image of the corn. The above implementation method can fully detect the degree of mold of each corn in all directions, reduce the misjudgment rate, and improve the credibility of the results.

[0176] The training set is used as the input of the detection model for feature extraction. The network is trained for a total of K rounds, K=200, the input image size is 512×512, a 3-channel RGB color image, the Adam optimizer is selected, the initial learning rate is defined to be no more than 0.0001, the learning decay rate is 0.5 times the original, and the number of iterations is set to 200 times.

[0177] The network weight file corresponding to the model with the best average accuracy value in the training phase is taken as the final model to obtain the corn image semantic segmentation network model.

[0178] In this embodiment, the interval of 120 degrees is because the corn rotates 360° in one circle, and the three pictures collected at an interval of 120° have a small deviation from the overall surface of the corn, and the processing process is fast, which can cooperate with the device to meet the needs of real-time detection; the initial learning rate is defined to be less than or equal to 0.0001, and the learning decay rate is 0.5 times the original rate, in order to make the loss curve of the model converge faster during training and speed up the training process; the number of iterations is set to 200 times, because when the training reaches the 200th time, the loss rate of the model tends to be stable, so the training is stopped.

[0179] In this embodiment, the corn image semantic segmentation network model realizes pixel-level segmentation for the moldy, healthy and missing kernel areas of the input corn image, and can obtain a higher segmentation accuracy; the image output after detection is clear and distinct, dark green represents the moldy area, light yellow represents the missing kernel area, dark yellow represents the healthy area, and the black area represents the irrelevant background. Compared with the prior art that only segments the defective area, the corn image semantic segmentation network model processes the input image, removes redundant background information, makes the detection result image clearer, and outputs and saves the missing kernel degree of corn at the same time, so that the corn image semantic segmentation network model can detect corn more accurately, and the classification indicators and basis are more comprehensive.

[0180] The detection method of the online detection system for the degree of mildew of corn based on semantic segmentation collects images of various types of healthy, missing grains and mildewed corn and marks each area. After the data set is amplified by the data enhancement method, it is divided into a training set and a test set in proportion, and the training model is input for training to obtain the optimal weight, a corn mildew degree detection model is constructed and a detection system is built. At the same time, the control system realizes the coordinated movement of the control device and the detection system through the serial communication protocol to realize the information interaction between each terminal. The detection system realizes automatic unloading and detection of corn materials, scans the code to identify corn batches, displays and stores the detection results, grades corn, and detects local pictures, solves the problems of large labor workload, long time consumption, and low efficiency of manual operations, and has good social and economic benefits.

Claims

1. An online detection system for corn mildew degree based on semantic segmentation, characterized in that: include: Material conveying mechanism, material rotating mechanism, control mechanism and image acquisition and detection mechanism; The material conveying mechanism comprises: a frame and a conveying driving mechanism, wherein the conveying driving mechanism comprises a first stepping motor (704); the material rotating mechanism comprises: a transmission crawler (509) and a rotating driving mechanism, wherein the rotating driving mechanism comprises a second stepping motor (803); the control mechanism comprises: an embedded controller and a beam-type photoelectric switch group; the image acquisition and detection mechanism comprises: a detection and calculation device based on semantic segmentation, a camera bracket (511) and an industrial camera (501); The transmission drive mechanism is arranged on the top of the frame, and two packaging side plates (504) are arranged in parallel on the two side walls of the frame; a rotating drive mechanism is fixed between the two packaging side plates (504), and the rotating drive mechanism drives the transmission crawler (509) to move, and the upper surface of the transmission crawler (509) contacts the lower surface of the transmission drive mechanism; a camera bracket (511) is arranged along the center line of the positive projection of the upper surface of the transmission crawler (509) on the top of the frame, and the center line is perpendicular to the moving direction of the material to be measured; an industrial camera (501) is arranged at the center of the top of the camera bracket (511), and a beam-type photoelectric switch group is arranged at two intersections of the center line and the top of the frame, and the beam-type photoelectric switch group and the industrial camera (501) form a detection plane, and the detection plane is perpendicular to the upper surface of the transmission drive mechanism; the first stepper motor (704), the second stepper motor (803), and the beam-type photoelectric switch group are respectively connected to the embedded controller; the detection calculation device based on semantic segmentation is respectively connected to the embedded controller and the industrial camera (501); The material conveying mechanism is used to convey the material to be tested; the material rotating mechanism is used to control the material to be tested to rotate one circle; the control mechanism is used to detect whether the material to be tested has reached the testing station, and control the material conveying mechanism and the material rotating mechanism to switch the working state; the image acquisition and detection mechanism is used to capture an image of the material to be tested at an interval of 120° when the material to be tested rotates one circle, and the three acquired images of the material to be tested are input into the corn image semantic segmentation network model for online detection of the degree of corn mildew, and calculate the grade of the material to be tested.

2. The online detection system for corn mildew degree based on semantic segmentation according to claim 1 is characterized in that: The transmission drive mechanism further comprises: a first synchronous pulley (705), a first synchronous belt (703), a first transmission sprocket (706), a second transmission sprocket (710), a first transmission chain (701), a third transmission sprocket, a fourth transmission sprocket, a second transmission chain and a chain tray (707); the first stepper motor (704) drives the first synchronous pulley (705) to rotate, the first synchronous pulley (705) drives the fourth synchronous pulley (601) through the first synchronous belt (703), and the fourth synchronous pulley (601) drives the first transmission sprocket (706) to rotate; the first transmission sprocket (706) drives the first transmission chain (701) to rotate. ) drives the second transmission sprocket (710) to rotate; the first transmission sprocket (706) is coaxial with the third transmission sprocket, and the third transmission sprocket drives the fourth transmission sprocket to rotate through the second transmission chain; a plurality of roller mounting seats (702) are arranged at equal intervals on the outer surfaces of the first transmission chain (701) and the second transmission chain arranged in parallel, and bearing terminals (711) are embedded at both ends of each rotating roller, and the bearing terminals (711) are fixedly connected to the roller mounting seats (702) at the relative position; the spacing between two adjacent rotating rollers is less than the width of a single material to be tested; and the chain tray (707) is used to support the first transmission chain (701) and the second transmission chain.

3. The online detection system for corn mildew degree based on semantic segmentation according to claim 1 is characterized in that: The rotary drive mechanism further comprises: a second synchronous pulley (804), a second synchronous belt (802), a third synchronous pulley (805), a first tensioning roller shaft (801), a second tensioning roller shaft (808), and a track tray (708); the second stepping motor (803) drives the second synchronous pulley (804) to rotate; the second synchronous pulley (804) drives the third synchronous pulley (805) to rotate via the second synchronous belt (802); the third synchronous pulley (805) is coupled to the first tensioning roller shaft (801) and drives the first tensioning roller shaft (801) to rotate; the first tensioning roller shaft (801) drives the second tensioning roller shaft (808) to rotate via the transmission track (509); the track tray 708 is used to support the transmission track (509) so that the upper surface of the transmission track (509) contacts the lower surface of the rotating roller.

4. The online detection system for corn mildew degree based on semantic segmentation according to claim 1 is characterized in that: The control mechanism further comprises: a first stepper driver and a second stepper driver; the opposing photoelectric switch group comprises: a first opposing photoelectric switch (507) and a second opposing photoelectric switch (510); the first stepper motor (704) is connected to the embedded controller via the first stepper driver, the second stepper motor (803) is connected to the embedded controller via the second stepper driver, and the first opposing photoelectric switch (507) and the second opposing photoelectric switch (510) are connected to the embedded controller.

5. The online detection system for corn mildew degree based on semantic segmentation according to claim 1 is characterized in that: A material discharge mechanism is provided at one end of the material conveying mechanism, which is used to enable a single material to be tested to vertically drop through the bottom opening of the discharge funnel (502) to between two adjacent rotating rollers of the material conveying mechanism, and ensure that when the material conveying mechanism stops moving, the next material to be tested cannot fall; a material discharge buffer box (508) is provided at the other end of the material conveying mechanism, which is used to store the material to be tested that has completed the test; The material discharge mechanism comprises: a material discharge funnel (502) and a funnel support (503), wherein the width of the funnel support (503) is greater than the width of the material conveying mechanism; the bottom of the material discharge funnel (502) vertically passes through the opening on the upper surface of the funnel support (503), the material discharge funnel (502) is an inverted trapezoidal funnel, the bottom of the material discharge funnel (502) is a rectangular opening, and the distance between the bottom opening of the material discharge funnel (502) and the rotating roller is 50 mm.

6. The online detection system for corn mildew degree based on semantic segmentation according to claim 1, characterized in that: The image acquisition and detection mechanism is also provided with a scanner, which is connected to the detection and calculation device based on semantic segmentation to scan the batch identification code of the material to be tested, so as to confirm the type and batch of the material to be tested.

7. The online detection system for corn mildew degree based on semantic segmentation according to claim 1, characterized in that: The image acquisition and detection mechanism is also provided with a display device, which is connected to the detection and calculation device based on semantic segmentation to realize real-time output of detection results. The detection results include: batch, quantity, degree of corn mildew, degree of corn missing kernels and grade of the material to be tested.

8. The online detection system for corn mildew degree based on semantic segmentation according to claim 1, characterized in that: The materials to be tested are classified into: When the classification parameter K satisfies: 0≤K<0.1, the classification of the material to be tested is defined as Class I; When the classification parameter K satisfies: 0.1≤K<0.3, the material to be tested is defined as Class II; When the classification parameter K satisfies: 0.3≤K<0.5, the classification of the material to be tested is defined as Class III; When the classification parameter K satisfies: 0.5≤K≤1.0, the material to be tested is classified as Class IV; The classification parameter K = (Q + M) / 2; Where Q is the degree of corn kernel deficiency, M is the degree of corn mildew, and both Q and M are percentages.

9. The online detection system for corn mildew degree based on semantic segmentation according to claim 1, characterized in that: The corn image semantic segmentation network model includes: a feature extraction part and an upsampling part, the feature extraction part includes 2 convolutional layers and 1 maximum pooling layer, and the upsampling part includes 1 feature fusion layer and 2 convolutional layers; The convolution layer uses a 3x3 convolution kernel and does not include a padding operation to extract local features of the image; The maximum pooling layer uses a 2x2 window with a step size of 2, which halves the size of the feature map but retains the most important feature information; The feature fusion layer fuses the feature layer obtained by convolution with the feature layer obtained by upsampling, and gradually restores the spatial dimension and detail information of the image; Add CBAM attention mechanism between convolutional layer and feature fusion layer; The input of the corn image semantic segmentation network model is the surface image of the material to be tested; The output of the corn image semantic segmentation network model is to segment the input surface image of the material to be tested to obtain a detection result image; the detection result image uses different colors to represent the moldy area, the missing grain area, the healthy area and the irrelevant background area; The surface image of the material to be tested is obtained by splicing three images of the material to be tested collected at intervals of 120°; The size of the surface image of the material to be tested is 512×512, the unit is pixel, and the color channel is 3 channels; The comprehensive data set used by the corn image semantic segmentation network model includes: an original data set and an enhanced data set; the original data set includes a number of surface images of the material to be tested; the comprehensive data set is an enhanced image obtained by performing enhancement processing based on the surface images of the material to be tested in the original data set; the enhancement processing includes: randomly adding noise and flipping operations to the original surface images of the material to be tested; the noise adding operation is to randomly add salt and pepper noise; the flipping operation is to randomly rotate 45° or 90°; The number of images in the comprehensive data set is greater than 500, and the comprehensive data set is divided into a training set and a test set, and the ratio of the training set to the test set is 9:1; The corn image semantic segmentation network model uses a training set for feature extraction; the Adam optimizer is selected, the initial learning rate is defined to be less than or equal to 0.0001, the learning decay rate is 0.5 times the original, and the number of iterations is set to 200 times.

10. A detection method for an online detection system for corn mildew degree based on semantic segmentation according to any one of claims 1 to 9, characterized in that: The steps include: The material to be tested falls in sequence between the adjacent first rotating roller (809) and the second rotating roller (810) on the material conveying mechanism; The control mechanism controls the first stepper motor (704) to rotate, and the first stepper motor (704) drives a group of transmission sprockets (706) to rotate through the first synchronous pulley (705), and the first transmission chain (701) then drives the rotating rollers to move at a uniform speed in sequence; the material to be tested is lifted by the adjacent first rotating rollers (809) and second rotating rollers (810) to achieve horizontal movement; When the material to be tested passes through the detection plane, the first and second photoelectric switches (507 and 510) are triggered, and the control mechanism determines that the material to be tested has reached the detection station, and controls the first stepper motor (704) to stop rotating, and the second stepper motor (803) to start rotating. Under the action of the second synchronous pulley (804), the track tensioning roller (801) is driven to move, and the tensioned transmission track (509) rotates accordingly under the action of friction. Under the action of friction between the upper surface of the transmission track (509) and the lower surfaces of the first rotating roller (809) and the second rotating roller (810), the first rotating roller (809) and the second rotating roller (810) rotate axially in the same direction. Under the action of friction, the material to be tested begins to rotate 360° axially at the detection station. During this process, the image acquisition and detection mechanism takes a surface image of the material to be tested every 120°, and takes a total of 3 surface images of the material to be tested after rotating 360°. The surface image of the material to be tested is input into the corn image semantic segmentation network model for processing, and the three surface images of the material to be tested are merged into a merged image with a size of 512×512, a unit of pixel, and a color channel of 3 channels, and the merged image is segmented to obtain a detection result image; the detection result image uses different colors to represent the moldy area, the missing kernel area, the healthy area, and the irrelevant background area; the image acquisition and detection mechanism calculates the corn missing kernel degree Q and the corn mildew degree M based on the detection result image, calculates the grading parameter K based on the corn missing kernel degree Q and the corn mildew degree M, and calculates the grading of the material to be tested based on the grading parameter K; The image acquisition and detection mechanism sends a signal to the control mechanism, and the control mechanism controls the second stepper motor (803) to stop rotating, while the first stepper motor (704) continues to rotate, so as to move the material to be tested out of the detection station until the next material to be tested triggers the first photoelectric switch (507) and the second photoelectric switch (510) again, and the cycle continues until all materials to be tested have been detected.

Images