A method for obtaining wheat posture information based on deep learning and image algorithm

By improving the combination of YOLOv5 network and traditional image processing, the problem of insufficient accuracy and anti-interference ability in the acquisition of seed pose information is solved, and fast and accurate acquisition of seed pose information is achieved, which is suitable for seed absorption operations of robotic arms.

CN118115587BActive Publication Date: 2024-11-12ZHEJIANG UNIV
View PDF 16 Cites 0 Cited by

Patent Information

Application Number
CN202410220380.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-28
Publication Date
2024-11-12
Estimated Expiration
2044-02-28

AI Technical Summary

Technical Problem

When obtaining seed pose information in the prior art, deep learning methods have fast recognition speed but insufficient accuracy and anti-interference ability. Although traditional image processing algorithms can quickly obtain object contour information, their accuracy and anti-interference ability are insufficient. It is difficult to meet the accuracy and real-time requirements for obtaining seed position information.

Method used

Combining the YOLOv5 deep learning algorithm and traditional image processing, by improving the backbone and neck structure of the YOLOv5 network, the ECA attention mechanism and SEEDLU activation function are increased, and combined with data augmentation and feature fusion, the embryo direction angle and projection area of ​​wheat seeds are calculated. and center of mass coordinates to obtain the position information of wheat seeds.

Benefits of technology

Improve detection speed and accuracy, enhance the robustness of the model, solve the problem of insufficient accuracy and anti-interference ability of a single method, and realize the accuracy and real-time accuracy of quickly obtaining wheat seed position information, suitable for mechanical arms seed absorption operation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118115587B_ABST
    Figure CN118115587B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for obtaining wheat posture information based on deep learning and image algorithm. The present invention comprises the following steps: first, constructing a wheat seed detection model; then, using training data to train the wheat seed detection model; according to the wheat seed detection model obtained by training, using camera real-time image data as network input, the network outputs corner point coordinates, classification, and confidence; then, calculating seed length and width information according to corner point coordinates, and calculating wheat seed embryo angle based on wheat seed angle algorithm; finally, processing the camera real-time image data, obtaining a single seed area, calculating the projection surface area and centroid coordinates, and thus obtaining seed posture information. The present invention solves the problem that traditional image detection methods and deep learning detection methods cannot meet the accuracy and real-time requirements when identifying seed angles, lengths, widths, and centroids.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a method for acquiring seed posture information in the technical fields of computer vision, deep learning and image processing, and in particular to a method for acquiring wheat posture information based on deep learning and traditional image fusion processing. Background Art

[0002] During the operation of the seed slicing robot, it is necessary to obtain the seed's position information for the robot's automated operation of agricultural materials, such as the seed's centroid coordinates to determine the grasping position and the seed embryo's orientation angle to prevent the sampling device from damaging the embryo and destroying seed activity.

[0003] At present, the image recognition results of deep learning are mainly divided into rectangular box target recognition represented by YOLO and instance segmentation represented by mask-R-CNN. Rectangular box target recognition has a high recognition speed, but ignores the object contour information. Instance segmentation can completely retain the object contour information, but the recognition computing power is high and the recognition real-time performance is poor. Traditional image processing algorithms can quickly obtain object contour information, but their accuracy and anti-interference ability are insufficient. Therefore, there is an urgent need for a method to quickly obtain seed pose information while maintaining high accuracy and reducing the algorithm's demand for computing power. Summary of the invention

[0004] In order to solve the problems existing in the background technology, the present invention provides a method for acquiring wheat pose information based on deep learning and image algorithm, which combines the high precision of the YOLOv5 deep learning algorithm and the high acquisition ability of traditional image processing for complex information to improve the efficiency and accuracy of the object's pose and phenotypic information.

[0005] The technical solution adopted by the present invention is:

[0006] 1. A method for obtaining wheat posture information based on deep learning and image algorithm

[0007] S1: Construction of wheat seed detection model;

[0008] S2: train the wheat seed detection model using the wheat seed dataset to obtain a trained wheat seed detection model;

[0009] S3: The real-time image data of the camera is used as the input of the trained wheat seed detection model, and the model outputs the corresponding wheat seed detection results;

[0010] S4: Calculate wheat seed posture information according to the wheat seed detection results.

[0011] In S1, the wheat seed detection model is obtained by improving the YOLOv5 network, specifically:

[0012] After adding the ECA attention mechanism module between the last C3_X module and the SPPF module in the backbone network of the YOLOv5 network, the improved backbone network is obtained;

[0013] The feature fusion part in the neck network is changed from the FPN+PAN structure to the BiFPN structure to obtain an improved neck network;

[0014] Replace the ReLU activation function of the Conv module in the YOLOv5 network with the SEEDLU nonlinear activation function;

[0015] The formula of the SEEDLU nonlinear activation function is as follows:

[0016]

[0017] Among them, SEEDLU(x) is the value of SEEDLU nonlinear activation function, and x is the input received by the neuron.

[0018] In S2, after data enhancement is performed on the wheat seed data set, an expanded data set is obtained, and the wheat seed detection model is trained using the expanded data set.

[0019] The S4 is specifically:

[0020] According to the wheat seed detection results, the embryo angle of the wheat seeds is calculated; and the projection surface area and center of mass coordinates of the wheat seeds are calculated based on the current camera real-time image data. The wheat seed embryo angle, projection surface area and center of mass coordinates constitute the wheat seed posture information.

[0021] In S4, the embryo angle of the wheat seeds is calculated according to the wheat seed detection result, specifically:

[0022] Firstly, all the detection targets of seed categories in the wheat seed detection results are put into the seed set to be selected and the detection targets of seed categories are sorted according to the confidence values; then the seed category detection target with the largest confidence value in the seed set to be selected is selected, and the center point coordinates of the hairy end of the wheat seed and the center point coordinates of the embryo end of the wheat seed are calculated according to the coordinate information of its corner points; if the center point coordinates of the hairy end of the wheat seed are within the detection target of the hairy end category of the wheat seed and the center point coordinates of the embryo end of the wheat seed are within the detection target of the embryo end category of the wheat seed, then the embryo direction angle of the wheat seed corresponding to the current seed category detection target is calculated according to the center point coordinates of the hairy end of the wheat seed and the center point coordinates of the embryo end of the wheat seed; otherwise, the seed category detection targets are selected in turn and the center point coordinates of their embryo end and hairy end are determined until the traversal of the seed category detection targets in the seed set to be selected is completed, and the embryo direction angle of the wheat seed in the real-time image data of the current camera is output.

[0023] In S4, the projection surface area and centroid coordinates of the wheat seeds are calculated based on the current camera real-time image data, specifically:

[0024] B1: Keep the pixel values ​​within the identification frame where the wheat seeds are located in the current camera real-time image data unchanged, and cover the pixel values ​​outside the identification frame to [0, 0, 0] to obtain the mask image pig2;

[0025] B2: After converting each pixel in the mask image pig2 from RGB space to HSV space, the image pig3 in HSV space is obtained. After segmenting the image pig3 in HSV space, the mask pig4 of the seed area is obtained.

[0026] B3: Use the centroid formula to traverse all pixel points in the mask pig4 where the seed is located, and calculate the projection surface area s and centroid coordinates (x0, y0) of the wheat seed.

[0027] 2. A computer device

[0028] The computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.

[0029] 3. A computer-readable storage medium

[0030] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described are implemented.

[0031] IV. A COMPUTER PROGRAM PRODUCT

[0032] The computer program product comprises a computer program / instructions which, when executed by a processor, implement the steps of the method.

[0033] The beneficial effects of the present invention are:

[0034] 1. The wheat seed detection model based on YOLOV5 in the present invention improves the detection speed and accuracy of wheat seeds and enhances the robustness of the model.

[0035] 2. The method for acquiring wheat posture information based on deep learning and image algorithm in the present invention solves the problem that the single traditional image detection method has insufficient accuracy and anti-interference ability, and the deep learning detection method cannot meet the requirements of both accuracy and real-time when identifying seed angles, lengths, widths and centroids. With the help of deep learning and traditional image fusion processing, deep learning is used to extract the target, potential interference items in the image are eliminated, and traditional image detection methods are used to further extract features from the processed image, thereby quickly acquiring phenotypic information while ensuring the accuracy of the results.

[0036] 3. The method for acquiring wheat posture information based on deep learning and image algorithm in the present invention can obtain the center of mass coordinates which can provide a reference for the suction of the negative pressure suction nozzle of the robot arm. Compared with the method of directly using the center point of the rectangular recognition frame as the suction target, the use of center of mass coordinates for suction can make the force during the seed holding process more balanced, and it is not easy to deviate or fall during the process. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 : is a step diagram of the method for obtaining seed pose information based on YOLO-SEED in this embodiment;

[0038] Figure 2 : is a structural diagram of a YOLO-SEED neural network in the prior art in this embodiment;

[0039] Figure 3 This is the staged output effect in the image processing flow of the seed pose information acquisition method in this embodiment;

[0040] Figure 4 Graph showing the test results of the YOLO-SEED neural network test set in this embodiment. DETAILED DESCRIPTION

[0041] In order to enable those skilled in the art to better understand the technical solution of the present invention, the method for obtaining wheat posture information based on deep learning and image algorithm provided by the present invention is described in detail below with reference to the accompanying drawings.

[0042] The present invention provides a method for acquiring wheat posture information based on deep learning and image algorithms. By combining the high precision of the YOLOv5 deep learning algorithm and the high acquisition capability of traditional image processing for complex information, the efficiency and accuracy of acquiring the posture and phenotypic information of the object are improved.

[0043] In this embodiment, RGB images are collected and annotated using the "Rolabeling" annotation software, and the annotation information includes at least the annotations of the following three targets: the whole wheat seed, the embryo end of the wheat seed, and the villi end of the wheat seed, thereby constructing a wheat seed dataset.

[0044] like Figure 1 As shown, the present invention comprises the following steps:

[0045] S1: Construction of wheat seed detection model;

[0046] Among them, the wheat seed detection model is obtained by improving the YOLOv5 network, such as Figure 2 As shown, specifically:

[0047] In view of the fact that all the detection targets are small and medium-sized features, the Backbone network is changed. An ECA attention mechanism module is added between the last C3_X module and the SPPF module in the backbone network Backbone of the YOLOv5 network, and nine modules are set in the Backbone, namely the P0 module to the P8 module to obtain an improved backbone network; that is, the Backbone P0 / P1 / P3 / P5 layers are conv structures, P2 / P4 / P6 are C3_X structures, and the ECA attention mechanism module is introduced in the P7 layer, so that the network training focuses more on the task target features, and the P8 layer is the SPPF module. Backbone inputs a 640×640×3 color image, and the P2 / P4 / P8 layers of Backbone output 160×160×128, 80×80×256, and 40×40×512 matrices to the Neck network respectively, so that the detection head size is changed from 20×20 (large target), 40×40 (medium target), and 80×80 (small target) to 40×40 (medium target), 80×80 (small target), and 160×160 (ultra-small target), thereby improving the recognition accuracy of seed detail features.

[0048] The feature fusion part in the neck network Neck is changed from the FPN+PAN structure to the more efficient BiFPN feature fusion structure, and an improved neck network is obtained to achieve bidirectional fusion of top-down and bottom-up deep and shallow features, and enhance the transmission of feature information between different network layers;

[0049] Replace the ReLU activation function of the Conv module in the YOLOv5 network with the SEEDLU nonlinear activation function, and then perform batch normalization on the data;

[0050] The head network head has an output dimension of 9+nc+180, where 9 corresponds to (x1, y1, x2, y2, x3, y3, x4, y4, score) of the prediction box, that is, the four corner points of the prediction box and the classification confidence, and a value between 0 and 1 is used to evaluate the accuracy of the recognition box; nc represents the number of original data categories, and 180 represents the number of categories corresponding to the rotation angle; in the embodiment of the present invention, nc=3, which respectively includes wheat seeds, wheat seed heads, and wheat seed tails.

[0051] The network loss consists of four parts: angle loss, classification loss, prediction box loss and confidence loss:

[0052] L=λ1L θ +λ2L cls +λ3L box +λ4L obj

[0053] In the formula, L represents the total loss, L θ Represents the angle loss, L cls represents the classification loss, L box Represents the prediction box loss, L obj represents the confidence loss, λ1, λ2, λ3, and λ4 are the weights corresponding to angle loss, classification loss, prediction box loss, and confidence loss, respectively.

[0054] The formula of SEEDLU nonlinear activation function is as follows:

[0055]

[0056] Among them, SEEDLU(x) is the value of SEEDLU nonlinear activation function, and x is the input received by the neuron.

[0057] The SEEDLU activation function solves the problem that the negative gradient is not activated when passing through the ReLU unit, resulting in the death of the neuron. Because its slope is relatively gentle, it ensures the unilateral inhibition advantage.

[0058] S2: train the wheat seed detection model using the wheat seed dataset to obtain a trained wheat seed detection model;

[0059] In the specific implementation, after data enhancement of the wheat seed dataset, an expanded dataset is obtained, and the wheat seed detection model is trained using the expanded dataset. Specifically: 4 images are used in a group, and spliced ​​by random scaling, random cropping, random arrangement, saturation, brightness adjustment, affine transformation, and perspective transformation to obtain 10 new images, so that the total number of inputs remains unchanged, and the obtained new images are randomly occluded. Then, the expanded dataset is randomly divided into a training set and a validation set in a ratio of 8:2 for the training and validation of the wheat seed detection model.

[0060] S3: The real-time image data of the camera is used as the input of the trained wheat seed detection model, and the model outputs the corresponding wheat seed detection results; the seed detection results include corner point coordinates, classification and confidence.

[0061] S4: Calculate wheat seed posture information according to the wheat seed detection results.

[0062] like Figure 3As shown, S4 is specifically:

[0063] According to the wheat seed detection results, the embryo angle of the wheat seeds is calculated; and the projection surface area and center of mass coordinates of the wheat seeds are calculated based on the current camera real-time image data. The wheat seed embryo angle, projection surface area and center of mass coordinates constitute the wheat seed posture information.

[0064] In S4, according to the wheat seed detection results, the wheat seed angle algorithm is used to calculate the embryo angle of the wheat seed, specifically:

[0065] First, all the detection targets of the seed categories in the wheat seed detection results are put into the candidate seed set and the detection targets of the seed categories are sorted according to the confidence value score; then the seed category detection target with the largest confidence value in the candidate seed set is selected, and the center point coordinates of the wheat seed villi end and the center point coordinates of the wheat seed embryo end are calculated according to their corner point coordinate information; if the center point coordinates of the wheat seed villi end are within the detection target of the wheat seed villi end category and the center point coordinates of the wheat seed embryo end are within the detection target of the wheat seed embryo end category, then according to the center point coordinates of the wheat seed villi end (x h ,y h ) and the coordinates of the center point of the wheat seed embryo end (x t ,y t ) Calculate the embryo angle of the wheat seed corresponding to the current seed category detection target. The embryo angle of the wheat seed = atan2(x t -x h ,y t -y h ); otherwise, the seed category detection targets are selected in turn and the coordinates of the center points of the embryo end and the villi end are determined until the traversal of the seed category detection targets in the candidate seed set is completed, and the embryo angle of the wheat seeds in the current camera real-time image data is output. If there is still no seed category detection target that meets the conditions after the traversal is completed, that is, the embryo angle of all wheat seeds cannot be calculated, then the seeds in the current camera real-time image data do not meet the capture conditions, and the detection and posture calculation of the next camera real-time image data are performed.

[0066] Specifically:

[0067] A1: The wheat seed angle algorithm first traverses the category information nc of all detected targets in the picture, stores the coordinate information of the corner points of the wheat seed category into the dictionary list seed, and uses the Hill sort algorithm to sort the dictionary list seed from high to low according to the confidence score value;

[0068] A2: Calculate the coordinates of the center point of the recognition frame according to the coordinates of the corner points, store the coordinate information of the center point of the hair end of the wheat seed in the dictionary list head, and store the coordinate information of the center point of the embryo end of the wheat seed in the dictionary list tail;

[0069] A3: Select the seed with the highest confidence in the dictionary list seed, read its corner point coordinate information, traverse the dictionary list head and dictionary list tail, and calculate whether there is a center point falling within the rectangular box formed by the corner point coordinates of the recognition box. If no center point falls within the rectangular box formed by the corner point coordinates, the count of the corresponding dictionary list head or dictionary list tail is 0; if a center point falls within the rectangular box formed by the corner point coordinates, record the corresponding coordinates (x h ,y h ) or (x t ,y t ), the count of the corresponding dictionary list head or dictionary list tail + 1;

[0070] A4: After the traversal of the dictionary list head and the dictionary list tail is completed, if the count of the dictionary list head or the dictionary list tail = 1, it means that the selected seed meets the angle discrimination requirements, and the seed angle vector (x t -x h ,y t -y h ), use atan2(x t -x h ,y t -y h ) function calculates the vector pointing angle, which is the angle from the embryo end to the villus end; if the count of the dictionary list head or the dictionary list tail is ≠ 1, it means that the selected seed does not meet the angle discrimination requirement, then select the seed with the second highest confidence in the dictionary list seed and repeat the above steps.

[0071] In S4, the projection surface area and centroid coordinates of the wheat seeds are calculated based on the current camera real-time image data, specifically:

[0072] B1: Read the latest single-frame image data pig1 from the camera's video stream and use it as the camera's real-time image data; keep the pixel values ​​within the identification frame where the wheat seeds (i.e., the wheat seeds whose embryo angle can be calculated) in the current camera's real-time image data are located unchanged, and overwrite the pixel values ​​outside the identification frame to [0, 0, 0] to obtain a mask image pig2. At this time, all areas of the image except the identification frame are filled with black;

[0073] B2: After converting each pixel in the mask image pig2 from the RGB space to the HSV space, the image pig3 in the HSV space is obtained. After segmenting the image pig3 in the HSV space using the HSV color segmentation algorithm, the mask pig4 of the seed area is obtained.

[0074] B3: Use the centroid formula to traverse all pixel points in the mask pig4 where the seed is located, and calculate the projection surface area s and centroid coordinates (x0, y0) of the wheat seed.

[0075] The formula for calculating the centroid coordinates is:

[0076]

[0077]

[0078]

[0079] Among them, x0, y0 are the two values ​​of the centroid coordinates, f(x i ,y j ) is the seed area mask pig4 at point (x i ,y j ), m and n are the resolutions of a single frame image in the directions of the two pixel coordinate axes.

[0080] Experimental results verification:

[0081] Experimental operating environment: CPU is Intel(R) i5-12600K, GPU is NVIDIAGeForce RTX 3060, memory is 16G, operating system is Ubuntu16.04, Cuda11.3 version library file is installed, development language is Python, Pytorch framework.

[0082] like Figure 4 As shown in the figure, the detection accuracy and results of the wheat seed detection model for small-sized seed targets are tested. The wheat seed detection model has high detection accuracy and robustness for multi-pose and multi-category wheat seeds in the recognition scene.

[0083] Table 1 is a comparison table of the detection results of the present invention and the traditional image processing method

[0084]

[0085] As shown in Table 1, the accuracy of the seed head and tail orientation discrimination of the present invention is >98%, the average angle error is <3°, and the length and width error variation coefficient is <0.3. Through testing, when the graphics card is 3060 and the input image is 640×640, the detection speed of the model reaches 112.8FPS.

[0086] It can be seen that the present invention has a significant improvement in detection performance compared to traditional image recognition, can achieve accurate phenotypic information acquisition, and at the same time has a lightweight network structure, which can greatly reduce the requirements of the detection model for hardware storage capacity and computing power.

[0087] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for obtaining wheat posture information based on deep learning and image algorithm, characterized in that: The following steps are involved: S1: Construction of wheat seed detection model; S2: train the wheat seed detection model using the wheat seed dataset to obtain a trained wheat seed detection model; S3: The real-time image data of the camera is used as the input of the trained wheat seed detection model, and the model outputs the corresponding wheat seed detection results; S4: Calculate wheat seed posture information according to the wheat seed detection results; In S1, the wheat seed detection model is obtained by improving the YOLOv5 network, specifically: After adding the ECA attention mechanism module between the last C3_X module and the SPPF module in the backbone network of the YOLOv5 network, the improved backbone network is obtained; The feature fusion part in the neck network is changed from the FPN+PAN structure to the BiFPN structure to obtain an improved neck network; Replace the ReLU activation function of the Conv module in the YOLOv5 network with the SEEDLU nonlinear activation function; The formula of the SEEDLU nonlinear activation function is as follows: Among them, SEEDLU(x) is the value of SEEDLU nonlinear activation function, and x is the input received by the neuron; The S4 is specifically: According to the wheat seed detection results, the embryo angle of the wheat seed is calculated; and the projection surface area and centroid coordinates of the wheat seed are calculated based on the current camera real-time image data, and the wheat seed embryo angle, projection surface area and centroid coordinates constitute the wheat seed posture information; In S4, the embryo angle of the wheat seeds is calculated according to the wheat seed detection result, specifically: First, all the detection targets of the seed categories in the wheat seed detection results are put into the seed set to be selected and the detection targets of the seed categories are sorted according to the confidence values; then the seed category detection target with the largest confidence value in the seed set to be selected is selected, and the center point coordinates of the hairy end of the wheat seed and the center point coordinates of the embryo end of the wheat seed are calculated according to its corner point coordinate information; if the center point coordinates of the hairy end of the wheat seed are within the detection target of the hairy end category of the wheat seed and the center point coordinates of the embryo end of the wheat seed are within the detection target of the embryo end category of the wheat seed, then the embryo angle of the wheat seed corresponding to the current seed category detection target is calculated according to the center point coordinates of the hairy end of the wheat seed and the center point coordinates of the embryo end of the wheat seed; otherwise, the seed category detection targets are selected in turn and the center point coordinates of their embryo end and hairy end are determined until the traversal of the seed category detection targets in the seed set to be selected is completed, and the embryo angle of the wheat seed in the real-time image data of the current camera is output; In S4, the projection surface area and centroid coordinates of the wheat seeds are calculated based on the current camera real-time image data, specifically: B1: Keep the pixel values ​​within the identification frame where the wheat seeds are located in the current camera real-time image data unchanged, and cover the pixel values ​​outside the identification frame to [0, 0, 0] to obtain the mask image pig2; B2: After converting each pixel in the mask image pig2 from RGB space to HSV space, the image pig3 in HSV space is obtained. After segmenting the image pig3 in HSV space, the mask pig4 of the seed area is obtained. B3: Use the centroid formula to traverse all pixel points in the mask pig4 where the seed is located, and calculate the projection surface area s and centroid coordinates (x0, y0) of the wheat seed.

2. According to the method for obtaining wheat posture information based on deep learning and image algorithm in claim 1, it is characterized in that: In S2, after data enhancement is performed on the wheat seed data set, an expanded data set is obtained, and the wheat seed detection model is trained using the expanded data set.

3. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 2 are implemented.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.

5. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 2 are implemented.

Citation Information

Patent Citations

  • Extraction method for mouse video multidimensional characteristic value and video classification method

    CN108664942A

  • Wheat seed classification method based on multi-scale feature extraction

    CN112949725A

  • Plant seed real-time classification detection method based on YOLOv5 improvement

    CN114548208A

  • Bolt looseness detection method and system based on deep learning and storage medium

    CN114742820A

  • Wheat ear detection method based on improved YOLOv5

    CN114973002A