A soybean plant stem related phenotype automatic acquisition method based on deep learning

Abstract

The present application relates to a kind of soybean plant stalk related phenotypes automatic acquisition method based on deep learning, through eight steps, the deep learning and image processing technology are fused and used, for mature soybean whole plant image in combination with soybean plant morphological characteristics, through the deep learning training target detection model, stem node is identified, training semantic segmentation model extracts stem region, in combination with stem region extraction result optimization stem node detection result, again in combination with image binary processing, design multi-path planning algorithm is respectively realized stem node positioning, sequencing, counting and stem node spacing calculation to the main stem and branch of reconstruction, with maximum inscribed circle method obtains main stem and branch stem thick, realizes the automatic acquisition of mature soybean whole plant stem related phenotypes, provides mass phenotype data for breeder. Overcome the deficiency of present artificial and sensor detection method and image processing method to collect soybean whole plant phenotype data, with the characteristics of objective, accurate, automatic, high efficiency to obtain phenotype data.

Description

TECHNICAL FIELD

[0001] The present application relates to a soybean plant stem related phenotype automatic acquisition method based on deep learning, belonging to the field of agricultural information technology. BACKGROUND

[0002] Soybean is an important oil crop and food crop, and its yield is closely related to human production and life. With the continuous growth of global population, the demand for soybean is increasing. The soybean planting area in China is less than 1 / 20 of the total arable land, resulting in that China's soybean consumption mainly depends on imported soybeans. Therefore, cultivating high-yield soybean seeds has become an important way to solve the problem of insufficient soybean production in China. Using artificial intelligence technology to develop an automatic acquisition algorithm for soybean phenotype can greatly improve the collection efficiency of phenotype data, quickly provide large-scale data for genetic research, and solve the problems of low efficiency, high cost, dependence on professional knowledge, and strong subjectivity of the current manual measurement method. The present technology focuses on the automatic detection of soybean plant stem related phenotypes, combines deep learning technology with image processing technology, and provides a light, fast and accurate automatic detection method for soybean stem related phenotypes, which helps to shorten the breeding process and provides ideas for the automatic detection of related crop phenotypes.

[0003] Currently, there are three main ways to obtain stem related phenotypes from soybean plants: (1) manual method; (2) sensor detection method; (3) image processing method.

[0004] The manual method requires planting or breeding personnel to use vernier calipers and other professional tools to measure and count the number of main stems and branches, stem internode distance, and main stem and branch stem diameter of each plant one by one, which is labor-intensive, low-efficiency, and easily affected by subjective factors. The sensor detection method uses sensors to detect soybean plants in a specific area in real time to record the growth status of the plants, which is easily disturbed by external temperature, humidity and other factors, and has high maintenance cost. The image processing method is a new method currently used in automatic detection of crop phenotypes. It generally uses a camera to take pictures of soybean plants collected by breeding personnel, constructs a phenotype recognition model based on the distinguishing features of different types of phenotypes, and realizes automatic recognition of phenotypes. The effectiveness of the image processing method highly depends on the domain knowledge and rules used when designing the underlying phenotype recognition model, and has low scalability and robustness. SUMMARY

[0005] The present application aims at the deficiencies of the prior art, and provides a soybean plant stem related phenotype automatic acquisition method based on deep learning, which combines deep learning and image processing technology, and uses a deep learning technology to train a target detection model to identify and locate the stem nodes in the soybean plant image affected by occlusion noise, trains a semantic segmentation model to extract the stem regions in the plant, designs a path planning algorithm to extract the main stem and branch stem, measures the distance between the stem nodes, and finally acquires the stem diameters of the main stem and branches by the maximum inscribed circle method.

[0006] The technical scheme of the present application is:

[0007] A soybean plant stem related phenotype automatic acquisition method based on deep learning, which is realized by the following steps:

[0008] 1. Soybean plant image acquisition and preprocessing;

[0009] Soybean breeders acquire the images of mature soybean plants by installing high-definition industrial cameras on a closed black box. The soybean plant morphology involved is diverse, including unbranched soybean plant images, single-branched soybean plant images, and complex branched soybean plant images. After the images are transmitted to a computer for storage, LabelImg and Labelme tools are used to label the data of the obtained image data; the stem node information is labeled by LabelImg, and each stem node target region on the image is marked as node type; the stem region is labeled by Labelme, and the stem edge region on the image is continuously marked as stem type. The labeled image data is enhanced as the training input data set of the deep learning network;

[0010] 2. Construction of a soybean phenotype extraction deep learning network model;

[0011] 2.1) Stem node feature recognition network model;

[0012] The stem node feature recognition network model is constructed by adding a CA attention mechanism to the existing YOLOv5s network. The soybean plant image data set obtained in step 1 is input into the stem node feature recognition network to obtain a complete stem node feature recognition network model. The specific operations are as follows:

[0013] Input the soybean plant image data set Input->Backbone layer->Neck layer->Head layer, and thus the feature extraction and detection operations are completed, and the stem node prediction map of the soybean plant is obtained;

[0014] The input image is resized to 640x640x3 using the resize function as the input of the Backbone layer, and after 6*6 convolution operation, the picture becomes 320x320x64, and after a 3*3 convolution block, the picture size becomes 160x160x128, enters the C3 module and a 3*3 convolution block, and the size becomes 80x80x256, and then after a C3 module, the output 80x80x256 is obtained, and after a 3*3 convolution and a C3 module, it becomes 40x40x512, and then after a 3*3 convolution, it becomes 20x20x1024, and after CA and SPPF layer, the three outputs of the Backbone layer are obtained: the output of the second C3 module from shallow to deep in the network 80x80x256, the output of the second C3 module 40x40x512 and the output of the SPPF layer 20x20x1024. In the Backbone layer, the C3 module is proposed in YOLOv5, aiming to replace the CSP module, and the C3 module is mainly composed of CBS and Bottleneck Block module, and the CBS structure contains convolution layer, batch normalization layer and SiLU activation function; these components work together to enable the C3 module to effectively process and extract features;

[0015] The Bottleneck Block structure is first a 1x1 convolution layer, then a 3x3 convolution layer, and finally added to the initial input through a shortcut, and the CA attention mechanism is fused before the SPPF module;

[0016] The CA attention mechanism simultaneously focuses on position information and channel information, and aggregates features from these two directions, which is suitable for stem node feature extraction; the specific process is that the input of the C3 module is reduced to 1 / reduction times of the original channel number of the input feature map through 1x1 convolution operation, and the reduced channel number is processed by Batch Normalization and ReLU activation function, and two 1x1 convolution operations are performed on the reduced channel number, respectively for generating horizontal and vertical attention distributions, finally, the horizontal and vertical attention distributions are normalized to [0, 1] by Sigmoid function, and the two distributions are used as weights to weight the input feature map, and the output channel number is consistent with the input;

[0017] The output 20x20x1024 of Backbone is taken as the input of the Neck layer, and after 1*1 convolution operation, the output 20x20x512 is obtained, and after downsampling operation, it is concatenated with the output 40x40x512 of Backbone to obtain 40x40x256, and after C3 module and 1*1 convolution operation, the output 40x40x256 is obtained, and after downsampling operation, it is concatenated with the output 80x80x256 of Backbone to obtain the first output 80x80x256 of the Neck layer, and after 3*3 convolution and output 40x40x256, the output is concatenated, and after C3 module, the output is taken as the second output of the Neck layer, and after 3*3 convolution, the output 20x20x512 is obtained, and after concatenation with the first output 20x20x512, the output 20x20x1024 of the Neck layer is obtained after C3 module;

[0018] The output 80x80x256, 40x40x512, and 80x80x256 of the Neck layer after the above processing is input into the Head layer, and the Head layer usually includes a series of Conv2d modules for extracting target position and category information from the feature map; the Conv2d module can include multiple convolution operations for further processing the feature map, and usually has different convolution kernel sizes and channel numbers; these Conv2d modules output the predicted target box coordinates, class confidence, and class information, and the output result classifies and regresses each spatial position to generate a prediction box for target detection;

[0019] Common processing includes applying activation functions (such as sigmoid or softmax) to obtain class probabilities, and applying Anchor-base boundary box regression algorithm to predict the position of the boundary box;

[0020] The soybean image is input into the stem node feature recognition network model to obtain the position of each stem node detection box in the image, which is recorded in array form as node1: [0, x1, y1, w1, h1], node2: [0, x2, y2, w2, h2], node3: [0, x3, y3, w3, h3]…, wherein the first bit is a label bit, 0 represents that the detection box corresponds to a stem node, x i , y i are the horizontal and vertical coordinates of the center point of the stem node detection box, w i , h i are the length and width of the rectangular stem node detection box, respectively;

[0021] 2.2) Stem extraction network model;

[0022] The stem extraction network model is a U 2The U-Net is mainly composed of an encoding layer, a decoding layer, and a saliency map fusion module; 2 The encoder of the U-Net is used to extract feature information of the image, and gradually reduces the size of the feature map; the decoder restores the size of the feature map to the size of the original input image, and fuses the feature information with the feature map of the corresponding layer of the encoder to generate a high-resolution segmentation result; the encoder adopts an improved structure, including a U-Net structure composed of two parallel paths and a connection module with a lightweight attention mechanism, which realizes the reduction of calculation amount while maintaining high performance, especially suitable for processing small targets and details;

[0023] U 2 The decoder of the U-Net uses a skip connection mechanism to connect the feature maps of different layers in the encoder with the feature maps of the corresponding layers in the decoder to preserve more spatial information and semantic information; in addition, the decoder uses deconvolution operation and up-sampling technology to restore the size of the feature map to the size of the original input image;U 2 The U-Net also uses a loss function suitable for image segmentation tasks, usually a cross-entropy loss function, which can measure the difference between the model's prediction results and the true labels, and guide the model to learn how to more accurately perform image segmentation;

[0024] During the training process, the model parameters are adjusted using the stochastic gradient descent optimization algorithm to obtain the minimum loss function value by training on a large-scale labeled image dataset; the training process aims to enable the model to learn how to extract features from the input image and perform pixel-level segmentation;

[0025] Input soybean plant image dataset -> encoder level En1 -> encoder level En2 -> encoder level En3 -> encoder level En4 -> encoder level En5 -> encoder level En6 -> decoder level De5 -> decoder level De4 -> decoder level De3 -> decoder level De2 -> decoder level De1 -> saliency map fusion module, using a 1x1 convolution layer and a sigmoid function to generate the final saliency probability map, thus completing the image segmentation operation and obtaining the soybean image stem segmentation map;

[0026] The U-Net is mainly composed of an encoding layer, a decoding layer, and a saliency map fusion module; 2 The most important part of the U-Net encoder structure is the residual U module (RSU), which has the following specific structure:

[0027] For the residual module, set the input channel number, intermediate channel number, and output channel number;

[0028] 1) 3x3 convolution (input channel, output channel),

[0029] 2) 3x3 convolution (input channel, intermediate channel) 1 / 2 down-sampling,

[0030] 3) 3x3 convolution (intermediate channel, intermediate channel) 1 / 2 down-sampling,

[0031] 4) 3x3 convolution (intermediate channel, intermediate channel) 1 / 2 down-sampling,

[0032] 5) 3x3 convolution (intermediate channel, intermediate channel) 1 / 2 down-sampling,

[0033] 6) 3x3 convolution (intermediate channel, intermediate channel) 1 / 2 down-sampling,

[0034] 7) 3x3 convolution (intermediate channel, intermediate channel),

[0035] 8) 3x3 convolution (intermediate channel, intermediate channel),

[0036] 9) stack 7) and 8) by channel, 3x3 convolution (intermediate channel, intermediate channel) 2x up-sampling,

[0037] 10) stack 6) and 9) by channel, 3x3 convolution (intermediate channel, intermediate channel) 2x up-sampling,

[0038] 11) stack 5) and 10) by channel, 3x3 convolution (intermediate channel, intermediate channel) 2x up-sampling,

[0039] 12) stack 4) and 11) by channel, 3x3 convolution (intermediate channel, intermediate channel) 2x up-sampling,

[0040] 13) stack 4) and 12) by channel, 3x3 convolution (intermediate channel, intermediate channel) 2x up-sampling,

[0041] 14) stack 2) and 13) by channel, 3x3 convolution (intermediate channel, intermediate channel), finally add 14) and 1) by normal addition;

[0042] The structure of the decoder stage and the encoder stage are similar, and in the encoding process, each time to the next layer, the output will also be input to the corresponding decoding layer of the "U" shape; the saliency map fusion module is to effectively fuse the saliency map with the original image to produce an enhanced image, so that the salient region in the image is more prominent or more easily noticed;

[0043] The soybean plant image data set obtained in step 1 is input into the U 2 -Net network for related feature extraction operation, that is, a complete stem extraction network model is obtained;

[0044] In the stem extraction network model, in the input image, the background area of the plant is converted to black except the soybean stem target area, which is equivalent to separating the stem target from the original image as a separate individual, referred to as a stem segmentation image, and superimposing the stem segmentation image and the binary image in step 4 to improve the accuracy of phenotype acquisition compared to the traditional method of directly binary processing the image;

[0045] The stem extraction network model is trained by inputting the soybean image to obtain the class value to which each pixel point in the image belongs. The pixel point value of the background is 0, and the soybean plant stem area is marked with green. The pixel points in the stem area have the same value, which is 2.

[0046] 3. According to the results of the stem node feature recognition network model and the stem extraction network model, a stem node automatic filtering module is constructed.

[0047] 3.1) Through steps 1 and 2, the stem nodes have been identified from the soybean plant image, and the stem region has been segmented from the original image. The image labeled with the stem region is referred to as the stem segmentation image.

[0048] 3.2) Filter the candidate stem node bounding box.

[0049] To solve the problem that the pods in the mature soybean plant image block the stem and cause some pods to be misidentified as stem nodes, a method of using semantic segmentation results to filter misidentified stem nodes is proposed. First, the pixel points of the stem region obtained by semantic segmentation are simplified into two categories: stem and background, with 0 representing background and 2 representing the stem region. The stem segmentation image is stored in the form of a numpy array with only 0 and 2 elements. Second, the detected stem node box information is filtered one by one. If any pixel point within the detection box has a value of 2 in the stem segmentation image, the stem node box is retained. Otherwise, it is considered that the stem node box is not covered by the stem and is an invalid stem node. In this way, the stem node box located in the stem region is filtered out, improving the detection accuracy of the stem node box.

[0050] 4. According to the results of stem node filtering and stem segmentation, a stem reconstruction algorithm is constructed.

[0051] The detection of stem nodes, stem node spacing, and stem thickness requires first identifying the main stem and branches from the soybean plant image, i.e., reconstructing the stem. The present invention proposes that the reconstruction of the main stem and branches is converted into a pathfinding problem based on a pixel map, which integrates the stem node detection results, the stem segmentation image, and the binary image. A multi-path planning algorithm based on A* algorithm is designed to reconstruct the plant stem from the soybean plant image, including the main stem and each branch of the plant. Based on the reconstructed main stem and each branch, the stem node number, stem node spacing, and stem thickness phenotypes are obtained.

[0052] 4.1) Construct a path planning pixel map;

[0053] The original plant image is converted into a binary image, and the stem segmentation image obtained by the stem extraction network is superimposed; the pixel points with a value of 2 in the stem segmentation image are marked as 1, and the pixel points with a value of 0 in the binary soybean plant image and a value of 0 in the stem segmentation image are marked as 0; thus, a pixel map representing the stem region is constructed; the map is the input of the path planning algorithm;

[0054] The path planning algorithm using the superimposed image has good robustness: it will not deviate from the stem region due to a single binary image, and it can compensate for the stem region segmentation model, and in the image where the stem region segmentation is insufficient to disconnect the local stem region, the superimposed binary image can effectively improve the connectivity of the stem region, so that the path planning algorithm can still successfully plan the path;

[0055] 4.2) Combine the stem node detection box information to determine the starting point and ending point of the path planning, construct a multi-stem path planning algorithm, and reconstruct the main stem and branches of the plant;

[0056] In view of the problem that the number of pixel points in the stem region of the soybean plant leads to low efficiency of path planning, the present application proposes: combining the stem node detection box information obtained by the stem node feature extraction network, and using the path planning algorithm to reconstruct the main stem and branches of the plant;

[0057] The process of the multi-stem path planning algorithm is as follows: first, the topmost and bottommost stem node detection boxes in the plant image are taken as the starting point and ending point of the main stem path respectively; generally, the top of the main stem is the highest point of the entire plant, so the topmost stem node detection box is taken as the starting point of the main stem path; a path planning algorithm is executed to obtain the path between the starting point and the ending point, and the obtained path is taken as the main stem path; then, the image is scanned from top to bottom, and the topmost stem node detection box of each branch stem is detected in turn, which is taken as a new starting point of path planning, and a path planning algorithm is executed to obtain the path between the new starting point and the ending point, and the obtained path is taken as a branch path; the intersection point of the branch path and the main stem path is calculated, and the point is recorded as the ending point of the branch path, and the ending point of the branch path is updated; the above process is repeated, and each stem node detection box in the plant is scanned from top to bottom until all stem node detection boxes belong to a branch; only one main stem path can be obtained in a plant, and multiple branch paths can be obtained; the obtained main stem path is taken as the main stem of the plant, and the obtained multiple branch paths are taken as the branches of the plant;

[0058] The A* algorithm is a combination of heuristic search and depth-first search. It defines the starting and ending points of a path and uses heuristic information to plan the path with the lowest overall expected cost. The basis for the A* algorithm's path planning is shown in Formula 1:

[0059]

[0060] In formula (1), This indicates the nodes passed from the starting point. The estimated cost of reaching the destination. Indicates the distance from the starting point to the node. The actual cost of the optimal path, Indicates from node The estimated cost of the optimal path to the destination is selected from all nodes to be evaluated. The node with the smallest value is selected as the next node to be expanded.

[0061] The specific process of the multi-stalk path planning algorithm is as follows:

[0062] Because the plants were positioned correctly during the shooting, the first stem node detection box obtained by scanning downwards from the top of the image was designated as the starting point of the main stem, and the stem node detection box at the bottom of the plant in the image was designated as the ending point of the main stem. That is, the stem node detection boxes were sorted in ascending order of their ordinates, and the center points [x1, y_min] and [x2, y_max] of the stem node detection boxes containing the minimum and maximum ordinate values ​​were taken as the starting and ending points of the path planning algorithm.

[0063] Before starting path planning, an initial flag is added to the representation of all stem node detection boxes: node1: [x1,y1,w1,h1,0], node2: [x2,y2,w2,h2,0], node3: [x3,y3,w3,h3,0]..., indicating that the stem node detection box has not been visited. After the path planning algorithm has executed one round and obtained the main stem path, the flag of all stem node detection boxes on the main stem is modified to 1, indicating that the stem node detection box has been visited by the main stem path. That is, at this time, the information of the stem node detection boxes on the main stem is node1: [x1,y1,w1,h1,1], node2: [x2,y2,w2,h2,1], node3: [x3,y3,w3,h3,1]...;

[0064] Scan the image downwards, take the first unvisited stem node detection box as the starting point of the first branch, and perform path planning again with the endpoint to obtain a branch path; record the intersection point of this branch path and the main stem path, and take this intersection point as the endpoint of the branch path; thus, the path of the first branch is obtained, and modify the flag bit of all stem node detection boxes on this branch path to 2, indicating that the stem node detection box has been visited and belongs to the first branch;

[0065] The acquisition process of the remaining branch paths is similar to that of the first branch path, and the flag value of the stem node detection frame is sequentially incremented, such as the flag value of the stem node detection frame on the second branch being 3; the above process is repeated until the flag value of all stem node detection frames is not 0, that is, all stem node detection frames have been accessed and belong to the main stem or a branch;

[0066] After completion, the stem nodes on the main stem and branches are recorded according to morphological characteristics, that is, from the bottom end of the stem to the top end of the stem, that is:

[0067] The stem node detection frame node1 on the main stem: [x1, y1, w1, h1, 1], node2: [x2, y2, w2, h2, 1], node3: [x3, y3, w3, h3, 1]…;

[0068] The stem node detection frame node1 on the first branch: [x1, y1, w1, h1, 2], node2: [x2, y2, w2, h2, 2], node3: [x3, y3, w3, h3, 2]…;

[0069] The stem node detection frame node1 on the second branch: [x1, y1, w1, h1, 3], node2: [x2, y2, w2, h2, 3], node3: [x3, y3, w3, h3, 3]…;

[0070] Among them, node1 in each branch is the stem node detection frame at the bottom end of the stem;

[0071] 5. According to the results of stem reconstruction, the stem node detection frame is positioned, and a stem node counting method is constructed;

[0072] The stem node number phenotype is divided into main stem node number and branch node number; after obtaining the main stem path and branch path, the stem node detection frame with the same flag value is located on the same stem, thereby completing the positioning of the stem node detection frame; for example, the stem node detection frame flag value is 1, indicating that the stem node detection frame is located on the main stem of the plant, and the stem node detection frame flag value is 2, indicating that the stem node detection frame is located on the first branch of the plant; the stem node detection frame information with the flag is stored in the array b, the flag values of all stem node detection frames are grouped, the number of stem node detection frame elements belonging to each group is calculated, and the stem node number phenotype of the main stem and each branch is obtained, and the stem node counting is realized;

[0073] 6. According to the positioning results of the stem node detection frame, the stem node detection frame is sorted, and a stem node spacing calculation method is constructed;

[0074] In the stem node detection frame information array b, the elements are composed of [x, y, w, h, n], respectively representing the center coordinates (x, y), width (w), height (h) and the stem number (n) of the stem node detection frame; the stem node detection frames with the same n value belong to the same stem, for example, they all belong to the same branch or all belong to the main stem; the stem node detection frame information is stored in the array according to the n value: b1, b2, …, b k+1 The stem node detection frame information on the main stem, the first branch and the kth branch is recorded respectively; the stem node detection frames in each array are sorted in ascending order according to their y values; after sorting, the adjacent stem node detection frames in the array are the adjacent stem nodes in the actual stem; the Euclidean distances d1, d2, …, d j-1 between the center points of the adjacent stem node detection frames are calculated, and a set of stem node distance values are obtained and stored in the arrays c1, c2, …, c k+1 , respectively recording the stem node distance on the main stem, the first branch and the kth branch;

[0075] The calculation formula of the Euclidean distance is shown in formula 2:

[0076]

[0077] 7. According to the sorting results of the stem node detection frame and the results of the stem region extraction, a stem diameter calculation method is constructed;

[0078] The automatic calculation of the stem diameter needs to combine the sorting results of the stem node detection frame and the results of the stem region extraction; the stem diameter is divided into the main stem diameter and the branch stem diameter; taking the main stem diameter as an example, the stem diameter at the middle position between the 5th stem node and the 6th stem node from the cotyledon node to the top of the main stem is obtained as the prediction value; based on the stem region extracted by the semantic segmentation network and the results of the stem node recognition and sorting, a maximum inscribed circle method is designed to detect and calculate the main stem diameter and the branch stem diameter;

[0079] Taking the main stem diameter as an example, the maximum inscribed circle of the stem region between the 5th stem node and the 6th stem node is drawn, and the diameter of the inscribed circle is calculated as the prediction value of the main stem diameter;

[0080] Firstly, the Canny operator is used to detect the edge of the target stem region to obtain the edge contour of the target stem region; the Canny operator uses the gradient change of the pixels, and the more obvious the gradient change between two pixels, the greater the difference between them; in the stem region segmentation graph, the difference between the stem region and the background color leads to a large gradient change at the edge, so the edge contour of the stem is accurately obtained by the Canny operator; the gradient calculation is divided into two directions, and the calculation method is shown in formula 3, wherein, represents the gradient value of the point, two-dimensional coordinates of the point;

[0081]

[0082] The contour detection Canny function in OpenCV is used to find the contour of the target stem internode stem region, and after the gradient calculation is completed, the non-maximum suppression method is used to retain the edge pixel points with the largest gradient change;

[0083] Then, the information node5: [x5, y5, w5, h5, 1] and node6: [x6, y6, w6, h6, 1] of the fifth stem node detection frame and the sixth stem node detection frame of the main stem are found from the array b1, and the horizontal and vertical coordinates of the center points of the two stem node detection frames are calculated respectively, and the calculation formulas are as formulas 4 and 5:

[0084]

[0085] wherein, and respectively represent the horizontal and vertical coordinates of the center point of the stem node detection frame k. The average value of the center point coordinates of the two stem node detection frames is taken as the target position for measuring the stem diameter, that is, the stem diameter measurement point, denoted as and , and the calculation formulas are as formulas 6 and 7:

[0086]

[0087] Whether the stem diameter measurement point is inside the edge contour of the target stem region obtained above is determined by the pointPolygonTest function of OpenCV. If the stem diameter measurement point is above the edge contour of the region, 0 is returned. If the stem diameter measurement point is outside the edge contour of the region, a negative value is returned. If the stem diameter measurement point is inside the edge contour of the region, a positive value is returned. When the return value is positive, the shortest distance of all pixel points inside the edge contour of the target stem region from the edge contour of the target stem region is calculated and stored in the matrix m. The maximum value in the matrix m is found by the minMaxLoc function, which is taken as the radius of the maximum inscribed circle, and the position corresponding to the maximum value is taken as the center of the maximum inscribed circle. The radius and center are input into the Circle function to draw the maximum inscribed circle of the edge contour of the target stem region on the segmented stem region image. The scale is calculated according to the standard reference beside the soybean plant image, and the diameter of the maximum inscribed circle is converted into the actual stem diameter. Thus, the prediction of the stem diameter of the main stem is completed.

[0088] The method for obtaining the branch stem thickness is similar to the method for obtaining the main stem thickness; the difference is that the target position for measuring the branch stem thickness is located between the first stem node and the second stem node of the branch; for the ith branch, the center position between node1[x1, y1, w1, h1, i+1] and node2[x2, y2, w2, h2, i+1] on the branch is taken as the target position for measuring the branch stem thickness;

[0089] 8. phenotype acquisition;

[0090] The collected soybean plant image is input into the soybean plant phenotype extraction deep learning network model, and the output of the soybean plant phenotype extraction deep learning network model is subjected to data processing; in combination with a multi-stalk path planning algorithm and a traditional image processing algorithm, a soybean plant stem related phenotype is acquired; the steps are as follows:

[0091] 8.1) deploying a computing environment relied on by the soybean plant phenotype extraction deep learning network model on a computing device;

[0092] 8.2) taking the processed data obtained in step 1 as input data of the soybean plant stem related phenotype deep learning network model;

[0093] 8.3) transplanting the complete soybean plant stem related phenotype deep learning network model obtained in step 2 into the computing device, and inputting the collected image into the computing device through a hardware interface;

[0094] 8.4) taking the result of the soybean plant stem related phenotype deep learning network model obtained in step 3 after filtering the stem nodes as a prior condition for phenotype extraction;

[0095] 8.5) taking the stem node detection result obtained in step 3 and the stem segmentation image obtained in step 2, processing the output of the soybean plant stem related phenotype extraction module in the manner of steps 4, 5, 6 and 7, and respectively obtaining the statistical results of the stem phenotypes of the soybean plant main stem node number and branch node number, stem node spacing, main stem thickness and branch stem thickness.

[0096] The closed black box is a three-dimensional structure, a scale is placed around the black box, a high-definition camera is placed above the black box, the background color is black, there is a stable light source, and the fixed height is 1.5 meters.

[0097] Compared with the prior art, the present application has the beneficial effects that:

[0098] This method for automatically acquiring soybean stem-related phenotypic data based on deep learning networks integrates deep learning and image processing techniques. Deep learning is used to train a target detection model to identify and extract stem node features, and a semantic segmentation model to identify and extract stem regions from whole soybean plant images. Combined with image binarization, the A* algorithm is designed for multi-stem path planning to reconstruct the main stem and branches, enabling the identification, location, and counting of soybean stem nodes, calculation of node spacing, and finally, the maximum inscribed circle method to obtain the stem diameter of the main stem and the stem diameter of each branch. This method is suitable for providing breeders with massive amounts of phenotypic data. It overcomes the shortcomings of existing manual methods, sensor detection methods, and image processing methods, achieving objective and automatic phenotypic data acquisition from whole soybean plant images. This method is unaffected by environmental factors, highly efficient, and yields accurate phenotypic information. Attached Figure Description

[0099] Figure 1 This is a flowchart illustrating the automatic acquisition of soybean plant stem-related phenotypes according to the present invention.

[0100] Figure 2 This is a partial image of a soybean plant collected in the experiment of this invention, after annotation.

[0101] Figure 3 This is a diagram of the architecture of the soybean plant stem node feature recognition network model in this invention;

[0102] Figure 4 This is a graph showing the prediction results of the soybean plant stem node feature recognition network model in the experiment of this invention;

[0103] Figure 5 This is a diagram of the architecture of the soybean plant stem extraction network model in this invention;

[0104] Figure 6 This is a stem segmentation image generated by the soybean plant stem extraction network model in the experiment of this invention;

[0105] Figure 7 This is a comparison of images before and after processing by the automatic stem segment filtering module in the experiment of this invention;

[0106] Figure 8 This is a diagram showing the effect of multi-stem path planning based on binarization and stalk segmentation image overlay in the experiment of this invention;

[0107] Figure 9 This is a diagram showing the effect after the stem nodes were located and sorted in the experiment of this invention;

[0108] Figure 10 This diagram illustrates the stem diameter detection method and stem internode spacing calculation method in this invention, as well as the image and detection value after phenotypic detection. Detailed Implementation

[0109] The design idea of the present application is to combine deep learning technology with image processing to provide a portable, fast and accurate method for phenotype acquisition, which helps to shorten the breeding process and opens up ideas for the extraction of soybean stem-related phenotypes.

[0110] There are currently four main methods for collecting soybean plant stem-related phenotypes: (1) manual method; (2) sensor monitoring method; (3) image processing method. The manual method requires planters or breeders to use professional tools such as vernier calipers to measure and count the number of main stems and branches, stem internode distance, and main stem and branch stem diameter of each plant, which is labor-intensive, low-efficiency and easily affected by subjective factors; the sensor detection method uses sensors to detect soybean plants in a specific area in real time to record the growth status of the plants, which is easily affected by external temperature, humidity and other factors, and has high maintenance cost; the image processing method is a new method currently used in automatic detection of crop phenotypes, which generally uses a camera to take pictures of the soybean plants collected by breeders, constructs a phenotype recognition model based on the distinguishing features of different types of phenotypes, and realizes automatic recognition of phenotypes. The effectiveness of the image processing method is highly dependent on the domain knowledge and rules used to design the underlying phenotype recognition model, and the scalability and robustness are low.

[0111] Soybean plants often have pod shading, which poses certain challenges to stem node detection and stem extraction, and there is no effective method for extracting stem diameter in the current literature and patents.

[0112] The present application focuses on the intelligent acquisition of stem phenotypes, and proposes an intelligent detection method for soybean plant stem-related phenotypes by combining target detection, semantic segmentation, path planning algorithm and traditional image processing method, which can automatically detect main stem node number, branch node number, main stem diameter, branch stem diameter and stem internode distance and other stem-related phenotypes on soybean plant images with dense pod shading, thereby reducing manual participation, reducing detection cost, improving detection efficiency and accuracy.

[0113] The following is a detailed description of the specific implementation of the present application, a deep learning-based automatic acquisition method for soybean plant stem-related phenotypes, (see Figures 1-10 ):

[0114] A deep learning-based automatic acquisition method for soybean plant stem-related phenotypes, the hardware includes a closed black box containing mature soybean plants, the closed black box is a three-dimensional structure, a ruler is placed around the closed black box, a high-definition camera is placed above the black box, the background color is black, there is a stable light source, and the fixed height is 1.5 meters; the characteristic is that the deep learning-based automatic acquisition method for soybean plant stem-related phenotypes is realized by including the following steps, (see Figure 1 ):

[0115] Step 1, soybean plant image acquisition and preprocessing; after the image of the mature soybean plant is obtained, the image data is labeled using LabelImg and Labelme tools, and the image is preprocessed and data augmented;

[0116] 1.1) Data source: mature soybean whole plant sample data collected by professional equipment, including unbranched, single-branched and complex branched plants.

[0117] Specifically, soybean plant images are collected: soybean breeders obtain mature soybean plant images by installing high-definition industrial cameras in airtight black boxes. The soybean plant forms involved include images of unbranched, single-branched and complex branched soybean plants. The images are transmitted to a computer for storage, and the stem nodes and stems in the soybean plants are labeled.

[0118] 1.2) Data annotation: the stem nodes and stems in the soybean plants are labeled.

[0119] Specifically, LabelImg visual image labeling tool and Labelme image labeling tool are used to label the image data. In LabelImg labeling, each stem node target area on the image is labeled with the class node1, node2, node3…, and in Labelme labeling, the stem edge area on the image is continuously labeled with stem. The labeled data is used as the input data set of the deep learning network model. (See Figure 2 ).

[0120] 1.3) Image preprocessing: the obtained images are input into the deep learning network as training data sets, and image size normalization, image translation, rotation, flipping, mirroring and other image enhancement methods are used for data augmentation to expand the original soybean plant image data set.

[0121] Step 2, constructing a soybean phenotype extraction deep learning network model; including constructing a stem node feature recognition network model; constructing a stem extraction network model; using the training data set obtained in step 1 to train the stem node feature recognition network model and the stem extraction network model, achieving the functions of identifying soybean plant stem nodes and segmenting soybean plant stems; obtaining stem segmentation images;

[0122] 2.1) Stem node feature recognition network model;

[0123] The stem node feature recognition network model is constructed by adding a CA attention mechanism to the existing YOLOv5s network. (See Figure 3); input the soybean plant image data set obtained in step 1 into the stem node feature recognition network to obtain a complete stem node feature recognition network model; and the specific operation performed is as follows:

[0124] Input the soybean plant image data set Input->Backbone layer->Neck layer->Head layer, so as to complete the feature extraction and detection operation, and obtain the stem node prediction map of the soybean plant;

[0125] The input image is adjusted to 640x640x3 size by using the resize function as the input of the Backbone layer, and after 6*6 convolution operation, the picture becomes 320x320x64, and then a 3*3 convolution block is used, and the picture size becomes 160x160x128, and then enters the C3 module and a 3*3 convolution block, and the size becomes 80x80x256, and then after the C3 module, the output 80x80x256 is obtained, and then a 3*3 convolution is performed to become 40x40x512; after the C3 module and the 3*3 convolution, it becomes 20x20x1024, and after the CA and SPPF layer, the three outputs of the Backbone layer are obtained: the output of the second C3 module from shallow to deep in the network 80x80x256, the output of the second C3 module 40x40x512 and the output of the SPPF layer 20x20x1024;

[0126] In the Backbone layer, the C3 module is proposed in YOLOv5, which aims to replace the CSP module, and the C3 module is mainly composed of CBS and Bottleneck Block module, and the CBS structure contains convolution layer, batch normalization layer and SiLU activation function; these components work together to enable the C3 module to effectively process and extract features;

[0127] The Bottleneck Block structure is first a 1x1 convolution layer, then a 3x3 convolution layer, and finally added to the initial input through a shortcut, and before the SPPF module, the CA attention mechanism is fused (see Figure 3 );

[0128] The CA attention mechanism simultaneously focuses on position information and channel information, aggregates features from both directions, and is suitable for stem node feature extraction. The specific process is as follows: the input of the C3 module is reduced to 1 / reduction times of the original channel number of the input feature map through 1*1 convolution operation, and the reduced channel number is processed through Batch Normalization and ReLU activation function. Two 1*1 convolution operations are performed on the reduced channel number, respectively for generating horizontal and vertical attention distributions. Finally, the horizontal and vertical attention distributions are normalized to values between 0 and 1 through the Sigmoid function, and the two distributions are used as weights to weight the input feature map. The output channel number is consistent with the input. (See Figure 3 );

[0129] The output 20*20*1024 of the Backbone is taken as the input of the Neck layer, and after 1*1 convolution operation, 20*20*512 is output. After downsampling operation, it is concatenated with the output 40*40*512 of the Backbone, and after C3 module and 1*1 convolution operation, 40*40*256 is obtained. After downsampling operation, it is concatenated with the output 80*80*256 of the Backbone layer to obtain the first output 80*80*256 of the Neck layer. After 3*3 convolution and output 40*40*256, the output is concatenated, and after C3 module, the second output of the Neck layer is output. After 3*3 convolution, 20*20*512 is obtained, which is concatenated with the first output 20*20*512. After C3 module, the third output 20*20*1024 of the Neck layer is obtained.

[0130] The output 80*80*256, 40*40*512 and 80*80*256 of the Neck layer after the above processing are input into the Head layer. The Head layer usually includes a series of Conv2d modules for extracting target position and category information from the feature map. The Conv2d module may include multiple convolution operations for further processing of the feature map, and usually has different convolution kernel sizes and channel numbers. These Conv2d modules output the predicted coordinates of the target box, the category confidence and the category information. The output result classifies and regresses each spatial position to generate a prediction box of target detection.

[0131] Common processing includes applying activation functions (such as sigmoid or softmax) to obtain category probabilities, and applying an Ahchor-base boundary box regression algorithm to predict the position of the boundary box.

[0132] Input soybean image to train the stem node feature recognition network model, get the position of each stem node detection box in the image, record it in array form as node1: [0, x1, y1, w1, h1], node2: [0, x2, y2, w2, h2], node3: [0, x3, y3, w3, h3]…, wherein the first bit is the label bit, 0 represents that the detection box corresponds to the stem node, x i , y i are the horizontal and vertical coordinates of the upper left corner of the stem node detection box, w i , h i are the length and width of the rectangular stem node detection box, respectively (see Figure 4 );

[0133] 2.2) Segment the stem, and construct a stem extraction network model;

[0134] The stem extraction network model is based on the existing U 2 -Net network, and the U 2 -Net network mainly consists of an encoding layer, a decoding layer and a saliency map fusion module; the encoder of the U 2 -Net is used to extract the feature information of the image, and gradually reduces the size of the feature map; the decoder restores the size of the feature map to the size of the original input image, and fuses the feature information with the feature map of the corresponding layer of the encoder, to generate a high-resolution segmentation result; the encoder adopts an improved structure, including a U-Net structure composed of two parallel paths and a connection module with a lightweight attention mechanism; these improvements can reduce the amount of calculation while maintaining high performance, especially suitable for processing small targets and details;

[0135] The decoder of the U 2 -Net uses a skip connection mechanism to connect the feature maps of different layers in the encoder with the feature maps of the corresponding layers in the decoder, so as to retain more spatial information and semantic information; in addition, the decoder also adopts deconvolution operation and up-sampling technology to restore the size of the feature map to the size of the original input image, and the U 2 -Net also uses a loss function suitable for image segmentation tasks, usually a cross-entropy loss function, which can measure the difference between the model prediction result and the true label, and guide the model to learn how to more accurately perform image segmentation.

[0136] In the training process, the model parameters are adjusted using optimization algorithms such as stochastic gradient descent by training on a large-scale labeled image dataset to obtain the minimum loss function value, and the training process aims to make the model learn how to extract features from the input image and perform pixel-level segmentation (see Figure 5 );

[0137] The specific process is that the soybean plant image data set Inputs -> encoder level En1 -> encoder level En2 -> encoder level En3 -> encoder level En4 -> encoder level En5 -> encoder level En6 -> decoder level De5 -> decoder level De4 -> decoder level De3 -> decoder level De2 -> decoder level De1 -> a saliency map fusion module, using a 1x1 convolution layer and a sigmoid function to generate a final saliency probability map, thus completing the image segmentation operation, and obtaining a soybean image stem segmentation map;

[0138] U 2 The most important one in the Net encoder structure is the residual U module (RSU), and the specific structure thereof is as follows:

[0139] For the residual module, the input channel number, the intermediate channel number, and the output channel number are set 1) 3x3 convolution (input channel, output channel);

[0140] 1) 3x3 convolution (input channel, output channel),

[0141] 2) 3x3 convolution (input channel, intermediate channel) 1 / 2 down sampling,

[0142] 3) 3x3 convolution (intermediate channel, intermediate channel) 1 / 2 down sampling,

[0143] 4) 3x3 convolution (intermediate channel, intermediate channel) 1 / 2 down sampling,

[0144] 5) 3x3 convolution (intermediate channel, intermediate channel) 1 / 2 down sampling,

[0145] 6) 3x3 convolution (intermediate channel, intermediate channel) 1 / 2 down sampling,

[0146] 7) 3x3 convolution (intermediate channel, intermediate channel),

[0147] 8) 3x3 convolution (intermediate channel, intermediate channel),

[0148] 9) 7) and 8) are stacked according to the channel, 3x3 convolution (intermediate channel, intermediate channel) 2 times up sampling,

[0149] 10) 6) and 9) are stacked according to the channel, 3x3 convolution (intermediate channel, intermediate channel) 2 times up sampling,

[0150] 11) 5) and 10) are stacked according to the channel, 3x3 convolution (intermediate channel, intermediate channel) 2 times up sampling,

[0151] 12) 4) and 11) are stacked according to the channel, 3x3 convolution (intermediate channel, intermediate channel) 2 times up sampling,

[0152] 13) stack 4) and 12) channel-wise, 3x3 convolution (intermediate channel, intermediate channel) 2x up-sampling,

[0153] 14) stack 2) and 13) channel-wise, 3x3 convolution (intermediate channel, intermediate channel), finally add 14) and 1) as normal addition;

[0154] The structure of the decoder stage and the encoder stage are similar, in the encoding process, each to the next layer, also will be the output input to the corresponding decoding layer of the "U" shape; through the saliency map fusion module, the saliency map is effectively fused with the original image to generate an enhanced image, so that the salient region in the image is more prominent or more easily noticed;

[0155] The soybean plant image data set obtained in step 1 is input into the U 2 -Net network to perform relevant feature extraction operations, and a complete stem extraction network model is obtained.

[0156] In the stem extraction network model, the background region of the soybean in the image is converted to black, which is equivalent to separating the stem target from the original image as a single individual, referred to as a stem segmentation image.

[0157] The superposition processing can improve the accuracy of phenotype acquisition compared to the traditional method of directly binarizing the image.

[0158] The soybean image is input to train the stem extraction network model, and the class value to which each pixel point in the image belongs is obtained. The pixel point value of the background is 0, the soybean plant stem region is marked with green, and the pixel point value of the stem region is consistent, all being 2 (see Figure 6 );

[0159] Step 3: According to the results of the stem node feature recognition network model and the stem extraction network model, an automatic stem node filtering module is constructed. According to the results of stem node detection and stem segmentation, an automatic stem node filtering module is constructed. The stem segmentation image obtained in step 2 is filtered to remove false stem node detection boxes, improving the detection accuracy of the stem node box.

[0160] 3.1) Through steps 1 and 2, the stem nodes have been identified from the soybean plant image, and the stem node detection box is labeled. The stem region is segmented from the original image, and the image labeled with the stem region is referred to as the stem segmentation image.

[0161] 3.2) Filter the candidate stem node detection box

[0162] To solve the problem that the pods in the image of mature soybean plants shield the stems, causing some pods to be misidentified as stem nodes, a method of using semantic segmentation results to filter misidentified stem nodes is proposed: first, the pixel points in the stem region obtained by semantic segmentation are simplified into two categories: stem and background, with 0 representing background and 2 representing the stem region. The stem segmentation image is stored in the form of a numpy array containing only 0 and 2 elements; second, the detected stem node box information is filtered in turn. If any pixel point within the stem node box has a value of 2 in the stem segmentation image, the stem node box is retained. Otherwise, the stem node box is considered not to pass through the stem and is considered invalid. In this way, stem node boxes located on the stem region are filtered out, improving the detection accuracy of the stem node box. Figure 7

[0163] Step 4: Based on the results of stem node filtering and stem segmentation, a stem reconstruction algorithm is constructed. The reconstruction of the main stem and branches is converted into a pathfinding problem based on a pixel map. A multi-path planning algorithm based on A* algorithm is designed to reconstruct the stem of the soybean plant from the soybean plant image. This includes converting the original soybean plant image into a binary image and superimposing it with the stem segmentation image obtained by the stem extraction network. This allows the A* algorithm to successfully plan a path in the binary region even if the semantic segmentation is disconnected. Through formula (1) of the A* algorithm, the smallest node among all the nodes to be evaluated is selected as the next expansion node. The main stem and branches are identified separately by combining the stem node feature recognition network to obtain the main stem and branches of the soybean plant.

[0164] Specifically, the detection of the number of nodes, node spacing, and stem thickness requires the identification of the main stem and branches from the soybean plant image, i.e., the reconstruction of the stem. The present invention proposes that the reconstruction of the main stem and branches be converted into a pathfinding problem based on a pixel map. The stem node detection results, stem segmentation image, and binary image are fused. A multi-path planning algorithm based on A* algorithm is designed to reconstruct the stem of the soybean plant from the soybean plant image, including the main stem and branches of the plant. Based on the reconstructed main stem and branches, the number of nodes, node spacing, and stem thickness phenotypes are obtained. Figure 8

[0165] 4.1) Construct a path planning pixel map.

[0166] The original plant image is converted into a binary image and superimposed with the stem segmentation image obtained by the stem extraction network. The pixel points with a value of 2 in the stem segmentation image are marked as 1, and the pixel points with a value of 0 in the binary soybean plant image and a value of 0 in the stem segmentation image are marked as 0. In this way, a pixel map representing the stem region is constructed. This map is the input for the path planning algorithm.

[0167] ​​​The path planning algorithm has good robustness using the superimposed graph: neither deviates from the stem area due to a single binary image, nor compensates for the stem area segmentation model. In the image where the local stem area is disconnected due to insufficient stem area segmentation, the superimposed binary image can effectively improve the connectivity of the stem area, so that the path planning algorithm can still successfully plan the path. Figure 8

[0168] 4.2) Combined with the stem node detection box information, the starting point and the ending point of the path planning are determined, the multi-stem path planning algorithm is constructed, and the main stem and each branch of the plant are reconstructed.

[0169] In view of the problem that the number of stem area pixels in the soybean plant leads to low efficiency of path planning, the present application proposes: combined with the stem node detection box information obtained by the stem node feature extraction network, the main stem and branches of the plant are reconstructed by using the path planning algorithm.

[0170] The flow of the multi-stem path planning algorithm is as follows: first, the topmost and the bottommost stem node detection boxes in the plant image are taken as the starting point and the ending point of the main stem path respectively; generally, the top of the main stem is the highest point of the whole plant, so the topmost stem node detection box is taken as the starting point of the main stem path; a path planning algorithm is executed to obtain the path between the starting point and the ending point, and the obtained path is taken as the main stem path; then, the picture is scanned from top to bottom, and the topmost stem node detection boxes in each branch stem are detected in turn, which are taken as new starting points of path planning in turn, and a path planning algorithm is executed to obtain the path between the new starting point and the ending point, and the obtained path is taken as the branch path; the intersection of the branch path and the main stem path is calculated, and the point is recorded as the ending point of the branch path, and the ending point of the branch path is updated; the above process is repeated, and each stem node detection box in the plant is scanned from top to bottom until all the stem node detection boxes belong to a certain branch; only one main stem path can be obtained in a plant, and multiple branch paths can be obtained; the obtained main stem path is taken as the main stem of the plant, and the obtained multiple branch paths are taken as the branches of the plant.

[0171] A* algorithm is an algorithm combining heuristic search algorithm and depth-first search algorithm, which specifies the starting point and the ending point of the path, and plans a path with the minimum overall expected cost by using heuristic information. The basis for A* algorithm to plan the path is shown in formula 1:

[0172]

[0173] In formula (1), represents the estimated cost of reaching the ending point from the starting point through node , represents the actual cost of the optimal path from the starting point to node , and ​​representing a node the estimated cost of the optimal path to the end point, select the node with the minimum value among all nodes to be evaluated as the next expansion node;

[0174] The specific process of the multi-stem path planning algorithm is as follows:

[0175] Since the plant is placed in a standard manner during shooting, the first stem node detection box obtained by scanning the image from the top end downward is specified as the starting point of the main stem, and the stem node detection box at the bottom end of the plant in the image is specified as the end point of the main stem, that is, the stem node detection boxes are sorted in ascending order of the vertical coordinates, and the center points [x1, y_min], [x2, y_max] of the stem node detection boxes with the minimum and maximum vertical coordinates are taken as the starting point and the end point of the path planning algorithm, respectively.

[0176] Before starting the path planning, an initial flag bit node1: [x1, y1, w1, h1, 0], node2: [x2, y2, w2, h2, 0], node3: [x3, y3, w3, h3, 0]… is added to the representation of all stem node detection boxes, representing that the stem node detection box has not been visited; after the path planning algorithm is executed for one round and the main stem path is obtained, the flag bits of all stem node detection boxes on the main stem are modified to 1, representing that the stem node detection box has been visited by the main stem path, that is, the information of the stem node detection boxes on the main stem at this time is node1: [x1, y1, w1, h1, 1], node2: [x2, y2, w2, h2, 1], node3: [x3, y3, w3, h3, 1]…

[0177] The image is scanned downward, and the first unvisited stem node detection box obtained by scanning is taken as the starting point of the first branch, and the end point is planned again to obtain a branch path; the intersection point of the branch path and the main stem path is recorded as the end point of the branch path; in this way, the path of the first branch is obtained, and the flag bits of all stem node detection boxes on the branch path are modified to 2, representing that the stem node detection box has been visited and belongs to the first branch.

[0178] The acquisition process of the remaining branch paths is similar to that of the first branch path, and the flag bit values of the stem node detection boxes are sequentially incremented, such as 3 for the stem node detection boxes on the second branch; the above process is repeated until the flag bits of all stem node detection boxes are not 0, that is, all stem node detection boxes have been visited and belong to the main stem or a branch;

[0179] After completion, the stems on the main stem and the branches are recorded according to morphological characteristics, that is, from the bottom end of the stem to the top end of the stem, that is:

[0180] ​The stem node detection frame node1 on the main stem: [x1, y1, w1, h1, 1], node2: [x2, y2, w2, h2, 1], node3: [x3, y3, w3, h3, 1]…;

[0181] The stem node detection frame node1 on the main stem: [x1, y1, w1, h1, 1], node2: [x2, y2, w2, h2, 1], node3: [x3, y3, w3, h3, 1]…;

[0182] The stem node detection frame node1 on the main stem: [x1, y1, w1, h1, 1], node2: [x2, y2, w2, h2, 1], node3: [x3, y3, w3, h3, 1]…;

[0183] Wherein, the node1 in each branch is the stem node detection frame at the bottom of the stem, (see Figure 9 ).

[0184] Step 5, according to the results of stem reconstruction, the stem node detection frame is positioned, and the stem node counting method is constructed; the stem node detection frames belonging to the main stem and belonging to each branch are distinguished and counted respectively, and the main stem node number and the branch node number phenotype are obtained;

[0185] The stem node number phenotype is divided into main stem node number and branch node number; after obtaining the main stem path and the branch path, the stem node detection frames with the same flag value are located on the same stem, thereby completing the positioning of the stem node detection frame; if the stem node detection frame flag value is 1, it indicates that the stem node detection frame is located on the main stem of the plant, and if the stem node detection frame flag value is 2, it indicates that the stem node detection frame is located on the first branch of the plant; the stem node detection frame information with the flag is stored in the array b, the flag values of all stem node detection frames are grouped, the number of stem node detection frame elements belonging to each group is calculated, the main stem and the stem node number of each branch are obtained, the stem node counting is realized, (see Figure 9 ).

[0186] Step 6, calculate the stem node distance, the calculation method; the stem node distance of the main stem and the branch is calculated respectively; including the stem node detection frames belonging to the same branch or the main stem, the Euclidean distance between the center points of adjacent stem node detection frames is calculated as the stem node distance through the Euclidean distance calculation formula (2);

[0187] According to the positioning results of the stem node detection frame, the stem node detection frame is sorted, and the stem node distance calculation method is constructed;

[0188] In the stem node detection frame information array b, each element is composed of [x, y, w, h, n], which respectively represents the center coordinates (x, y), width (w), height (h) and the stem number (n) of the stem node detection frame; the stem node detection frames with the same n value belong to the same stem, for example, they all belong to the same branch or all belong to the main stem; the stem node detection frame information is stored in the array according to the value of n: b1, b2, …, b k+1 The stem node detection frame information on the main stem, the first branch and the kth branch is recorded respectively; the stem node detection frames in each array are sorted in ascending order according to their y values; after sorting, the adjacent stem node detection frames in the array are the adjacent stem nodes in the actual stem; the Euclidean distances d1, d2, …, d j-1 between the center points of the adjacent stem node detection frames are calculated, and a set of stem node distance values are obtained and stored in the arrays c1, c2, …, c k+1 , which respectively record the stem node distances on the main stem, the first branch and the kth branch (see Figure 10 ).

[0189] The calculation formula of the Euclidean distance is shown in formula (2).

[0190]

[0191] Step 7, according to the sorting result of the stem node detection frame and the result of the stem region extraction, a stem diameter calculation method is constructed, based on the stem region extracted by the aforementioned semantic segmentation network and the results of stem node recognition and sorting, a maximum inscribed circle method is designed to calculate the stem diameter, which is suitable for the detection and calculation of the main stem diameter and the branch stem diameter;

[0192] The automatic calculation of the stem diameter needs to combine the sorting result of the stem node detection frame and the result of the stem region extraction; the stem diameter is divided into the main stem diameter and the branch stem diameter; taking the main stem diameter as an example, the stem diameter of the position between the 5th stem node and the 6th stem node from the cotyledon node to the top of the main stem is obtained as the prediction value; based on the stem region extracted by the aforementioned semantic segmentation network and the results of stem node recognition and sorting, a maximum inscribed circle method is designed to detect and calculate the main stem diameter and the branch stem diameter;

[0193] Taking the main stem diameter as an example, the maximum inscribed circle of the stem region between the 5th stem node and the 6th stem node is drawn, and the diameter d of the inscribed circle is calculated as the prediction value of the main stem diameter;

[0194] Firstly, the edge of the target stem region is detected by using Canny operator to obtain the edge contour of the target stem region; Canny operator uses the gradient change of pixels, and the more obvious the gradient change between two pixels, the greater the difference between them; in the stem region segmentation graph, the difference between the stem region and the background color leads to large gradient change at the edge, and the edge contour of the stem is accurately obtained by Canny operator; the calculation of gradient is divided into two directions, and the calculation method is shown in formula 3, wherein, The gradient value of the point is represented by G(x,y), and the two-dimensional coordinates of the point are represented by (x,y).

[0195]

[0196] The Canny function of OpenCV is used to find the contour of the target stem region, and after the gradient calculation, the non-maximum suppression method is used to retain the edge pixel point with the largest gradient change.

[0197] Then, the information node5: [x5, y5, w5, h5, 1] and node6: [x6, y6, w6, h6, 1] of the fifth stem node detection frame and the sixth stem node detection frame of the main stem are found from the array b1, and the horizontal and vertical coordinates of the center points of the two stem node detection frames are calculated, and the calculation formulas are shown in formula 4 and formula 5:

[0198]

[0199] Among them, and respectively represent the horizontal and vertical coordinates of the center point of the stem node detection frame. The average value of the center point coordinates of the two stem node detection frames is taken as the target position for measuring the stem diameter, that is, the stem diameter measurement point, which is denoted by and , and the calculation formulas are shown in formula 6 and formula 7:

[0200]

[0201] Whether the stem diameter measurement point is inside the edge contour of the target stem region obtained above is determined by the pointPolygonTest function of OpenCV, if the stem diameter measurement point is above the edge contour of the region, 0 is returned; if the stem diameter measurement point is outside the edge contour of the region, a negative value is returned; if the stem diameter measurement point ​​Within the region edge contour, return a positive value; when the return value is positive, calculate the shortest distance of all pixel points within the target stem region edge contour from the target stem region edge contour, and store it in matrix m; find the maximum value in matrix m through the minMaxLoc function as the radius of the maximum inscribed circle, and find the position corresponding to the maximum value as the center of the maximum inscribed circle; input the radius and center into the Circle function to draw the maximum inscribed circle of the target stem region edge contour on the stem region segmentation image; calculate the scale according to the standard reference beside the soybean plant image, and convert the diameter of the maximum inscribed circle into the actual stem thickness; thus, the prediction of the main stem thickness is completed;

[0202] The method for obtaining the branch stem thickness is similar to the method for obtaining the main stem thickness; the difference lies in that the target position for measuring the branch stem thickness is located between the first node and the second node of the branch; for the ith branch, the center position between node1[x1, y1, w1, h1, i+1] and node2[x2, y2, w2, h2, i+1] on the branch is taken as the target position for measuring the branch stem thickness, (see Figure 10 ).

[0203] Step 8, phenotype acquisition,

[0204] Input the collected image into the soybean plant phenotype extraction deep learning network model, and perform data processing on the output of the soybean plant phenotype extraction deep learning network model; combine the multi-stem path planning algorithm and the traditional image processing algorithm to obtain the soybean plant stem-related phenotypes, (see Figure 1 ).

[0205] 8.1) Deploy the computing environment relied on by the soybean plant phenotype extraction deep learning network model on the computing device;

[0206] 8.2) Take the processed data obtained in step 1 as the input data of the soybean plant stem-related phenotype deep learning network model;

[0207] 8.3) Transplant the complete soybean plant stem-related phenotype deep learning network model obtained in step 2 into the computing device, and input the collected image into the computing device through the hardware interface;

[0208] 8.4) Take the result of the filtered soybean plant stem-related phenotype deep learning network model obtained in step 3 as the prior condition for phenotype extraction;

[0209] 8.5) The stem node detection result obtained in step 3 and the stem segmentation image obtained in step 2 are combined, and the output of the soybean plant stem related phenotype extraction module is processed in the manner of steps 4, 5, 6, and 7 to obtain the statistical results of the number of main stem nodes and branch nodes, the stem node spacing, the main stem diameter, and the branch stem diameter of the soybean plant stem phenotype.

[0210] The soybean plant stem related phenotype automatic acquisition method based on the deep learning network uses deep learning and image processing technology in combination. For the mature period soybean whole plant image, the morphological characteristics of the soybean plant are combined, the target detection model is trained by deep learning technology to identify the stem node, the semantic segmentation model is trained to extract the stem region, the stem node detection result is optimized in combination with the stem region extraction result, the image binarization processing is combined, the multi-path planning algorithm is designed to reconstruct the main stem and branches, the stem node positioning, sorting, counting, and stem node spacing calculation are realized for the main stem and branches, the maximum inscribed circle method is used to obtain the main stem diameter and branch stem diameter, the automatic acquisition of the stem related phenotype of the mature period soybean whole plant is realized, and the method is suitable for providing massive phenotype data for breeders. The method overcomes the shortcomings of the existing artificial detection method, sensor detection method, and image processing method for collecting soybean whole plant phenotype data, and has the characteristics of objectivity, accurate phenotype data acquisition, automation, high efficiency, and freedom from natural environmental factors.

Claims

1. A deep learning-based automatic acquisition method for soybean plant stem-related phenotypes, the hardware comprising a closed black box containing mature soybean plants, the closed black box being a three-dimensional structure with a ruler placed around it, a high-definition camera placed above the black box, a black background color, a stable light source, and a fixed height of 1.5 meters; characterized in that The deep learning-based automatic acquisition method for soybean plant stem-related phenotypes is achieved by including the following steps: Step 1: Soybean plant image acquisition and preprocessing; after storing the images of mature soybean plants on the computer, use LabelImg and Labelme tools to label the data of the obtained images; and preprocess the images for data enhancement; Step 2: Constructing stem node feature recognition network model and stem extraction network model; using the training data set obtained in step 1, train the stem node feature recognition network model and the stem extraction network model to achieve the function of identifying soybean plant stem nodes and segmenting soybean plant stems; obtain stem segmentation images; Step 3: According to the results of stem node detection and stem segmentation, construct a stem node automatic filtering module; filter the stem node detection box of step 2 to improve the detection accuracy of the stem node box; Step 4: According to the results of stem node filtering and stem segmentation, construct a stem reconstruction algorithm; convert the main stem and branches into a pixel-based map pathfinding problem, and design a multi-path planning algorithm based on A* algorithm to reconstruct the soybean plant stem from the soybean plant image; This includes converting the original soybean plant image into a binary image, overlaying it with the stem segmentation image obtained by the stem extraction network, and achieving successful path planning by the A* algorithm in the binary region in the semantically segmented disconnected area; through the A* algorithm formula (1) , in formula (1), This indicates the nodes passed from the starting point. The estimated cost of reaching the destination. Indicates the distance from the starting point to the node. The actual cost of the optimal path, Indicates from node The estimated cost of the optimal path to the destination is selected from all nodes to be evaluated. The smallest node is used as the next expansion node; and by using multi-path planning and combining the stem node detection box information obtained by the stem node feature recognition network, the main stem and branches are identified separately to obtain the main stem and each branch of the soybean plant. Step 5: According to the results of stem reconstruction, position the stem node detection box and construct a stem node counting method; distinguish the stem node detection box belonging to the main stem and the branches, respectively, and count to obtain the main stem node number and branch node number phenotype; Step 6, according to the positioning result of the stem node detection frame, the stem node detection frame is sorted, and a stem node distance calculation method is constructed; the main stem and the branch are calculated respectively; including the stem node detection frame belonging to the same branch or main stem, the Euclidean distance calculation formula (2) is used to calculate the Euclidean distance between the center points of adjacent stem node detection frames as the stem node distance; In the stem node detection frame information array b, each element is composed of [x, y, w, h, n], x and y represent the horizontal and vertical coordinates of the stem node detection frame center coordinates, w represents the width, h represents the height, and n represents the stem number to which it belongs; stem node detection frames with the same n value belong to the same stem, and all belong to the same branch or the main stem; the stem node detection frame information is stored in the arrays b1, b2, …, bn according to the n value. k+1 The stem node detection frame information of the main stem, the first branch, and the kth branch is recorded respectively; the stem node detection frames in each array are sorted in ascending order according to their y values; after sorting, the adjacent stem node detection frames in the array are the adjacent stem nodes in the actual stem; the Euclidean distances d1, d2, …, dn between the center points of the adjacent stem node detection frames are calculated. j-1 A set of stem node distance values are obtained and stored in the arrays c1, c2, …, cn respectively, which record the stem node distances on the main stem, the first branch, and the kth branch respectively. k+1 ​ Step 7: According to the sorting results of the stem node detection box and the results of the stem region extraction, construct a stem diameter calculation method; Based on the results of stem region extraction by the above semantic segmentation network and stem node recognition and sorting, design a maximum inscribed circle method to calculate the stem diameter, which is suitable for the detection and calculation of main stem diameter and branch stem diameter; Step 8: Phenotype acquisition, input the collected images into the stem node feature recognition network model and the stem extraction network model, and process the output of the stem node feature recognition network model and the stem extraction network model; combine the multi-stem path planning algorithm and the traditional image processing algorithm to obtain the soybean plant stem-related phenotypes. 2.The automatic acquisition method of soybean plant stem related phenotypes based on deep learning according to claim 1, characterized in that, The step 1, in LabelImg labeling, each stem node target area on the image of the zero-branch soybean plant, the single-branch soybean plant and the complex-branch soybean plant is marked with a class label node_1, node_2, …, node_i, and in Labelme labeling, the stem edge area on the image is continuously marked as stem, and the labeled data is taken as a training data set of a deep learning network; the zero-branch soybean plant, the single-branch soybean plant and the complex-branch soybean plant images obtained above are taken as a training data set of a deep learning network, and data enhancement is performed in the image size normalization, image translation, rotation, flip and mirror image enhancement mode to expand the original soybean plant image data set. 3.The automatic acquisition method of soybean plant stem related phenotypes based on deep learning according to claim 1, characterized in that, The step 2 includes 2.1) a stem node feature recognition network model and 2.2) a stem extraction network model; 2.1) the stem node feature recognition network model The stem node feature recognition network model is constructed by adding a CA attention mechanism on the basis of an existing YOLOv5s network; the soybean plant image data set obtained in step 1 is input into the stem node feature recognition network to obtain a complete stem node feature recognition network model, and the specific operation is as follows: The soybean plant image data set is input into the Backbone layer, the Neck layer and the Head layer, that is, the feature extraction and detection operation is completed, and a stem node prediction map of the soybean plant is obtained; The input image is adjusted to 640*640*3 in size by using a resize function, and is taken as the input of the Backbone layer, and after 6*6 convolution operation, the picture becomes 320*320*64, and then after a 3*3 convolution block, the picture size becomes 160*160*128, and enters a C3 module and a 3*3 convolution block, and the size becomes 80*80*256, and then after a C3 module, the output 80*80*256 is obtained, and then after a 3*3 convolution and a C3 module, 40*40*512 is obtained, and then after a 3*3 convolution, 20*20*1024 is obtained, and after a CA and a SPPF layer, three outputs of the Backbone layer are obtained, that is, the output 80*80*256 of the second C3 module from shallow to deep of the network, the output 40*40*512 of the second C3 module and the output 20*20*1024 of the SPPF layer; in the Backbone layer, the C3 module is proposed in YOLOv5, and is used to replace the CSP module, and the C3 module is mainly composed of a CBS and a Bottleneck Block module, and the CBS structure includes a convolution layer, a batch normalization layer and a SiLU activation function; these components jointly act on the C3 module to enable the C3 module to strongly process and extract features; the Bottleneck Block structure is firstly a 1*1 convolution layer, then a 3*3 convolution layer, and finally a shortcut is added to the initial input, and a CA attention mechanism is fused before the SPPF module; The CA attention mechanism simultaneously focuses on position information and channel information, aggregates features from the two directions, and is suitable for stem node feature extraction; the specific process is that the 1x1 convolution operation of the C3 module reduces the channel number of the input feature map to 1 / reduction times of the original, and the reduced channel number is processed by Batch Normalization and ReLU activation function, and the reduced channel number is processed by two 1x1 convolution operations, which are respectively used to generate horizontal and vertical attention distributions, finally, the horizontal and vertical attention distributions are normalized to values between 0 and 1 by the Sigmoid function, and the two distributions are used as weights to weight the input feature map, and the output channel number is consistent with the input; The output 20x20x1024 of the Backbone is taken as the input of the Neck layer, and after 1*1 convolution operation, 20x20x512 is output, and after downsampling operation, it is concatenated with the output 40x40x512 of the Backbone, and after C3 module and 1*1 convolution operation, 40x40x256 is obtained, and after downsampling operation, it is concatenated with the output 80x80x256 of the Backbone layer, and the first output 80x80x256 of the Neck layer is obtained, and after 3*3 convolution and output 40x40x256, the second output of the Neck layer is output after C3 module, and after 3*3 convolution, 20x20x512 is obtained, and after Concat operation with the first output 20x20x512, the third output 20x20x1024 of the Neck layer is obtained after C3 module; The outputs 80x80x256, 40x40x512 and 80x80x256 of the Neck layer after the above processing are input into the Head layer, and the Head layer includes a series of Conv2d modules for extracting target position and category information from the feature map; the Conv2d module includes multiple convolution operations for further processing the feature map, and different convolution kernel sizes and channel numbers are used; these Conv2d modules output the predicted coordinates of the target frame, the category confidence and the category information, and the output result classifies and regresses each spatial position to generate a prediction frame of target detection; Input the soybean image to train the stem node feature recognition network model, and obtain the position of each stem node in the image and the total number of stem nodes, which are recorded in array form as node_1:[0,x1,y1,w1,h1], node_2:[0,x2,y2,w2,h2], …node_i:[0,x i ,y i ,w i ,h i ], wherein the first bit is a label bit, 0 represents that the detected box corresponds to a stem node, x i , y i are the horizontal and vertical coordinates of the center point of the detected box, w i , h i are the width and height of the rectangular stem node detection box; 2.2) Stem extraction network model The stem extraction network model is carried out on the basis of the existing U 2 -Net network, mainly composed of an encoding layer, a decoding layer and a saliency map fusion module; 2 The encoder of the U-Net is used for extracting feature information of the image, and gradually reducing the size of the feature map; the decoder is used for restoring the size of the feature map to the size of the original input image, and fusing the feature information with the feature map of the corresponding layer of the encoder, so as to generate a high-resolution segmentation result; the encoder adopts an improved structure, including a U-Net structure composed of two parallel paths and a connection module with a lightweight attention mechanism; U 2 The decoder of Net uses a skip connection mechanism to connect the feature maps of different layers in the encoder with the feature maps of the corresponding layers in the decoder, preserving more spatial information and semantic information; in the decoder, the size of the feature maps is restored to the size of the original input image through deconvolution operation and up-sampling technology;U 2 Net also uses a cross-entropy loss function suitable for image segmentation tasks to measure the difference between the model's prediction results and the true labels, and guide the model to learn how to more accurately perform image segmentation; During the training process, the model parameters are adjusted by using the stochastic gradient descent optimization algorithm to train on a large-scale labeled image dataset, so as to obtain the minimum loss function value, and at the same time, the training process aims to enable the model to learn how to extract features from the input image and perform pixel-level segmentation; Input soybean plant image dataset -> encoder level En1 -> encoder level En2 -> encoder level En3 -> encoder level En4 -> encoder level En5 -> encoder level En6 -> decoder level De5 -> decoder level De4 -> decoder level De3 -> decoder level De2 -> decoder level De1 -> saliency map fusion module, using a 1x1 convolution layer and a sigmoid function to generate a final saliency probability map, thus completing the image segmentation operation, obtaining a soybean image stem segmentation map; Composition U 2 The most important in the Net encoder structure is the residual U module (RSU), whose specific structure is: For the residual module, the input channel number, the intermediate channel number, and the output channel number are set; 1) 3x3 convolution, set input channel and output channel, get output channel feature map; 2) 3x3 convolution, set input channel and intermediate channel, use 1 / 2 down sampling on input channel feature map, get intermediate channel feature map; 3) 3x3 convolution, set intermediate channel and intermediate channel, use 1 / 2 down sampling on the feature map of the first intermediate channel, get intermediate channel feature map; 4) 3x3 convolution, set intermediate channel and intermediate channel, use 1 / 2 down sampling on the feature map of the first intermediate channel, get intermediate channel feature map; 5) 3x3 convolution, set intermediate channel and intermediate channel, use 1 / 2 down sampling on the feature map of the first intermediate channel, get intermediate channel feature map; 6) 3x3 convolution, set intermediate channel and intermediate channel, use 1 / 2 down sampling on the feature map of the first intermediate channel, get intermediate channel feature map; 7) 3x3 convolution, set intermediate channel and intermediate channel, use 1 / 2 down sampling on the feature map of the first intermediate channel, get intermediate channel feature map; 8) 3x3 convolution, set intermediate channel and intermediate channel, use 1 / 2 down sampling on the feature map of the first intermediate channel, get intermediate channel feature map; 9) stack 7) and 8) by channel, 3x3 convolution, set intermediate channel and intermediate channel, use 2 times up sampling on the feature map of the first intermediate channel, get intermediate channel feature map; 10) stack 6) and 9) by channel, 3x3 convolution, set intermediate channel and intermediate channel, use 2 times up sampling on the feature map of the first intermediate channel, get intermediate channel feature map; 11) stack 5) and 10) by channel, 3x3 convolution, set intermediate channel and intermediate channel, use 2 times up sampling on the feature map of the first intermediate channel, get intermediate channel feature map; 12) stack 4) and 11) by channel, 3x3 convolution, set intermediate channel and intermediate channel, use 2 times up sampling on the feature map of the first intermediate channel, get intermediate channel feature map; 13) stack 4) and 12) by channel, 3x3 convolution, set intermediate channel and intermediate channel, use 2 times up sampling on the feature map of the first intermediate channel, get intermediate channel feature map; 14) stack 2) and 13) by channel, 3x3 convolution, set intermediate channel and intermediate channel, finally add 14) and 1) according to ordinary addition; The structure of the decoder stage and the encoder stage is similar, and in the encoding process, each layer is input to the corresponding decoding layer of the "U" shape while being input to the next layer; the saliency map fusion module is to effectively fuse the saliency map with the original image to generate an enhanced image, so that the salient region in the image is more prominent or more easily noticed; The soybean plant image dataset obtained in step 2 is input into the stem extraction network U 2 The relevant feature extraction operation is performed in the Net, that is, a complete stem segmentation model is obtained. In the stem extraction model, in the input image, the background region of the soybean is converted into black color except the target region of the soybean stem, and the stem target is segmented from the original image as a separate individual, which is referred to as a stem segmentation image; and the superposition of the stem segmentation image and the binary image in step 4 is performed, which improves the accuracy of phenotype acquisition compared with the traditional way of directly binaryzing the image. The stem extraction network model is trained by inputting the soybean image, and the class value to which each pixel point in the image belongs is obtained, the pixel point value of the background is 0, and the pixel point value of the soybean plant stem region is consistent and is 2. 4.The automatic acquisition method of soybean plant stem related phenotypes based on deep learning according to claim 1, characterized in that, The step 3 includes 3.1) obtaining a stem segmentation image and 3.2) filtering a candidate stem node detection box; 3.1) Through steps 1 and 2, the stem nodes have been identified from the soybean plant image, and the stem node detection box is labeled; and the stem region is segmented from the original image, and the image labeled with the stem region is referred to as a stem segmentation image; 3.2) In view of the problem that the pods shield the stem and cause part of the pods to be misidentified as stem nodes in the mature soybean plant image, the result of semantic segmentation is used to filter the misidentified stem nodes: first, the stem region pixels obtained by semantic segmentation are simplified into two categories: stem and background, 0 represents background, and 2 represents the stem region, and the stem segmentation image is stored in the form of a numpy array with only 0 and 2 elements; second, the detected stem node box information is filtered in turn, if any pixel value in the range of the detection box is 2 in the stem segmentation image, the stem node box is retained, otherwise it is considered that the stem node box does not pass the stem, and it is an invalid stem node, so as to filter the stem node box located in the stem region and improve the detection accuracy of the stem node box.

5. The method according to claim 1, wherein, The step 4 comprises 4.1) constructing a path planning pixel map; converting the original soybean plant image into a binary image, and superimposing the stem segmentation image obtained by the stem extraction network to make the path planning algorithm still successfully plan a path in the binary region in the region segmented by the semantic segmentation; 4.2) designing a multi-stem path planning algorithm based on the A* algorithm formula (1) Combined with the stem node detection box information, the A* algorithm is used to realize multi-path planning, and each path obtained corresponds to a main stem or branch. 4.1) Construct a path planning pixel map; The specific process is: the original soybean plant image is converted into a binary image, and the stem segmentation image segmented by the stem extraction network is superimposed, the pixel points with a value of 2 in the stem segmentation image are marked as 1, and the pixel points with a value of 0 in the binary soybean plant image and a value of 0 in the stem segmentation image are marked as 0; thereby constructing a pixel map representing the stem region; The map is the input of the path planning algorithm; 4.2) Construct a multi-stem path planning algorithm to reconstruct the main stem and each branch; The specific process is: first, the topmost and the bottommost stem node detection boxes in the plant image are taken as the starting point and the ending point of the main stem path respectively; the topmost stem node detection box is taken as the starting point of the main stem path because the top of the main stem is the highest point of the whole plant; a path planning algorithm is executed to obtain the path between the starting point and the ending point, and the obtained path is taken as the main stem path; then, the picture is scanned from top to bottom, and the topmost stem node detection box in each branch stem is detected in turn, which is taken as the new starting point of the path planning, and a path planning algorithm is executed to obtain the path between the new starting point and the ending point, and the obtained path is taken as the branch path; the intersection of the branch path and the main stem path is calculated, and the point is recorded as the ending point of the branch path, and the ending point of the branch path is updated; the above process is repeated to scan each stem node detection box in the plant from top to bottom until all the stem node detection boxes belong to a branch; only one main stem path and multiple branch paths are allowed in one plant; the obtained main stem path is taken as the main stem of the plant, and the obtained multiple branch paths are taken as the branches of the plant; Since the plant is placed in a standard manner when being photographed, the first stem node detection box obtained by scanning the picture from the top down is specified as the starting point of the main stem, and the stem node detection box at the bottommost end of the plant in the image is specified as the ending point of the main stem, that is, the stem node detection boxes are sorted in ascending order of the vertical coordinates, and the center points [x1, y_min] and [x2, y_max] of the detection boxes with the minimum and maximum vertical coordinates are taken as the starting point and the ending point of the path planning algorithm respectively; Before starting path planning, add an initial flag to the representation of all stem node detection boxes: node_1: [x1,y1,w1,h1,0], node_2: [x2,y2,w2,h2,0], ..., node_i: [x i ,y i ,w i ,h i [0] indicates that the stem node detection box has not been visited; after the path planning algorithm has executed one round and obtained the main stem path, the flag bits of all stem node detection boxes on the main stem are modified to 1, indicating that the stem node detection box has been visited by the main stem path. That is, at this time, the information of the stem node detection boxes on the main stem is node_1: [x1,y1,w1,h1,1], node_2: [x2,y2,w2,h2,1], ..., node_i: [x i ,y i ,w i ,h i ,1]; The image is scanned downward, and the first unvisited stem node detection box obtained by the scanning is taken as the starting point of the first branch, and the ending point is planned again to obtain a branch path; the intersection of the branch path and the main stem path is recorded as the ending point of the branch path; thus, the path of the first branch is obtained, and the flag bit of all the stem node detection boxes on the branch path is modified to 2, indicating that the stem node detection box has been visited and belongs to the first branch; The acquisition process of the remaining branch paths is similar to that of the first branch path, and the flag bit values of the stem node detection boxes are incremented in turn, and the flag bit value of the stem node detection box on the second branch is 3; the above process is repeated until the flag bits of all the stem node detection boxes are not 0, that is, all the stem node detection boxes have been visited and belong to the main stem or a branch; After completion, the stem nodes on the main stem and the branches are recorded according to the morphological characteristics respectively, that is, from the bottom end of the stem to the top end of the stem, that is: Stem node detection frame on main stem node_1: [x1, y1, w1, h1, 1], node_2: [x2, y2, w2, h2, 1], …, node_i: [x i ,y i ,w i ,h i ,1] Stem node detection boxes on branch 1 node_1 : [x1, y1, w1, h1, 2], node_2 : [x2, y2, w2, h2, 2], …, node_i : [xi, yi, wi, hi, 2] i ,y i ,w i ,h i ,2] Stem node detection boxes on branch 2: node_1: [x1, y1, w1, h1, 3], node_2: [x3, y3, w3, h3, 3], …, node_i: [x i ,y i ,w i ,h i ,3] Wherein, node_1 in each branch is the stem node detection box at the bottommost end of the stem.

6. The method according to claim 1, wherein, Step 5 constructs a stem node counting method; The stem node number is divided into main stem node number and branch node number; after obtaining the main stem path and branch path, the stem node detection frame with the same value of the flag bit is located on the same stem, thereby completing the positioning of the stem node detection frame; the stem node detection frame flag bit value is 1, indicating that the stem node detection frame is located on the main stem of the plant, and the stem node detection frame flag bit value is 2, indicating that the stem node detection frame is located on the first branch of the plant; the stem node detection frame information with the flag bit is stored in the array b, the values of the flag bits of all the stem node detection frames are grouped, the number of stem node detection frame elements belonging to each group is calculated, the main stem and the stem node number phenotype of each branch are obtained, and stem node counting is realized.

7. The method according to claim 1, wherein, The step 7 constructs a stem thickness counting method; Automatic calculation of the stem thickness needs to combine the sorting result of the stem node detection frame and the result of the stem region extraction; the stem thickness is divided into main stem thickness and branch stem thickness; taking the main stem thickness as an example, the stem thickness of the position between the fifth stem node and the sixth stem node from the cotyledon node to the top of the main stem is obtained as a prediction value; based on the stem region extracted by the semantic segmentation network and the results of stem node recognition and sorting, a maximum inscribed circle method is designed to detect and calculate the main stem thickness and the branch stem thickness; Taking the main stem thickness as an example, the maximum inscribed circle of the stem region between the fifth stem node and the sixth stem node is drawn, the diameter d of the inscribed circle is calculated as the prediction value of the main stem thickness; First, the target stem region is edge detected by using Canny operator to obtain the edge contour of the target stem region. Canny operator uses the gradient change of pixels, and the more obvious the gradient change, the greater the difference between two pixel points. In the stem region segmentation graph, the difference between the stem region and the background color leads to large gradient change at the edge. The edge contour of the stem is accurately obtained by Canny operator, and the calculation of the gradient is divided into two directions as shown in the following formula (3), wherein, , represents the gradient value of the point, , , is the two-dimensional coordinate of the point. ； ； （3）； The contour detection Canny function in OpenCV is used to find the contour of the target stem region between the stem nodes, and after gradient calculation, the non-maximum suppression method is used to retain the edge pixel points with the largest gradient change; Then, the fifth stem node detection frame and the sixth stem node detection frame information node5: [x5, y5, w5, h5, 1] and node6: [x6, y6, w6, h6, 1] of the main stem are found from the array b1, the horizontal and vertical coordinates of the center point positions of the two stem node detection frames are calculated, and the calculation formulas are as formulas (4) and (5): （4）； （5）； wherein, and respectively represent the horizontal and vertical coordinates of the center point of the stem node detection frame k; the average of the two stem node detection frame center point coordinates is taken as the target position for measuring the stem diameter, i.e. the stem diameter measurement point, denoted as and , and the calculation formulae are as shown in formula (6) and formula (7): （6）； （7）； The stem diameter measurement point is determined using OpenCV's pointPolygonTest function. , Whether it is within the edge contour of the target stem region obtained above, if the stem diameter measurement point ( , If the stem thickness measurement point is above the region's edge contour, return 0; if the stem thickness measurement point is above the edge contour, return 0. , If the stem diameter measurement point is outside the region's edge contour, a negative value is returned; if the stem diameter measurement point is outside the region's edge contour, a negative value is returned. , Within the edge contour of the target stem region, a positive value is returned. When the return value is positive, the shortest distance from all pixels within the edge contour of the target stem region to the edge contour of the target stem region is calculated and stored in matrix m. The maximum value in matrix m is found using the minMaxLoc function and used as the radius of the maximum inscribed circle. The position corresponding to the maximum value is found and used as the center of the maximum inscribed circle. The radius and center are input into the Circle function to draw the maximum inscribed circle of the target stem region edge contour on the stem region segmentation image. The scale is calculated based on the standard reference next to the soybean plant image, and the diameter of the maximum inscribed circle is converted into the actual stem thickness. Thus, the prediction of the main stem thickness is completed. The method for obtaining the branch stem thickness is similar to the method for obtaining the main stem thickness; the difference lies in that the target position for measuring the branch stem thickness is located between the first stem node and the second stem node of the branch; for the i th branch, the center point position between node_1 [x1, y1, w1, h1, i+1] and node_2 [x2, y2, w2, h2, i+1] on the branch is taken as the target position for measuring the branch stem thickness; The soybean plant image dataset obtained by the soybean plant image acquisition and preprocessing in step 1 is taken as input data, the stem node feature recognition network model and the stem extraction network model are trained respectively, the stem segmentation image obtained in step 2 and the stem node detection result obtained in step 3 are output, and the methods in steps 4, 5, 6 and 7 are used for processing, and the statistical results of the main stem node number and branch node number, the stem node distance, the main stem thickness and the branch stem thickness of the stem phenotype of the soybean plant are obtained respectively.

Images