Soybean plant stem related phenotype automatic acquisition method based on deep learning

Abstract

The invention relates to a deep learning-based soybean plant stem related phenotype automatic acquisition method, which comprises the following steps of: fusing deep learning and an image processing technology for use through eight steps, training a target detection model to identify stem nodes through deep learning aiming at a whole soybean plant image in a mature period in combination with soybean plant morphological characteristics, and automatically acquiring the related phenotype of a soybean plant stem. A semantic segmentation model is trained to extract a stem region, a stem node detection result is optimized in combination with a stem region extraction result, then in combination with image binarization processing, a multipath planning algorithm is designed to realize stem node positioning, sorting, counting and stem node spacing calculation on the reconstructed main stem and branches, and the stem thickness of the main stem and the stem thickness of the branches are obtained by using a maximum inscribed circle method; the automatic acquisition of the related phenotypes of the whole plant stalks of the soybeans in the mature period is realized, and massive phenotype data is provided for breeders. The defect that the phenotypic data of the whole soybean plant is acquired by an existing manual and sensor detection method and an image processing method is overcome, and the phenotypic data can be acquired objectively, accurately, automatically and efficiently.

Description

Technical Field

[0001] The invention relates to a method for automatically acquiring soybean plant stem-related phenotypes based on deep learning, and belongs to the field of agricultural information technology. Background Art

[0002] Soybean is an important oil crop and food crop, and its yield is closely related to human production and life. With the continuous growth of the global population, the demand for soybeans is increasing. The soybean planting area in my country is less than 1 / 20 of the total cultivated land area, resulting in my country's soybean consumption mainly relying on imported soybeans. Therefore, cultivating high-yield soybean seeds has become an important way to solve the current problem of insufficient soybean production in my country. The use of artificial intelligence technology to develop an automatic acquisition algorithm for soybean phenotypes can greatly improve the efficiency of phenotypic data collection, quickly provide large-scale data for genetic research, and solve the shortcomings of current manual measurement methods such as low efficiency, high cost, reliance on professional knowledge, and strong subjectivity. This technology focuses on the automatic detection of soybean plant stem-related phenotypes, combines deep learning technology with image processing technology, and provides a lightweight, fast, and accurate method for automatic detection of soybean stem-related phenotypes, which helps to shorten the breeding process and provide ideas for the automatic detection of related crop phenotypes.

[0003] Currently, there are three main ways to obtain stem-related phenotypes from soybean plants: (1) manual method; (2) sensor detection method; and (3) image processing method.

[0004] The manual method requires planting or breeding personnel to use professional tools such as vernier calipers to measure and count the number of main stems and branch nodes, internode spacing, and main stem and branch stem thickness of each plant one by one. It is labor-intensive, inefficient, and easily affected by subjective factors. The sensor detection method uses sensors to perform real-time detection of soybean plants in a specific area to record the growth status of the plants. It is easily disturbed by external factors such as temperature and humidity, and has high maintenance costs. Image processing is a new method currently used in automatic detection of crop phenotypes. Generally, a camera is used to take pictures of soybean plants collected by breeders, and a phenotype recognition model is constructed based on the distinguishing characteristics of different types of phenotypes to achieve automatic recognition of phenotypes. The effectiveness of the image processing method is highly dependent on the domain knowledge and laws used in the design of the underlying phenotype recognition model, and has low scalability and robustness. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and to provide a method for automatically acquiring soybean plant stem-related phenotypes based on deep learning. The method integrates deep learning and image processing technology. For soybean plant images affected by occlusion noise, a target detection model is trained through deep learning technology to identify and locate the stem nodes in the plant, a semantic segmentation model is trained to extract the stem area in the plant, and a path planning algorithm is designed to extract the main stem and branch stems respectively, and the spacing between the stem nodes is measured. Finally, the maximum inscribed circle method is used to obtain the stem thickness of the main stem and branches.

[0006] The technical solution of the present invention is: A method for automatically acquiring soybean plant stem-related phenotypes based on deep learning is implemented by comprising the following steps: 1. Soybean plant image acquisition and preprocessing; Soybean testers installed a high-definition industrial camera on a sealed black box to obtain images of mature soybean plants. The soybean plants involved were of various morphologies, including images of unbranched soybean plants, single-branched soybean plants, and complex-branched soybean plants. After the images were transferred to a computer for storage, the acquired image data were labeled using the LabelImg and Labelme tools, respectively. The stem node information was labeled using the LabelImg tool, and each stem node target area on the image was labeled as a node type. The stem area was labeled using the Labelme tool, and the stem edge area on the image was continuously labeled as a stem type. The labeled image data was enhanced and used as the training input data set for the deep learning network. 2. Construct a deep learning network model for soybean phenotype extraction; 2.1) Network model for stem node feature recognition; The stem node feature recognition network model is constructed by adding the CA attention mechanism to the existing YOLOv5s network. The soybean plant image dataset obtained in step 1 is input into the stem node feature recognition network to obtain a complete stem node feature recognition network model. The specific operations performed are as follows: Input the soybean plant image dataset Input->Backbone layer->Neck layer->Head layer, so that the feature extraction and detection operations can be completed, and the soybean plant stem node prediction map can be obtained; The input image is resized to 640×640×3 using the resize function as the input of the Backbone layer. After a 6*6 convolution operation, the image becomes 320×320×64, and then after a 3*3 convolution block, the image size becomes 160×160×128, enters the C3 module and a 3*3 convolution block, and the size becomes 80×80×256. At this time, after passing through a C3 module, the output is 80×80×256, and then after a 3*3 convolution and C3 module, it becomes 40×40×512, and then after a 3*3 convolution, it becomes 20×20×1024. After passing through the CA and SPPF layers, the three outputs of the Backbone layer are: the output of the second C3 module from shallow to deep network 80×80×256, the output of the second C3 module 40×40×512 and the output of the SPPF layer 20×20×1024. In the Backbone layer, the C3 module was proposed in YOLOv5 to replace the CSP module. The C3 module is mainly composed of CBS and Bottleneck Block modules. The CBS structure contains convolutional layers, batch normalization layers, and SiLU activation functions. These components work together to enable the C3 module to effectively process and extract features. The Bottleneck Block structure starts with a 1x1 convolution layer, followed by a 3x3 convolution layer, which is then added to the initial input through a shortcut, and the CA attention mechanism is integrated before the SPPF module; The CA attention mechanism focuses on both position information and channel information, aggregates features from these two directions, and is suitable for feature extraction of stem nodes. The specific process is as follows: the input of the C3 module reduces the number of channels of the input feature map to 1 / reduction times the original number through a 1x1 convolution operation, performs Batch Normalization and ReLU activation function processing on the reduced number of channels, and performs two 1x1 convolution operations on the reduced number of channels to generate horizontal and vertical attention distributions, respectively. Finally, the horizontal and vertical attention distributions are normalized to values ​​between [0,1] through the Sigmoid function, and the two distributions are used as weights to weight the input feature map. The number of output channels is consistent with the input. The output of Backbone 20×20×1024 is used as the input of the Neck layer. After the 1*1 convolution operation, the output is 20×20×512. After the downsampling operation, it is concat-joined with the 40×40×512 output of Backbone. After the C3 module and the 1*1 convolution operation, 40×40×256 is obtained. After the downsampling operation, it is concat-joined with the output of the Backbone layer 80×80×256 to obtain the first output of the Neck layer 80×80×256. After the 3*3 convolution and the output 40×40×256, the output is used as the second output of the Neck layer after the C3 module. After the 3*3 convolution, 20×20×512 is obtained, and the first output 20×20×512 is concat-joined. After the C3 module, the third output of the Neck layer 20×20×1024 is obtained. The outputs of the Neck layer after the above processing (80×80×256, 40×40×512, and 80×80×256) are input to the Head layer. The Head layer usually contains a series of Conv2d modules to extract the target position and category information from the feature map. The Conv2d module may contain multiple layers of convolution operations to further process the feature map, usually with different convolution kernel sizes and numbers of channels. These Conv2d modules will output the coordinates, category confidence, and category information of the predicted target box. The output results classify and regress each spatial position to generate a predicted box for target detection. Common processing includes applying an activation function (such as sigmoid or softmax) to obtain category probabilities, and applying a bounding box regression algorithm Anchor-base to predict the location of the bounding box; Input soybean image to train the stem node feature recognition network model, obtain the position of each stem node detection box in the image, and record it in the form of array node1:[0,x 1 ,y 1 ,w 1 ,h 1 ],node2:[0,x 2 ,y 2 ,w 2 ,h 2 ],node3:[0,x 3 ,y 3 ,w 3 ,h 3 ]…, where the first bit is the label bit, 0 means the object corresponding to the detection box is a stem node, x i ,y i are the horizontal and vertical coordinates of the center point of the stem node detection frame, w i ,h iare the length and width of the rectangular stem node detection frame respectively; 2.2) Stem extraction network model; The stem extraction network model is based on the existing U 2 -Net network, which is mainly composed of encoding layer, decoding layer and saliency map fusion module; 2 -Net's encoder is used to extract the feature information of the image and gradually reduce the size of the feature map; the decoder restores the size of the feature map to the size of the original input image and fuses the feature information with the feature map of the corresponding layer of the encoder to produce a high-resolution segmentation result; the encoder adopts an improved structure, including a U-Net structure consisting of two parallel paths and a connection module with a lightweight attention mechanism. These improvements reduce the amount of computation while maintaining high performance, which is particularly suitable for processing small targets and details; U 2 -Net's decoder uses a skip connection mechanism to connect the feature maps of different layers in the encoder with the feature maps of the corresponding layers in the decoder to retain more spatial and semantic information; in addition, the decoder uses deconvolution operations and upsampling techniques to restore the size of the feature map to the size of the original input image; U 2 -Net also uses a loss function suitable for image segmentation tasks, usually the cross entropy loss function, which measures the difference between the model prediction results and the true labels and guides the model to learn how to perform image segmentation more accurately; During the training process, the model parameters are adjusted using a stochastic gradient descent optimization algorithm to minimize the loss function value by training on a large-scale labeled image dataset. The training process aims to enable the model to learn how to extract features from the input image and perform pixel-level segmentation. Input the soybean plant image dataset Inputs->Encoder level En1->Encoder level En2->Encoder level En3->Encoder level En4->Encoder level En5->Encoder level En6->Decoder level De5->Decoder level De4->Decoder level De3->Decoder level De2->Decoder level De1->Saliency map fusion module, use 1×1 convolution layer and sigmoid function to generate the final saliency probability map, so as to complete the image segmentation operation and obtain the soybean image stem segmentation map; Composition 2 The most important part of the -Net encoder structure is the residual U module (RSU), and its specific structure is: For the residual module, set the number of input channels, the number of intermediate channels, and the number of output channels; 1) 3x3 convolution (input channel, output channel), 2) 3x3 convolution (input channel, middle channel) 1 / 2 downsampling, 3) 3x3 convolution (middle channel, middle channel) 1 / 2 downsampling, 4) 3x3 convolution (middle channel, middle channel) 1 / 2 downsampling, 5) 3x3 convolution (middle channel, middle channel) 1 / 2 downsampling, 6) 3x3 convolution (middle channel, middle channel) 1 / 2 downsampling, 7) 3x3 convolution (middle channel, middle channel), 8) 3x3 convolution (middle channel, middle channel), 9) Superimpose 7) and 8) by channel, 3x3 convolution (middle channel, middle channel) upsampled by 2 times, 10) Superimpose 6) and 9) by channel, 3x3 convolution (middle channel, middle channel) upsampled by 2 times, 11) 5) and 10) are stacked by channel, 3x3 convolution (middle channel, middle channel) is upsampled by 2 times, 12) Superimpose 4) and 11) by channel, 3x3 convolution (middle channel, middle channel) upsampled by 2 times, 13) Superimpose 4) and 12) by channel, 3x3 convolution (middle channel, middle channel) upsampled by 2 times, 14) Add 2) and 13) by channel, 3x3 convolution (middle channel, middle channel), and finally add 14) and 1) by normal addition; The structure of the decoder level is similar to that of the encoder level. During the encoding process, each time the output is passed to the next layer, the output is also input to the decoding layer corresponding to the "U" shape. The saliency map fusion module effectively fuses the saliency map with the original image to produce an enhanced image, making the salient area in the image more prominent or easier to notice. Input the soybean plant image dataset obtained in step 1 into U 2 -Net network to perform relevant feature extraction operations, that is, to obtain a complete stem extraction network model; In the stem extraction network model, in the input image, except for the soybean stem target area, the background area of ​​the plant is converted into black, which is equivalent to segmenting the stem target from the original image as a separate individual, referred to as the stem segmentation image, and the image is superimposed with the stem segmentation image and the binary image in step 4. Compared with the traditional method of directly binarizing the image, the superposition process can improve the accuracy of phenotypic acquisition; Input soybean images to train the stalk extraction network model, and obtain the category value of each pixel in the image. The pixel value of the background is 0, and the soybean plant stalk area is marked with green. The pixel values ​​in the stalk area are consistent, all 2. 3. According to the results of the node feature recognition network model and the stem extraction network model, a node automatic filtering module is constructed; 3.1) Through steps 1 and 2, the nodes have been identified from the soybean plant image and marked with node detection frames; and the stem region is segmented from the original image. The image with the marked stem region is called the stem segmentation image; 3.2) Filter candidate stem node detection frames; Aiming at the problem that pods block the stems in the images of mature soybean plants, causing some pods to be misidentified as stem nodes, a method of using semantic segmentation results to filter misidentified stem nodes is proposed: first, the pixels in the stem area obtained by semantic segmentation are simplified into two categories: stems and background, with 0 representing the background and 2 representing the stem area, and the stem segmentation image is stored in the form of a numpy array with only 0 and 2 elements; secondly, the detected stem node frame information is filtered in turn, if any pixel point within the detection frame has a value of 2 in the stem segmentation image, the stem node frame is retained, otherwise it is considered that the stem node frame does not pass through the stem and is an invalid stem node; in this way, the stem node frame located in the stem area is filtered out to improve the detection accuracy of the stem node frame; 4. Based on the results of stem node filtering and stem segmentation, a stem reconstruction algorithm is constructed; To detect the number of nodes, internode spacing and stem thickness, it is necessary to first identify the main stem and branches from the soybean plant image, that is, to reconstruct the stem. The present invention proposes: converting the reconstruction of the main stem and branches into a path-finding problem based on a pixel map, integrating the node detection results, the stem segmentation image and the binary image, and designing a multi-path planning algorithm based on the A* algorithm to reconstruct the plant stem from the soybean plant image, including the main stem and each branch of the plant; based on the reconstructed main stem and each branch, the number of nodes, internode spacing and stem thickness phenotypes are obtained; 4.1) Construct path planning pixel map; The original plant image is converted into a binary image and superimposed with the stem segmentation image obtained by the stem extraction network segmentation; the pixel points with a value of 2 in the stem segmentation image are marked as 1, and the pixel points with a value of 0 in the binary soybean plant image and a value of 0 in the stem segmentation image are marked as 0; thereby, a pixel map representing the stem area is constructed; the map is the input of the path planning algorithm; The path planning algorithm has good robustness when using superimposed images: it will not cause the path planning to deviate from the stem area due to a single binary image, and it can also make up for the stem area segmentation model. In images where the stem area segmentation is insufficient and the local stem area is disconnected, the connectivity of the stem area can be effectively improved by superimposing the binary image, so that the path planning algorithm can still successfully plan the path. 4.2) Combine the stem node detection frame information to determine the starting and ending points of the path planning, build a multi-stem path planning algorithm, and reconstruct the main stem and branches of the plant; Aiming at the problem that there are many pixels in the stem area of ​​soybean plants, which leads to low efficiency of path planning, the present invention proposes: combining the stem node detection frame information obtained by the stem node feature extraction network, and using the path planning algorithm to reconstruct the main stem and branches of the plant; The process of the multi-stem path planning algorithm is as follows: first, the topmost and bottommost stem node detection frames in the plant image are used as the starting point and end point of the main stem path respectively; in general, the top of the main stem is the highest point of the entire plant, so the topmost stem node detection frame is used as the starting point of the main stem path; the path planning algorithm is executed once to obtain the path between the starting point and the end point, and the obtained path is used as the main stem path; then, the image is scanned from top to bottom, and the topmost stem node detection frame in each branch stem is detected in turn, and it is used as the new starting point of the path planning in turn, and the path planning algorithm is executed once to obtain the path between the new starting point and the end point, and the obtained path is used as the branch path; the intersection of the branch path and the main stem path is calculated, and the point is recorded, and it is used as the end point of the branch path, and the end point of the branch path is updated; repeat the above process, scan each stem node detection frame in the plant from top to bottom, until all stem node detection frames belong to a branch; only one main stem path can be obtained in a plant, and multiple branch paths can be obtained; the obtained main stem path is used as the main stem of the plant, and the obtained multiple branch paths are used as the branches of the plant; The A* algorithm is an algorithm that combines the heuristic search algorithm and the depth-first search algorithm. It specifies the starting point and end point of the path and uses heuristic information to plan the path with the minimum overall expected cost. The basis for the A* algorithm to plan the path is shown in Formula 1:

[0007] In formula (1), Indicates passing through the node from the starting point The estimated cost to reach the destination, From the starting point to the node The actual cost of the optimal path, Represents a slave node The estimated cost of the optimal path to the destination, selected from all nodes to be evaluated The node with the smallest value is used as the next expansion node; The specific process of the multi-stalk path planning algorithm is as follows: Since the plants were placed in a standardized manner during the shooting, the first stem node detection frame obtained by scanning downward from the top of the image was defined as the starting point of the main stem, and the stem node detection frame at the bottom of the plant in the image was defined as the end point of the main stem. That is, the stem node detection frames were sorted in ascending order of the ordinate, and the center points of the stem node detection frames with the minimum and maximum ordinate values ​​were taken [x 1 ,y_min],[x 2 ,y_max] as the starting point and end point of the path planning algorithm; Before starting path planning, add an initial flag node1 to the representation of all stem node detection boxes: [x 1 ,y 1 ,w 1 ,h 1 ,0],node2:[x 2 ,y 2 ,w 2 ,h 2 ,0],node3:[x 3 ,y 3 ,w 3 ,h 3 ,0]…, indicating that the stem node detection frame has not been visited; after the path planning algorithm is executed for one round and the main stem path is obtained, the flag bits of all the stem node detection frames on the main stem are changed to 1, indicating that the stem node detection frame has been visited by the main stem path, that is, the information of the stem node detection frame located on the main stem at this time is node1:[x 1 ,y 1 ,w 1 ,h 1 ,1],node2:[x 2 ,y 2 ,w 2 ,h 2 ,1],node3:[x 3 ,y 3 ,w 3 ,h 3 ,1]…; Scan the image downward, take the first unvisited stem node detection frame scanned as the starting point of the first branch, and plan the path again with the end point to obtain a branch path; record the intersection of the branch path and the main stem path, and take the intersection as the end point of the branch path; thus, obtain the path of the first branch, modify the flag bit of all stem node detection frames on the branch path to 2, indicating that the stem node detection frame has been visited and belongs to the first branch; The acquisition process of the remaining branch paths is similar to that of the first branch path. The flag bit values ​​of the stem node detection frames are incremented in sequence. For example, the flag bit value of the stem node detection frame on the second branch is 3. Repeat the above process until the marked bits of all stem node detection frames are not 0, that is, all stem node detection frames have been visited and belong to the main stem or a branch. After completion, record the morphological characteristics of the nodes on the main stem and branches, that is, record from the bottom of the stem to the top of the stem, that is: The node detection box node1 on the main stem: [x 1 ,y 1 ,w 1 ,h 1 ,1],node2:[x 2 ,y 2 ,w 2 ,h 2 ,1],node3:[x 3 ,y 3 ,w 3 ,h 3 ,1]…; The node detection box node1 on branch 1: [x 1 ,y 1 ,w 1 ,h 1 ,2],node2:[x 2 ,y 2 ,w 2 ,h 2 ,2],node3:[x 3 ,y 3 ,w 3 ,h 3 ,2]…; The node detection box node1 on branch 2: [x 1 ,y 1 ,w 1 ,h 1 ,3],node2:[x 2 ,y 2 ,w 2 ,h 2 ,3],node3:[x 3 ,y 3 ,w 3 ,h 3 ,3]…; Among them, node1 in each branch is the stem node detection frame at the bottom of the stem; 5. According to the results of stem reconstruction, the stem node detection frame is located and a stem node counting method is constructed; The node number phenotype is divided into the main node number and the branch node number; after obtaining the main stem path and the branch path, the node detection frames with the same flag value are located on the same stem, thereby completing the positioning of the node detection frame; if the node detection frame flag value is 1, it means that the node detection frame is located on the main stem of the plant, and if the node detection frame flag value is 2, it means that the node detection frame is located on the first branch of the plant; the node detection frame information with the flag is stored in array b, the flag values ​​of all node detection frames are grouped, the number of node detection frame elements belonging to each group is calculated, the node number phenotype of the main stem and each branch is obtained, and the node counting is realized; 6. According to the positioning results of the stem node detection frames, the stem node detection frames are sorted and a method for calculating the stem node spacing is constructed; In the stem node detection frame information array b, each element is composed of [x, y, w, h, n], which respectively represents the center coordinates (x, y), width (w), height (h) and stem number (n) of the stem node detection frame; the stem node detection frames with the same n value belong to the same stem, for example, they all belong to the same branch or the main stem; the stem node detection frame information is stored in the array according to the n value: b 1 , b 2 , …, b k+1 Record the node detection frame information on the main stem, the first branch, and the kth branch respectively; sort the node detection frames in each array in ascending order according to their y-coordinate values; after sorting, the adjacent node detection frames in the array are the adjacent nodes in the actual stem; calculate the Euclidean distance d between the center points of adjacent node detection frames 1 , d 2 , …, d j-1 , and obtain a set of internode spacing values, which are stored in array c 1 , c 2 , …, c k+1 In the data, the internode distances on the main stem, the first branch, and the kth branch were recorded respectively; The calculation formula of Euclidean distance is shown in Formula 2:

[0008] 7. According to the sorting results of the stem node detection frames and the results of the stem area extraction, a stem thickness calculation method is constructed; The automatic calculation of stem thickness needs to be combined with the sorting results of the stem node detection frame and the results of stem area extraction; the stem thickness is divided into the main stem thickness and the branch stem thickness; taking the main stem thickness as an example, it is necessary to obtain the stem thickness between the 5th stem node and the 6th stem node from the cotyledon node to the top of the main stem as the prediction value; based on the results of stem area extraction and stem node identification and sorting obtained by the aforementioned semantic segmentation network, the maximum inscribed circle method is designed to detect and calculate the main stem thickness and branch stem thickness; Taking the main stem thickness as an example, draw the largest inscribed circle of the stem area between the 5th and 6th stem nodes, and calculate the diameter of the inscribed circle. , as the predicted value of the main stem thickness; First, the Canny operator is used to detect the edge of the target stem area and obtain the edge contour of the target stem area. The Canny operator uses the gradient change of pixels. The more obvious the gradient change is, the greater the difference between two pixels. In the stem area segmentation map, the difference between the stem area and the background color leads to a large gradient change at the edge. The Canny operator is used to accurately obtain the edge contour of the stem. The calculation of the gradient is divided into In two directions, the calculation method is shown in Formula 3, where: represents the gradient value at this point, is the two-dimensional coordinate of the point;

[0009] Use the contour detection Canny function in OpenCV to find the contour of the target stem internode area. After completing the gradient calculation, use the non-maximum suppression method to retain the edge pixels with the largest gradient change. Then, from array b 1 Find the information of the main stem node 5 and node 6 detection frames in node5: [x 5 ,y 5 ,w 5 ,h 5 ,1] and node6:[x 6 ,y 6 ,w 6 ,h 6 ,1], respectively calculate the horizontal and vertical coordinates of the center points of the two stem node detection frames, and the calculation formulas are as follows:

[0010] in, and The average value of the coordinates of the two center points of the stem node detection frame is taken as the target position for measuring stem thickness, i.e., the stem thickness measurement point, denoted as and , the calculation formula is as follows: Formula 6 and Formula 7:

[0011] Use OpenCV's pointPolygonTest function to determine the stem thickness measurement point Is it within the target stem area edge contour obtained above? If the stem thickness measurement point is above the edge of the area, it returns 0; Outside the edge of the area, a negative value is returned; if the stem thickness measurement point Return a positive value within the edge contour of the region; when the return value is positive, calculate the shortest distance between all pixel points within the edge contour of the target stalk region and the edge contour of the target stalk region, and store it in the matrix m; find the maximum value in the matrix m through the minMaxLoc function, as the radius of the maximum inscribed circle, find the position corresponding to the maximum value, as the center of the maximum inscribed circle; input the radius and center into the Circle function, and draw the maximum inscribed circle of the edge contour of the target stalk region on the stalk region segmentation image; calculate the scale according to the standard reference object next to the soybean plant image, and convert the maximum inscribed circle diameter into the actual stem thickness; at this point, the prediction of the main stem thickness is completed; The method for obtaining the branch stem diameter is similar to that for the main stem; the difference is that the target position for measuring the branch stem diameter is between the first and second stem nodes of the branch; for example, for the i-th branch, take node1[x 1 ,y 1 ,w 1 ,h 1 ,i+1] and node2[x 2 ,y 2 ,w 2 ,h 2 ,i+1] is the target position for measuring the branch stem thickness; 8. Phenotype acquisition; The collected soybean plant images are input into the soybean plant phenotype extraction deep learning network model, and the output of the soybean plant phenotype extraction deep learning network model is processed; the multi-stem path planning algorithm and the traditional image processing algorithm are combined to obtain the soybean plant stem-related phenotype; the steps are as follows: 8.1) Deploy the computing environment that the soybean plant phenotype extraction deep learning network model relies on on computing devices; 8.2) Using the processed data obtained in step 1 as input data for a deep learning network model of soybean plant stem-related phenotypes; 8.3) Transplanting the complete soybean plant stem-related phenotype deep learning network model obtained in step 2 into a computing device, and transferring the collected images to the computing device through a hardware interface; 8.4) The result of the deep learning network model of soybean plant stem-related phenotypes after filtering the stem nodes obtained in step 3 is used as a priori condition for phenotype extraction; 8.5) The node detection result obtained in step 3 and the stem segmentation image obtained in step 2 are processed by steps 4, 5, 6 and 7 to obtain the statistical results of the stem phenotype number of the soybean plant main stem nodes and branch nodes, the internode distance, the main stem thickness and the branch stem thickness.

[0012] The enclosed black box is a three-dimensional structure with rulers placed around it. A high-definition camera is placed above the black box. The background color is black, there is a stable light source, and the fixed height is 1.5 meters.

[0013] The beneficial effects of the present invention compared with the prior art are: The method for automatically acquiring soybean stem-related phenotypes based on a deep learning network integrates deep learning and image processing technology. The deep learning technology is used to train the target detection model to identify and extract the stem node features of soybeans, and the semantic segmentation model is trained to identify and extract the stem area in the whole soybean plant image. Then, combined with the image binarization processing, the A* algorithm is designed to perform multi-stem path planning to reconstruct the main stem and each branch, realize the recognition, positioning and counting of soybean stem nodes, and calculate the distance between stem nodes. Finally, the maximum inscribed circle method is used to obtain the main stem thickness and branch stem thickness of the soybean plant, which is suitable for providing massive phenotypic data for breeders. This method overcomes the shortcomings of existing manual methods, sensor detection methods and image processing methods, and realizes the objective automatic phenotypic data acquisition for the whole soybean plant image. This method is not affected by environmental factors, has high efficiency, and the phenotypic information obtained is accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A flow chart for automatically acquiring soybean plant stem-related phenotypes according to the present invention; Figure 2 This is a partial image of the soybean plant image collected in the experiment of the present invention after being annotated; Figure 3 This is a schematic diagram of the soybean plant node feature recognition network model in the present invention; Figure 4 This is a diagram showing the prediction results of the soybean plant node feature recognition network model in the experiment of the present invention; Figure 5 This is a schematic diagram of the soybean plant stem extraction network model in the present invention; Figure 6 It is a stalk segmentation image generated by the soybean plant stalk extraction network model in the experiment of the present invention; Figure 7 The image comparison before and after the stem node automatic filtering module is processed in the experiment of the present invention; Figure 8It is a multi-stem path planning effect diagram based on binarization and stem segmentation image overlay in the experiment of the present invention; Fig. 9 This is a diagram showing the effect of completing the positioning and sorting of stem nodes in the experiment of the present invention; Fig.10 It is a schematic diagram of the stem thickness detection method and the stem internode distance calculation method in the present invention, as well as the image and detection value after completing the phenotypic detection. DETAILED DESCRIPTION

[0015] The design idea of ​​the present invention is to combine deep learning technology with image processing to provide a light, fast and accurate method for acquiring phenotypes, which helps to shorten the breeding process and also opens up ideas for the extraction of soybean stem-related phenotypes.

[0016] There are currently four main ways to obtain the stem-related phenotypes of collected soybean plants: (1) manual method; (2) sensor monitoring method; (3) image processing method. The manual method requires planting or breeding personnel to use professional tools such as vernier calipers to measure and count the number of main stems and branch nodes, internode spacing, and main stem and branch stem thickness of each plant. This method is labor-intensive, inefficient, and easily affected by subjective factors. The sensor detection method uses sensors to conduct real-time detection of soybean plants in a specific area to record the growth status of the plants. It is easily affected by external factors such as temperature and humidity, and has high maintenance costs. Image processing is a new method currently used in automatic detection of crop phenotypes. Generally, a camera is used to take pictures of soybean plants collected by breeders, and a phenotype recognition model is constructed based on the distinguishing characteristics of different types of phenotypes to achieve automatic recognition of phenotypes. The effectiveness of image processing methods is highly dependent on the domain knowledge and laws used in the design of the underlying phenotype recognition model, and has low scalability and robustness. Soybean plants are usually obscured by pods, which makes the detection of stem nodes and the extraction of stems challenging. In addition, there is no effective method for extracting stem thickness in the literature and patents.

[0017] The present invention focuses on the intelligent acquisition of stem phenotypes. By combining target detection, semantic segmentation, path planning algorithms, and traditional image processing methods, an intelligent detection method for soybean plant stem-related phenotypes is proposed. The method can automatically detect stem-related phenotypes such as the number of main stem nodes, the number of branch nodes, the main stem thickness, the branch stem thickness, and the internode spacing on soybean plant images with dense pod occlusion, thereby reducing manual participation, reducing detection costs, and improving detection efficiency and accuracy.

[0018] The following is a detailed description of a specific implementation method of the present invention for automatically acquiring soybean plant stem-related phenotypes based on deep learning, (see Figures 1 to 10 ): A method for automatically acquiring soybean plant stem-related phenotypes based on deep learning, the hardware includes a sealed black box containing mature soybean plants, the sealed black box is a three-dimensional structure, rulers are placed around it, a high-definition camera is placed above the black box, the background color is black, there is a stable light source, and the fixed height is 1.5 meters; the method is characterized in that the method for automatically acquiring soybean plant stem-related phenotypes based on deep learning is implemented by including the following steps, (see Figure 1 ): Step 1, soybean plant image acquisition and preprocessing: After the images of mature soybean plants are transferred to the computer for storage, the acquired image data are labeled using the LabelImg and Labelme tools respectively; and the images are preprocessed and data enhanced; 1.1) Data source: Sample data of mature soybean plants collected using professional equipment. The samples include plants with no branches, single branches, and complex branches.

[0019] Specifically, soybean plant images are collected: soybean testers install a high-definition industrial camera in a sealed black box to obtain images of mature soybean plants. The soybean plants involved are of various morphologies, including images of unbranched soybean plants, single-branched soybean plants, and complex-branched soybean plants. After the images are transferred to a computer for storage, the nodes and stalks of the soybean plants are marked separately.

[0020] 1.2) Data labeling: Label the nodes and stalks of soybean plants separately.

[0021] Specifically, the LabelImg visual image calibration tool and the Labelme image annotation tool are used to label the acquired image data. In the LabelImg annotation, each stem node target area on the image is labeled with a category node1, node2, node3..., and in the Labelme annotation, the stem edge area on the image is continuously labeled stem. The labeled label data is used as the training data set model input data set of the deep learning network (see Figure 2 ).

[0022] 1.3) Image preprocessing: The obtained images are used as the training data set input of the deep learning network, and image enhancement methods such as image size normalization, image translation, rotation, flipping, and mirroring are performed to enhance the data and expand the original soybean plant image data set.

[0023] Step 2, constructing a deep learning network model for soybean phenotype extraction; including constructing a stem node feature recognition network model; constructing a stem extraction network model; using the training data set obtained in step 1 to train the stem node feature recognition network model and the stem extraction network model respectively, so as to achieve the functions of identifying soybean plant nodes and segmenting soybean plant stems; obtaining a stem segmentation image; 2.1) Network model for stem node feature recognition; The stem node feature recognition network model is built by adding the CA attention mechanism to the existing YOLOv5s network (see Figure 3 ); Input the soybean plant image dataset obtained in step 1 into the stem node feature recognition network to obtain a complete stem node feature recognition network model; the specific operations performed are as follows: Input the soybean plant image dataset Input->Backbone layer->Neck layer->Head layer, so that the feature extraction and detection operations can be completed, and the soybean plant stem node prediction map can be obtained; The input image is resized to 640×640×3 using the resize function as the input of the Backbone layer. After a 6*6 convolution operation, the image becomes 320×320×64, and then after a 3*3 convolution block, the image size becomes 160×160×128, and enters the C3 module and a 3*3 convolution block, the size becomes 80×80×256, then after another C3 module, the output is 80×80×256, and then after another 3*3 convolution, it becomes 40×40×512; after the C3 module and 3*3 convolution, it becomes 20×20×1024. After passing through the CA and SPPF layers, the three outputs of the Backbone layer are: the output of the second C3 module from the shallowest to the deepest network is 80×80×256, the output of the second C3 module is 40×40×512, and the output of the SPPF layer is 20×20×1024; In the Backbone layer, the C3 module was proposed in YOLOv5 to replace the CSP module. The C3 module is mainly composed of CBS and Bottleneck Block modules. The CBS structure contains convolutional layers, batch normalization layers, and SiLU activation functions. These components work together to enable the C3 module to effectively process and extract features. The Bottleneck Block structure starts with a 1x1 convolutional layer, followed by a 3x3 convolutional layer, and finally adds the initial input through a shortcut, and integrates the CA attention mechanism before the SPPF module (see Figure 3 ); The CA attention mechanism focuses on both position information and channel information, aggregates features from these two directions, and is suitable for feature extraction of stem nodes. The specific process is as follows: the input of the C3 module reduces the number of channels of the input feature map to 1 / reduction times the original number of channels through a 1x1 convolution operation, performs Batch Normalization and ReLU activation function processing on the reduced number of channels, and performs two 1x1 convolution operations on the reduced number of channels to generate horizontal and vertical attention distributions, respectively. Finally, the horizontal and vertical attention distributions are normalized to values ​​between [0,1] through the Sigmoid function, and the two distributions are used as weights to weight the input feature map. The number of output channels is consistent with the input (see Figure 3 ); The output of Backbone 20×20×1024 is used as the input of the Neck layer. After the 1*1 convolution operation, the output is 20×20×512. After the downsampling operation, it is concat-joined with the 40×40×512 output of Backbone. After the C3 module and the 1*1 convolution operation, 40×40×256 is obtained. After the downsampling operation, it is concat-joined with the output of the Backbone layer 80×80×256 to obtain the first output of the Neck layer 80×80×256. After the 3*3 convolution and the output 40×40×256, the output is used as the second output of the Neck layer after the C3 module. After the 3*3 convolution, 20×20×512 is obtained, and the first output 20×20×512 is concat-joined. After the C3 module, the third output of the Neck layer 20×20×1024 is obtained. The outputs of the Neck layer after the above processing (80×80×256, 40×40×512, and 80×80×256) are input to the Head layer. The Head layer usually contains a series of Conv2d modules to extract the target position and category information from the feature map. The Conv2d module may contain multiple layers of convolution operations to further process the feature map, usually with different convolution kernel sizes and numbers of channels. These Conv2d modules will output the coordinates, category confidence, and category information of the predicted target box. The output results classify and regress each spatial position to generate a predicted box for target detection. Common processing includes applying an activation function (such as sigmoid or softmax) to obtain class probabilities, and applying the bounding box regression algorithm Ahchor-base to predict the location of the bounding box; Input soybean image to train the stem node feature recognition network model, obtain the position of each stem node detection box in the image, and record it in the form of array node1:[0,x 1 ,y 1 ,w1 ,h 1 ],node2:[0,x 2 ,y 2 ,w 2 ,h 2 ],node3:[0,x 3 ,y 3 ,w 3 ,h 3 ]…, where the first bit is the label bit, 0 means the object corresponding to the detection box is a stem node, x i ,y i are the horizontal and vertical coordinates of the upper left corner of the stem node detection frame, w i ,h i are the length and width of the rectangular stem node detection box, respectively (see Figure 4 ); 2.2) Segment the stems and build a stem extraction network model; The stem extraction network model is based on the existing U 2 -Net network based on U 2 -Net network is mainly composed of encoding layer, decoding layer and saliency map fusion module; 2 -Net's encoder is used to extract the feature information of the image and gradually reduce the size of the feature map; the decoder restores the size of the feature map to the size of the original input image and fuses the feature information with the feature map of the corresponding layer of the encoder to produce a high-resolution segmentation result; the encoder adopts an improved structure, including a U-Net structure consisting of two parallel paths and a connection module with a lightweight attention mechanism; these improvements can reduce the amount of computation while maintaining high performance, and are particularly suitable for processing small targets and details; U 2 -Net’s decoder uses a skip connection mechanism to connect the feature maps of different layers in the encoder with the feature maps of the corresponding layers in the decoder to retain more spatial and semantic information; in addition, the decoder also uses deconvolution operations and upsampling techniques to restore the size of the feature map to the size of the original input image. 2 -Net also uses loss functions suitable for image segmentation tasks, usually cross-entropy loss functions, which can measure the difference between the model prediction results and the true labels and guide the model to learn how to perform image segmentation more accurately.

[0024] During the training process, the model parameters are adjusted by training on a large-scale annotated image dataset using optimization algorithms such as stochastic gradient descent to minimize the loss function value. The training process aims to enable the model to learn how to extract features from the input image and perform pixel-level segmentation (see Figure 5 ); The specific process is as follows: input soybean plant image dataset Inputs->encoder level En1->encoder level En2->encoder level En3->encoder level En4->encoder level En5->encoder level En6->decoder level De5->decoder level De4->decoder level De3->decoder level De2->decoder level De1->saliency map fusion module, use 1×1 convolution layer and sigmoid function to generate the final saliency probability map, thus completing the image segmentation operation and obtaining the soybean image stem segmentation map; Composition 2 The most important part of the -Net encoder structure is the residual U module (RSU), and its specific structure is: For the residual module, set the number of input channels, the number of intermediate channels, and the number of output channels 1) 3x3 convolution (input channel, output channel); 1) 3x3 convolution (input channel, output channel), 2) 3x3 convolution (input channel, middle channel) 1 / 2 downsampling, 3) 3x3 convolution (middle channel, middle channel) 1 / 2 downsampling, 4) 3x3 convolution (middle channel, middle channel) 1 / 2 downsampling, 5) 3x3 convolution (middle channel, middle channel) 1 / 2 downsampling, 6) 3x3 convolution (middle channel, middle channel) 1 / 2 downsampling, 7) 3x3 convolution (middle channel, middle channel), 8) 3x3 convolution (middle channel, middle channel), 9) Superimpose 7) and 8) by channel, 3x3 convolution (middle channel, middle channel) upsampled by 2 times, 10) Superimpose 6) and 9) by channel, 3x3 convolution (middle channel, middle channel) upsampled by 2 times, 11) 5) and 10) are stacked by channel, 3x3 convolution (middle channel, middle channel) is upsampled by 2 times, 12) Superimpose 4) and 11) by channel, 3x3 convolution (middle channel, middle channel) upsampled by 2 times, 13) Superimpose 4) and 12) by channel, 3x3 convolution (middle channel, middle channel) upsampled by 2 times, 14) Add 2) and 13) by channel, 3x3 convolution (middle channel, middle channel), and finally add 14) and 1) by normal addition; The decoder level has a similar structure to the encoder level. During the encoding process, each time the output is passed to the next layer, the output is also input to the decoding layer corresponding to the "U" shape. The saliency map is effectively fused with the original image through the saliency map fusion module to produce an enhanced image, making the salient area in the image more prominent or easier to notice. Input the soybean plant image dataset obtained in step 1 into U 2 -Net network to perform relevant feature extraction operations and obtain a complete stem extraction network model; In the stem extraction network model, except for the soybean stem target area, the soybean background area is converted to black, which is equivalent to segmenting the stem target from the original image as a separate individual, referred to as the stem segmentation image.

[0025] Compared with the traditional method of directly binarizing the image, overlay processing can improve the accuracy of phenotype acquisition.

[0026] Input soybean images to train the stalk extraction network model, and obtain the category value of each pixel in the image. The pixel value of the background is 0, and the stalk area of ​​the soybean plant is marked with green. The pixel values ​​of the stalk area are consistent, all 2. (See Figure 6 ); Step 3: Based on the results of the node feature recognition network model and the stem extraction network model, a node automatic filtering module is constructed; based on the results of node detection and stem segmentation, a node automatic filtering module is constructed; the stem segmentation image obtained in step 2 is used to filter out the incorrectly identified node detection frame to improve the detection accuracy of the node frame; 3.1) Through steps 1 and 2, the nodes have been identified from the soybean plant image and marked with node detection frames; and the stem region is segmented from the original image. The image with the marked stem region is called the stem segmentation image; 3.2) Filter candidate stem node detection frames Aiming at the problem that pods block the stems in the images of mature soybean plants, causing some pods to be misidentified as stem nodes, a method is proposed to filter out misidentified stem nodes using the results of semantic segmentation: first, the pixels in the stem area obtained by semantic segmentation are simplified into two categories: stems and background, with 0 representing the background and 2 representing the stem area, and the stem segmentation image is stored in the form of a numpy array with only 0 and 2 elements; secondly, the detected stem node frame information is filtered in turn. If the value of any pixel within the range of the stem node frame in the stem segmentation image is 2, the stem node frame is retained. Otherwise, it is considered that the stem node frame does not pass through the stem and is an invalid stem node. In this way, the stem node frame located in the stem area is filtered out to improve the detection accuracy of the stem node frame (see Figure 7 ); Step 4: Based on the results of stem node filtering and stem segmentation, a stem reconstruction algorithm is constructed; the reconstruction of the main stem and branches is converted into a path finding problem based on a pixel map, and a multi-path planning algorithm based on the A* algorithm is designed to reconstruct the soybean plant stems from the soybean plant image; including converting the original soybean plant image into a binary image, and superimposing it with the stem segmentation image obtained by the stem extraction network segmentation, so that in the area where the semantic segmentation is disconnected, the A* algorithm can still successfully plan the path with the binary area; select all the nodes to be evaluated through the A* algorithm formula (1) The smallest node is the next expansion node; and the main stem and branches are identified respectively by combining the stem node feature recognition network through multi-path planning and the stem node detection frame information to obtain the main stem and branches of the soybean plant; Specifically, the detection of the number of nodes, internode spacing and stem thickness requires first identifying the main stem and branches from the soybean plant image, that is, reconstructing the stem. The present invention proposes: converting the reconstruction of the main stem and branches into a path-finding problem based on a pixel map, integrating the node detection results, the stem segmentation image and the binary image, and designing a multi-path planning algorithm based on the A* algorithm to reconstruct the plant stem from the soybean plant image, including the main stem and each branch of the plant; based on the reconstructed main stem and each branch, the number of nodes, internode spacing and stem thickness phenotype are obtained (see Figure 8 ).

[0027] 4.1) Construct path planning pixel map; The original plant image is converted into a binary image and superimposed with the stem segmentation image obtained by the stem extraction network segmentation; the pixel points with a value of 2 in the stem segmentation image are marked as 1, and the pixel points with a value of 0 in the binary soybean plant image and a value of 0 in the stem segmentation image are marked as 0; thereby, a pixel map representing the stem area is constructed; the map is the input of the path planning algorithm; The path planning algorithm uses the overlay image, which has good robustness: it will not cause the path planning to deviate from the stem area due to a single binary image, and it can also make up for the stem area segmentation model. In images where the stem area segmentation is insufficient and the local stem area is disconnected, the connectivity of the stem area can be effectively improved by overlaying the binary image, so that the path planning algorithm can still successfully plan the path. (See Figure 8 ); 4.2) Combine the stem node detection frame information to determine the starting and ending points of the path planning, build a multi-stem path planning algorithm, and reconstruct the main stem and branches of the plant; Aiming at the problem that there are many pixels in the stem area of ​​soybean plants, which leads to low efficiency of path planning, the present invention proposes: combining the stem node detection frame information obtained by the stem node feature extraction network, and using the path planning algorithm to reconstruct the main stem and branches of the plant; The process of the multi-stem path planning algorithm is as follows: first, the topmost and bottommost stem node detection frames in the plant image are used as the starting point and end point of the main stem path respectively; in general, the top of the main stem is the highest point of the entire plant, so the topmost stem node detection frame is used as the starting point of the main stem path; the path planning algorithm is executed once to obtain the path between the starting point and the end point, and the obtained path is used as the main stem path; then, the image is scanned from top to bottom, and the topmost stem node detection frame in each branch stem is detected in turn, and it is used as the new starting point of the path planning in turn, and the path planning algorithm is executed once to obtain the path between the new starting point and the end point, and the obtained path is used as the branch path; the intersection of the branch path and the main stem path is calculated, and the point is recorded, and it is used as the end point of the branch path, and the end point of the branch path is updated; repeat the above process, scan each stem node detection frame in the plant from top to bottom, until all stem node detection frames belong to a branch; only one main stem path can be obtained in a plant, and multiple branch paths can be obtained; the obtained main stem path is used as the main stem of the plant, and the obtained multiple branch paths are used as the branches of the plant; The A* algorithm is an algorithm that combines the heuristic search algorithm and the depth-first search algorithm. It specifies the starting point and end point of the path and uses heuristic information to plan the path with the minimum overall expected cost. The basis for the A* algorithm to plan the path is shown in Formula 1:

[0028] In formula (1), Indicates passing through the node from the starting point The estimated cost to reach the destination, From the starting point to the node The actual cost of the optimal path, Represents a slave node The estimated cost of the optimal path to the destination, selected from all nodes to be evaluated The node with the smallest value is used as the next expansion node; The specific process of the multi-stalk path planning algorithm is as follows: Since the plants were placed in a standardized manner during the shooting, the first stem node detection frame obtained by scanning downward from the top of the image was defined as the starting point of the main stem, and the stem node detection frame at the bottom of the plant in the image was defined as the end point of the main stem. That is, the stem node detection frames were sorted in ascending order of the ordinate, and the center points of the stem node detection frames with the minimum and maximum ordinate values ​​were taken [x 1 ,y_min],[x 2 ,y_max] as the starting point and end point of the path planning algorithm; Before starting path planning, add an initial flag node1 to the representation of all stem node detection boxes: [x 1 ,y 1 ,w 1 ,h1 ,0],node2:[x 2 ,y 2 ,w 2 ,h 2 ,0],node3:[x 3 ,y 3 ,w 3 ,h 3 ,0]…, indicating that the stem node detection frame has not been visited; after the path planning algorithm is executed for one round and the main stem path is obtained, the flag bits of all the stem node detection frames on the main stem are changed to 1, indicating that the stem node detection frame has been visited by the main stem path, that is, the information of the stem node detection frame located on the main stem at this time is node1:[x 1 ,y 1 ,w 1 ,h 1 ,1],node2:[x 2 ,y 2 ,w 2 ,h 2 ,1],node3:[x 3 ,y 3 ,w 3 ,h 3 ,1]…; Scan the image downward, take the first unvisited stem node detection frame scanned as the starting point of the first branch, and plan the path again with the end point to obtain a branch path; record the intersection of the branch path and the main stem path, and take the intersection as the end point of the branch path; thus, obtain the path of the first branch, modify the flag bit of all stem node detection frames on the branch path to 2, indicating that the stem node detection frame has been visited and belongs to the first branch; The acquisition process of the remaining branch paths is similar to that of the first branch path. The flag bit values ​​of the stem node detection frames are incremented in sequence. For example, the flag bit value of the stem node detection frame on the second branch is 3. Repeat the above process until the marked bits of all stem node detection frames are not 0, that is, all stem node detection frames have been visited and belong to the main stem or a branch. After completion, record the morphological characteristics of the nodes on the main stem and branches, that is, record from the bottom of the stem to the top of the stem, that is: The node detection box node1 on the main stem: [x 1 ,y 1 ,w 1 ,h 1 ,1],node2:[x 2 ,y 2 ,w 2 ,h 2 ,1],node3:[x 3 ,y 3,w 3 ,h 3 ,1]…; The node detection box node1 on branch 1: [x 1 ,y 1 ,w 1 ,h 1 ,2],node2:[x 2 ,y 2 ,w 2 ,h 2 ,2],node3:[x 3 ,y 3 ,w 3 ,h 3 ,2]…; The node detection box node1 on branch 2: [x 1 ,y 1 ,w 1 ,h 1 ,3],node2:[x 2 ,y 2 ,w 2 ,h 2 ,3],node3:[x 3 ,y 3 ,w 3 ,h 3 ,3]…; Among them, node1 in each branch is the stem node detection frame at the bottom of the stem (see Fig. 9 ).

[0029] Step 5: According to the result of stem reconstruction, the stem node detection frame is located and a stem node counting method is constructed; the stem node detection frames belonging to the main stem and the branches are distinguished, and the stem node detection frames are counted separately to obtain the phenotypes of the main stem node number and the branch node number; The node number phenotype is divided into the main node number and the branch node number; after obtaining the main stem path and the branch path, the node detection frames with the same flag value are located on the same stem, thereby completing the positioning of the node detection frame; if the node detection frame flag value is 1, it means that the node detection frame is located on the main stem of the plant, and the node detection frame flag value is 2, it means that the node detection frame is located on the first branch of the plant; the node detection frame information with the flag is stored in array b, the flag values ​​of all node detection frames are grouped, the number of node detection frame elements belonging to each group is calculated, the node number phenotype of the main stem and each branch is obtained, and the node counting is realized (see Fig. 9 ).

[0030] Step 6, calculating the internode spacing, calculation method: calculating the internode spacing for the main stem and the branch respectively; including calculating the Euclidean distance between the center points of the adjacent node detection frames belonging to the same branch or main stem by using the Euclidean distance calculation formula (2) as the internode spacing; According to the positioning results of the stem node detection frames, the stem node detection frames are sorted and a method for calculating the stem node spacing is constructed; In the stem node detection frame information array b, each element is composed of [x, y, w, h, n], which respectively represents the center coordinates (x, y), width (w), height (h) and stem number (n) of the stem node detection frame; the stem node detection frames with the same n value belong to the same stem, for example, they all belong to the same branch or the main stem; the stem node detection frame information is stored in the array according to the n value: b 1 , b 2 , …, b k+1 Record the node detection frame information on the main stem, the first branch, and the kth branch respectively; sort the node detection frames in each array in ascending order according to their y-coordinate values; after sorting, the adjacent node detection frames in the array are the adjacent nodes in the actual stem; calculate the Euclidean distance d between the center points of adjacent node detection frames 1 , d 2 , …, d j-1 , and obtain a set of internode spacing values, which are stored in array c 1 , c 2 , …, c k+1 In the , record the internode distances on the main stem, the first branch, and the kth branch, respectively (see Fig.10 ).

[0031] The calculation formula of Euclidean distance is shown in formula (2).

[0032]

[0033] Step 7: According to the sorting results of the node detection frames and the results of the stem region extraction, a stem thickness calculation method is constructed. Based on the stem region extracted by the semantic segmentation network and the results of node identification and sorting, a maximum inscribed circle method is designed to calculate the stem thickness, which is suitable for the detection and calculation of the main stem thickness and the branch stem thickness; The automatic calculation of stem thickness needs to be combined with the sorting results of the stem node detection frame and the results of stem area extraction; the stem thickness is divided into the main stem thickness and the branch stem thickness; taking the main stem thickness as an example, it is necessary to obtain the stem thickness between the 5th stem node and the 6th stem node from the cotyledon node to the top of the main stem as the prediction value; based on the results of stem area extraction and stem node identification and sorting obtained by the aforementioned semantic segmentation network, the maximum inscribed circle method is designed to detect and calculate the main stem thickness and branch stem thickness; Taking the main stem thickness as an example, draw the largest inscribed circle of the stem area between the 5th and 6th stem nodes, and calculate the diameter of the inscribed circle. d , as the predicted value of the main stem thickness; First, the Canny operator is used to detect the edge of the target stem area and obtain the edge contour of the target stem area. The Canny operator uses the gradient change of pixels. The more obvious the gradient change is, the greater the difference between two pixels. In the stem area segmentation map, the difference between the stem area and the background color causes a large gradient change at the edge. The Canny operator is used to accurately obtain the edge contour of the stem. The calculation of the gradient is divided into: In two directions, the calculation method is shown in Formula 3, where: represents the gradient value of the point, is the two-dimensional coordinate of the point;

[0034] Use the contour detection Canny function in OpenCV to find the contour of the target stem internode area. After completing the gradient calculation, use the non-maximum suppression method to retain the edge pixels with the largest gradient change. Then, from array b 1 Find the information of the main stem node 5 and node 6 detection frames in node5: [x 5 ,y 5 ,w 5 ,h 5 ,1] and node6:[x 6 ,y 6 ,w 6 ,h 6 ,1], respectively calculate the horizontal and vertical coordinates of the center points of the two stem node detection frames, and the calculation formulas are as follows:

[0035] in, and Represents the stem node detection frame The average of the coordinates of the center points of the two stem node detection frames is used as the target position for measuring stem thickness, that is, the stem thickness measurement point, recorded as and , the calculation formula is as follows: Formula 6 and Formula 7:

[0036] Use OpenCV's pointPolygonTest function to determine the stem thickness measurement point Is it within the target stem area edge contour obtained above? If the stem thickness measurement point is above the edge of the area, it returns 0; Outside the edge of the area, a negative value is returned; if the stem thickness measurement point Return a positive value within the edge contour of the region; when the return value is positive, calculate the shortest distance between all pixel points within the edge contour of the target stalk region and the edge contour of the target stalk region, and store it in the matrix m; find the maximum value in the matrix m through the minMaxLoc function, as the radius of the maximum inscribed circle, find the position corresponding to the maximum value, as the center of the maximum inscribed circle; input the radius and center into the Circle function, and draw the maximum inscribed circle of the edge contour of the target stalk region on the stalk region segmentation image; calculate the scale according to the standard reference object next to the soybean plant image, and convert the maximum inscribed circle diameter into the actual stem thickness; at this point, the prediction of the main stem thickness is completed; The method for obtaining the branch stem diameter is similar to that for the main stem; the difference is that the target position for measuring the branch stem diameter is between the first and second stem nodes of the branch; for example, for the i-th branch, take node1[x 1 ,y 1 ,w 1 ,h 1 ,i+1] and node2[x 2 ,y 2 ,w 2 ,h 2 ,i+1] is the target position for measuring the branch stem thickness (see Fig.10 ).

[0037] Step 8: Phenotype acquisition The collected images are input into the soybean plant phenotype extraction deep learning network model, and the output of the soybean plant phenotype extraction deep learning network model is processed; the multi-stem path planning algorithm and the traditional image processing algorithm are combined to obtain the soybean plant stem-related phenotype, (see Figure 1 ).

[0038] 8.1) Deploy the computing environment that the soybean plant phenotype extraction deep learning network model relies on on computing devices; 8.2) Using the processed data obtained in step 1 as input data for a deep learning network model of soybean plant stem-related phenotypes; 8.3) Transplanting the complete soybean plant stem-related phenotype deep learning network model obtained in step 2 into a computing device, and transferring the collected images to the computing device through a hardware interface; 8.4) The result of the deep learning network model of soybean plant stem-related phenotypes after filtering the stem nodes obtained in step 3 is used as a priori condition for phenotype extraction; 8.5) The node detection result obtained in step 3 and the stem segmentation image obtained in step 2 are processed by steps 4, 5, 6 and 7 to obtain the statistical results of the stem phenotype number of the soybean plant main stem nodes and branch nodes, the internode distance, the main stem thickness and the branch stem thickness.

[0039] The method for automatically acquiring stem-related phenotypes of soybean plants based on deep learning networks integrates deep learning and image processing technologies. For the whole plant image of mature soybeans, combined with the morphological characteristics of soybean plants, the target detection model is trained by deep learning technology to identify stem nodes, the semantic segmentation model is trained to extract stem regions, and the stem region extraction results are combined to optimize the stem node detection results. Combined with image binarization processing, a multi-path planning algorithm is designed to reconstruct the main stem and branches, and the main stem and branches are respectively located, sorted, counted, and the stem node spacing is calculated. The maximum inscribed circle method is used to obtain the main stem stem diameter and branch stem diameter, and the stem-related phenotypes of the whole plant of mature soybeans are automatically acquired. It is suitable for providing massive phenotypic data for breeders. This method overcomes the shortcomings of the existing manual detection method, sensor detection method, and image processing method in collecting phenotypic data of the whole soybean plant, and has the characteristics of objectivity, accurate acquisition of phenotypic data, automaticity, high efficiency, and freedom from interference from natural environmental factors.

Claims

1. A method for automatically acquiring soybean plant stem-related phenotypes based on deep learning, the hardware includes a sealed black box containing mature soybean plants, the sealed black box is a three-dimensional structure, a ruler is placed around it, a high-definition camera is placed above the black box, the background color is black, there is a stable light source, and the fixed height is 1.5 meters; It is characterized in that The method for automatically acquiring soybean plant stem-related phenotypes based on deep learning is implemented by comprising the following steps: Step 1, soybean plant image acquisition and preprocessing: After the images of mature soybean plants are transferred to the computer for storage, the acquired image data are labeled using the LabelImg and Labelme tools respectively; and the images are preprocessed and data enhanced; Step 2, constructing a deep learning network model for soybean phenotype extraction; including constructing a stem node feature recognition network model; constructing a stem extraction network model; using the training data set obtained in step 1 to train the stem node feature recognition network model and the stem extraction network model respectively, so as to achieve the functions of identifying soybean plant nodes and segmenting soybean plant stems; obtaining a stem segmentation image; Step 3: construct a stem node automatic filtering module based on the results of stem node detection and stem segmentation; filter the incorrectly identified stem node detection frames from the stem segmentation image obtained in step 2 to improve the detection accuracy of the stem node frames; Step 4: Based on the results of stem node filtering and stem segmentation, a stem reconstruction algorithm is constructed; the reconstruction of the main stem and branches is converted into a path finding problem based on a pixel map, and a multi-path planning algorithm based on the A* algorithm is designed to reconstruct the soybean plant stems from the soybean plant image; The process includes converting the original soybean plant image into a binary image and superimposing it with the stalk segmentation image obtained by the stalk extraction network segmentation, so that in the area where the semantic segmentation is disconnected, the A* algorithm can still successfully plan the path with the binary area; selecting all the nodes to be evaluated through the A* algorithm formula (1) The smallest node is the next expansion node; and the main stem and branches are identified respectively by combining the stem node feature recognition network through multi-path planning and the stem node detection frame information to obtain the main stem and branches of the soybean plant; Step 5: According to the result of stem reconstruction, the stem node detection frame is located and a stem node counting method is constructed; the stem node detection frames belonging to the main stem and the branches are distinguished, and the stem node detection frames are counted separately to obtain the phenotypes of the main stem node number and the branch node number; Step 6: sort the node detection frames according to the positioning results of the node detection frames, and construct a node spacing calculation method; calculate the node spacing for the main stem and branches respectively; including calculating the Euclidean distance between the center points of adjacent node detection frames as the node spacing for node detection frames belonging to the same branch or main stem using the Euclidean distance calculation formula (2); Step 7: construct a stem thickness calculation method according to the sorting results of the stem node detection frames and the results of the stem region extraction; Based on the results of stem area extraction and stem node identification and sorting obtained by the semantic segmentation network, the maximum inscribed circle method is designed to calculate stem thickness, which is suitable for the detection and calculation of main stem thickness and branch stem thickness. Step 8, phenotype acquisition, input the collected image into the soybean plant phenotype extraction deep learning network model, and perform the above-mentioned data processing on the output of the soybean plant phenotype extraction deep learning network model; combine the multi-stem path planning algorithm and the traditional image processing algorithm to obtain the soybean plant stem-related phenotype.

2. The method for automatically acquiring soybean plant stem-related phenotypes based on deep learning according to claim 1, characterized in that: In the step 1, in the LabelImg annotation, each stem node target area on the image of the zero-branched soybean plant, the single-branched soybean plant and the complex-branched soybean plant is labeled with a category node1, node2, node3..., and in the Labelme annotation, the stem edge area on the image is continuously labeled with stem, and the labeled label data is used as a training data set for the deep learning network; the above-obtained image is input as a training data set for the deep learning network, and data enhancement is performed by image enhancement methods such as image size normalization, image translation, rotation, flipping, and mirroring to expand the original soybean plant image data set.

3. The method for automatically acquiring soybean plant stem-related phenotypes based on deep learning according to claim 1, characterized in that: The step 2 includes 2.1) a stem node feature recognition network model and 2.2) a stem extraction network model; 2.1) Network model for stem node feature recognition The stem node feature recognition network model is constructed by adding the CA attention mechanism to the existing YOLOv5s network. The soybean plant image dataset obtained in step 1 is input into the stem node feature recognition network to obtain a complete stem node feature recognition network model. The specific operations performed are as follows: Input the soybean plant image dataset Input->Backbone layer->Neck layer->Head layer to complete the feature extraction and detection operations and obtain the soybean plant node prediction map; The input image is resized to 640×640×3 using the resize function as the input of the Backbone layer. After a 6*6 convolution operation, the image becomes 320×320×64, and then after a 3*3 convolution block, the image size becomes 160×160×128, enters the C3 module and a 3*3 convolution block, and the size becomes 80×80×256. At this time, after passing through a C3 module, the output is 80×80×256, and then after a 3*3 convolution and C3 module, it becomes 40×40×512, and then after a 3*3 convolution, it becomes 20×20×1024. After passing through the CA and SPPF layers, the three outputs of the Backbone layer are: the output of the second C3 module from the shallowest to the deepest network is 80×80×256, the output of the second C3 module is 40×40×512, and the output of the SPPF is The output of the layer is 20×20×1024; in the Backbone layer, the C3 module was proposed in YOLOv5 to replace the CSP module. The C3 module is mainly composed of CBS and Bottleneck Block modules. The CBS structure contains convolutional layers, batch normalization layers, and SiLU activation functions. These components work together to enable the C3 module to effectively process and extract features. The Bottleneck Block structure starts with a 1x1 convolution layer, followed by a 3x3 convolution layer, and finally adds the initial input through a shortcut, and integrates the CA attention mechanism before the SPPF module; The CA attention mechanism focuses on both position information and channel information, aggregates features from these two directions, and is suitable for feature extraction of stem nodes. The specific process is as follows: the 1x1 convolution operation of the C3 module reduces the number of channels of the input feature map to 1 / reduction times of the original number of channels, performs Batch Normalization and ReLU activation function processing on the reduced number of channels, and performs two 1x1 convolution operations on the reduced number of channels to generate horizontal and vertical attention distributions, respectively. Finally, the horizontal and vertical attention distributions are normalized to values ​​between [0,1] through the Sigmoid function, and the two distributions are used as weights to weight the input feature map. The number of output channels is consistent with the input. The output of Backbone 20×20×1024 is used as the input of the Neck layer. After the 1*1 convolution operation, the output is 20×20×512. After the downsampling operation, it is concat-joined with the 40×40×512 output of Backbone. After the C3 module and the 1*1 convolution operation, 40×40×256 is obtained. After the downsampling operation, it is concat-joined with the output of the Backbone layer 80×80×256 to obtain the first output of the Neck layer 80×80×256. After the 3*3 convolution and the output 40×40×256, the output is used as the second output of the Neck layer after the C3 module. After the 3*3 convolution, 20×20×512 is obtained, and the first output 20×20×512 is concat-joined. After the C3 module, the third output of the Neck layer 20×20×1024 is obtained. The outputs of the Neck layer after the above processing (80×80×256, 40×40×512, and 80×80×256) are input to the Head layer. The Head layer usually contains a series of Conv2d modules to extract the target position and category information from the feature map. The Conv2d module may contain multiple layers of convolution operations to further process the feature map, usually with different convolution kernel sizes and numbers of channels. These Conv2d modules will output the coordinates, category confidence, and category information of the predicted target box. The output results classify and regress each spatial position to generate a predicted box for target detection. Input soybean image to train the node feature recognition network model, obtain the position of each node in the image and the total number of nodes, and record them in array form node1:[0,x1,y1,w1,h1],node2:[0,x2,y2,w2,h2],node3:[0,x3,y3,w3,h3]…, where the first bit is the label bit, 0 means the detection box corresponds to the node, x i ,y i are the horizontal and vertical coordinates of the center point of the detection frame, w i ,h i are the length and width of the rectangular stem node detection frame respectively; 2.2) Stem Extraction Network Model The stem extraction network model is based on the existing U 2 -Net network, which is mainly composed of encoding layer, decoding layer and saliency map fusion module; 2 -Net's encoder is used to extract the feature information of the image and gradually reduce the size of the feature map; the decoder restores the size of the feature map to the size of the original input image and fuses the feature information with the feature map of the corresponding layer of the encoder to produce a high-resolution segmentation result; the encoder adopts an improved structure, including a U-Net structure consisting of two parallel paths and a connection module with a lightweight attention mechanism; these improvements achieve high performance while reducing the amount of computation, and are particularly suitable for processing small targets and details; U 2 -Net's decoder uses a skip connection mechanism to connect the feature maps of different layers in the encoder with the feature maps of the corresponding layers in the decoder, retaining more spatial and semantic information; the decoder restores the size of the feature map to the size of the original input image through deconvolution operations and upsampling techniques; U 2 -Net also uses the cross entropy loss function suitable for image segmentation tasks to measure the difference between the model prediction results and the true labels, and guide the model to learn how to perform image segmentation more accurately; During the training process, the model parameters are adjusted by training on a large-scale annotated image dataset using optimization algorithms such as stochastic gradient descent to minimize the loss function value. At the same time, the training process aims to enable the model to learn how to extract features from the input image and perform pixel-level segmentation. Input the soybean plant image dataset Inputs->Encoder level En1->Encoder level En2->Encoder level En3->Encoder level En4->Encoder level En5->Encoder level En6->Decoder level De5->Decoder level De4->Decoder level De3->Decoder level De2->Decoder level De1->Saliency map fusion module, use 1×1 convolution layer and sigmoid function to generate the final saliency probability map, so as to complete the image segmentation operation and obtain the soybean image stem segmentation map; Composition 2 The most important part of the -Net encoder structure is the residual U module (RSU), and its specific structure is: For the residual module, set the number of input channels, the number of intermediate channels, and the number of output channels; 1) 3x3 convolution (input channel, output channel), 2) 3x3 convolution (input channel, middle channel) 1 / 2 downsampling, 3) 3x3 convolution (middle channel, middle channel) 1 / 2 downsampling, 4) 3x3 convolution (middle channel, middle channel) 1 / 2 downsampling, 5) 3x3 convolution (middle channel, middle channel) 1 / 2 downsampling, 6) 3x3 convolution (middle channel, middle channel) 1 / 2 downsampling, 7) 3x3 convolution (middle channel, middle channel), 8) 3x3 convolution (middle channel, middle channel), 9) Superimpose 7) and 8) by channel, 3x3 convolution (middle channel, middle channel) upsampled by 2 times, 10) Superimpose 6) and 9) by channel, 3x3 convolution (middle channel, middle channel) upsampled by 2 times, 11) 5) and 10) are stacked by channel, 3x3 convolution (middle channel, middle channel) is upsampled by 2 times, 12) Superimpose 4) and 11) by channel, 3x3 convolution (middle channel, middle channel) upsampled by 2 times, 13) Superimpose 4) and 12) by channel, 3x3 convolution (middle channel, middle channel) upsampled by 2 times, 14) Add 2) and 13) by channel, 3x3 convolution (middle channel, middle channel), and finally add 14) and 1) by normal addition; The decoder level has a similar structure to the encoder level. During the encoding process, each time the output is passed to the next layer, the output is also input to the decoding layer corresponding to the "U" shape. The saliency map fusion module effectively fuses the saliency map with the original image to produce an enhanced image, making the salient area in the image more prominent or easier to notice. The soybean plant image dataset obtained in step 2 is input into the stem extraction network U 2 -Net to perform relevant feature extraction operations to obtain a complete stem segmentation model; In the stem extraction model, in the input image, except for the soybean stem target area, the soybean background area is converted into black, which is equivalent to segmenting the stem target from the original image as a separate individual, referred to as the stem segmentation image; and the image is superimposed with the stem segmentation image and the binary image in step 4. Compared with the traditional method of directly binarizing the image, the superposition process can improve the accuracy of phenotypic acquisition; The soybean image is input to train the stem extraction network model to obtain the category value to which each pixel in the image belongs. The pixel value of the background is 0, and the value of all pixels in the soybean plant stem area is consistent with 2.

4. The method for automatically acquiring soybean plant stem-related phenotypes based on deep learning according to claim 1, characterized in that: The step 3 includes 3.1) obtaining a stem segmentation image and 3.2) filtering candidate stem node detection frames; 3.1) Through steps 1 and 2, the nodes have been identified from the soybean plant image and marked with node detection frames; and the stem region is segmented from the original image. The image with the marked stem region is called the stem segmentation image; 3.2) To address the problem of pods blocking the stems in images of mature soybean plants, causing some pods to be misidentified as stem nodes, it is proposed to use the results of semantic segmentation to filter out misidentified stem nodes: first, the pixels in the stem area obtained by semantic segmentation are simplified into two categories: stems and background, with 0 representing the background and 2 representing the stem area. The stem segmentation image is stored in the form of a numpy array with only 0 and 2 elements; secondly, the detected stem node frame information is filtered in turn. If any pixel value within the range of the detection frame is 2 in the stem segmentation image, the stem node frame is retained. Otherwise, it is considered that the stem node frame does not pass through the stem and is an invalid stem node. In this way, the stem node frame located in the stem area is filtered out to improve the detection accuracy of the stem node frame.

5. The method for automatically acquiring soybean plant stem-related phenotypes based on deep learning according to claim 1, characterized in that: The step 4 includes 4.1) constructing a path planning pixel map; converting the original soybean plant image into a binary image, and superimposing the image with the stem segmentation image obtained by the stem extraction network segmentation, so that in the area where the semantic segmentation is disconnected, the path planning algorithm can still successfully plan the path with the binary area; 4.2) designing a multi-stem path planning algorithm based on the A* algorithm formula (1), combining the stem node detection frame information, and using the A* algorithm to realize multi-path planning, and each obtained path corresponds to a main stem or branch; 4.1) Construct path planning pixel map; The specific process is as follows: the original soybean plant image is converted into a binary image, and the image is superimposed with the stem segmentation image obtained by the stem extraction network segmentation, and the pixel points with a value of 2 in the stem segmentation image are marked as 1, and the pixel points with a value of 0 in the binary soybean plant image and a value of 0 in the stem segmentation image are marked as 0; thereby, a pixel map representing the stem area is constructed; This map is the input to the path planning algorithm; 4.2) Construct a multi-stem path planning algorithm to reconstruct the main stem and branches; The specific process is as follows: first, the topmost and bottommost stem node detection frames in the plant image are used as the starting point and end point of the main stem path respectively; Generally, the top of the main stem is the highest point of the entire plant, so the topmost stem node detection frame is used as the starting point of the main stem path; execute a path planning algorithm to obtain the path between the starting point and the end point, and use the obtained path as the main stem path; then, scan the image from top to bottom, detect the topmost stem node detection frame in each branch stem in turn, use it as the new starting point for path planning in turn, execute a path planning algorithm to obtain the path between the new starting point and the end point, and use the obtained path as the branch path; calculate the intersection of the branch path and the main stem path, record the point, use it as the end point of the branch path, and update the end point of the branch path; repeat the above process, scan each stem node detection frame in the plant from top to bottom, until all stem node detection frames belong to a branch; only one main stem path can be obtained in a plant, and multiple branch paths can be obtained; the obtained main stem path is used as the main stem of the plant, and the obtained multiple branch paths are used as the branches of the plant; The basis for A* algorithm to plan the path is shown in formula (1): ； In formula (1), Indicates passing through the node from the starting point The estimated cost to reach the destination, From the starting point to the node The actual cost of the optimal path, Represents a slave node The estimated cost of the optimal path to the destination, selected from all nodes to be evaluated The smallest node is used as the next expansion node; Since the plants were placed in a standardized manner during shooting, the first stem node detection frame obtained by scanning downward from the top of the image was defined as the starting point of the main stem, and the stem node detection frame at the bottom of the plant in the image was defined as the end point of the main stem. That is, the stem node detection frames were sorted in ascending order of the ordinates, and the center points of the detection frames with the minimum and maximum ordinate values ​​[x1, y_min] and [x2, y_max] were taken as the starting point and end point of the path planning algorithm respectively; Before starting path planning, an initial flag is added to the representation of all stem node detection frames: node1: [x1, y1, w1, h1, 0], node2: [x2, y2, w2, h2, 0], node3: [x3, y3, w3, h3, 0]…, indicating that the stem node detection frame has not been visited; after the path planning algorithm is executed for one round and the main stem path is obtained, the flag of all stem node detection frames on the main stem is changed to 1, indicating that the stem node detection frame has been visited by the main stem path, that is, the information of the stem node detection frame on the main stem at this time is node1: [x1, y1, w1, h1, 1], node2: [x2, y2, w2, h2, 1], node3: [x3, y3, w3, h3, 1]…; Scan the image downward, take the first unvisited stem node detection frame scanned as the starting point of the first branch, and plan the path again with the end point to obtain a branch path; record the intersection of the branch path and the main stem path, and take the intersection as the end point of the branch path; thus, obtain the path of the first branch, modify the flag bit of all stem node detection frames on the branch path to 2, indicating that the stem node detection frame has been visited and belongs to the first branch; The acquisition process of the remaining branch paths is similar to that of the first branch path. The flag bit values ​​of the stem node detection frames are incremented in sequence. For example, the flag bit value of the stem node detection frame on the second branch is 3. Repeat the above process until the marked bits of all stem node detection frames are not 0, that is, all stem node detection frames have been visited and belong to the main stem or a branch. After completion, record the morphological characteristics of the nodes on the main stem and branches, that is, record from the bottom of the stem to the top of the stem, that is: The node detection box on the main stem is node1: [x1, y1, w1, h1, 1], node2: [x2, y2, w2, h2, 1], node3: [x3, y3, w3, h3, 1]…; The node detection box on branch 1 is node1: [x1, y1, w1, h1, 2], node2: [x2, y2, w2, h2, 2], node3: [x3, y3, w3, h3, 2]…; The node detection box on branch 2 node1: [x1, y1, w1, h1, 3], node2: [x2, y2, w2, h2, 3], node3: [x3, y3, w3, h3, 3]…; Among them, node1 in each branch is the stem node detection frame at the bottom of the stem.

6. The method for automatically acquiring soybean plant stem-related phenotypes based on deep learning according to claim 1, characterized in that: The step 5 is to construct a stem node counting method; The number of nodes is divided into the number of main nodes and the number of branch nodes. After obtaining the main stem path and the branch path, the node detection frames with the same flag value are located on the same stem, thereby completing the positioning of the node detection frame. For example, if the node detection frame flag value is 1, it means that the node detection frame is located on the main stem of the plant, and if the node detection frame flag value is 2, it means that the node detection frame is located on the first branch of the plant. The node detection frame information with the flag is stored in array b, the flag values ​​of all node detection frames are grouped, the number of node detection frame elements belonging to each group is calculated, the node number phenotype of the main stem and each branch is obtained, and the node counting is realized.

7. The method for automatically acquiring soybean plant stem-related phenotypes based on deep learning according to claim 1, characterized in that: The step 6 is to construct a method for calculating internode distance; In the stem node detection frame information array b, each element is composed of [x, y, w, h, n], which respectively represents the center coordinates (x, y), width (w), height (h) and stem number (n) of the stem node detection frame; the stem node detection frames with the same n value belong to the same stem, for example, they all belong to the same branch or the main stem; the stem node detection frame information is stored in the array according to the n value: b1, b2, ..., b k+1 Record the node detection frame information on the main stem, the first branch, and the kth branch respectively; sort the node detection frames in each array in ascending order according to their ordinate y values; after sorting, the adjacent node detection frames in the array are the adjacent nodes in the actual stem; calculate the Euclidean distances d1, d2, ..., d between the center points of adjacent node detection frames j-1 , a set of internode spacing values ​​are obtained and stored in arrays c1, c2, ..., c k+1 In the data, the internode distances on the main stem, the first branch, and the kth branch were recorded respectively; The Euclidean distance formula is shown in formula (2); 。 8. The method for automatically acquiring soybean plant stem-related phenotypes based on deep learning according to claim 1, characterized in that: The step 7, constructing a stem thickness counting method; The automatic calculation of stem thickness needs to be combined with the sorting results of the stem node detection frame and the results of stem area extraction; the stem thickness is divided into the main stem thickness and the branch stem thickness. Taking the main stem thickness as an example, it is necessary to obtain the stem thickness between the fifth and sixth stem nodes from the cotyledon node to the top of the main stem as the prediction value. Based on the results of stem area extraction and stem node identification and sorting obtained by the aforementioned semantic segmentation network, the maximum inscribed circle method is designed to detect and calculate the main stem thickness and branch stem thickness; Taking the main stem thickness as an example, draw the largest inscribed circle of the stem area between the 5th and 6th stem nodes, and calculate the diameter of the inscribed circle. d , as the predicted value of the main stem thickness; First, the Canny operator is used to detect the edge of the target stem area and obtain the edge contour of the target stem area. The Canny operator uses the gradient change of pixels. The more obvious the gradient change is, the greater the difference between two pixels. In the stem area segmentation map, the difference between the stem area and the background color causes a large gradient change at the edge. The Canny operator is used to accurately obtain the edge contour of the stem. The calculation of the gradient is divided into Two directions, as shown in the following formula (3). represents the gradient value at this point, is the two-dimensional coordinate of the point; ； Use the contour detection Canny function in OpenCV to find the contour of the target stem internode area. After completing the gradient calculation, use the non-maximum suppression method to retain the edge pixels with the largest gradient change. Then, find the main stem node 5 and node 6 detection frame information node5: [x5, y5, w5, h5, 1] ​​and node6: [x6, y6, w6, h6, 1] from array b1, and calculate the horizontal and vertical coordinates of the center points of the two node detection frames respectively. The calculation formulas are as follows: ； in, and The average value of the coordinates of the two center points of the stem node detection frame is taken as the target position for measuring stem thickness, i.e., the stem thickness measurement point, denoted as and , the calculation formula is as follows: ； Use OpenCV's pointPolygonTest function to determine the stem thickness measurement point Is it within the target stem area edge contour obtained above? If the stem thickness measurement point is above the edge of the area, it returns 0; Outside the edge of the area, a negative value is returned; if the stem thickness measurement point Return a positive value within the edge contour of the region; when the return value is positive, calculate the shortest distance between all pixel points within the edge contour of the target stalk region and the edge contour of the target stalk region, and store it in the matrix m; find the maximum value in the matrix m through the minMaxLoc function, as the radius of the maximum inscribed circle, find the position corresponding to the maximum value, as the center of the maximum inscribed circle; input the radius and center into the Circle function, and draw the maximum inscribed circle of the edge contour of the target stalk region on the stalk region segmentation image; calculate the scale according to the standard reference object next to the soybean plant image, and convert the maximum inscribed circle diameter into the actual stem thickness; at this point, the prediction of the main stem thickness is completed; The method for obtaining the branch stem diameter is similar to that of the main stem; the difference is that the target position for measuring the branch stem diameter is between the first and second nodes of the branch; for example, for the i-th branch, the center point between node1[x1,y1,w1,h1,i+1] and node2[x2,y2,w2,h2,i+1] on the branch is taken as the target position for measuring the branch stem diameter.

9. The method for automatically acquiring soybean plant stem-related phenotypes based on deep learning according to claim 1, characterized in that: The step 8 described above is to obtain the phenotype; Inputting the collected images into the soybean plant phenotype extraction deep learning network model, and performing data processing on the output of the soybean plant phenotype extraction deep learning network model; Combining multi-stem path planning algorithms with traditional image processing algorithms, the stem-related phenotypes of soybean plants are obtained; It includes: 8.1) Deploy the computing environment that the soybean plant phenotype extraction deep learning network model relies on on computing devices; 8.2) Using the processed data obtained in step 1 as input data for a deep learning network model of soybean plant stem-related phenotypes; 8.3) Transplanting the complete soybean plant stem-related phenotype deep learning network model obtained in step 2 into a computing device, and transferring the collected images to the computing device through a hardware interface; 8.4) The result of the deep learning network model of soybean plant stem-related phenotypes after filtering the stem nodes obtained in step 3 is used as a priori condition for phenotype extraction; 8.5) The node detection result obtained in step 3 and the stem segmentation image obtained in step 2 are processed by steps 4, 5, 6 and 7 to obtain the statistical results of the stem phenotype number of the soybean plant main stem nodes and branch nodes, the internode distance, the main stem thickness and the branch stem thickness.

Images