Image-based method and system for phenotyping legume plants

Abstract

The application provides an image-based legume plant phenotype analysis method and system. A side image of a legume plant can be obtained, and the side image is detected by using a legume fruit detection model to obtain an accurate legume fruit boundary detection result. A node detection model is used to detect the side image to obtain an accurate main stem node detection result. The main stem node detection result is converted into a main stem node sequence conforming to a spatial structure, and a plant phenotype parameter representing the overall phenotype of the legume plant is determined according to the main stem node sequence. The obtained legume fruit boundary detection result is used as a prompt to perform legume fruit segmentation processing on the side image, which can improve the accuracy and generalization ability of the segmentation processing, and obtain a segmentation mask of each legume fruit. Thus, accurate legume fruit phenotype parameters can be determined based on the segmentation mask. The obtained plant phenotype parameters and legume fruit phenotype parameters are output, thereby improving the efficiency and consistency of the phenotype analysis of the legume plant.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a legume plant phenotype analysis method and system based on images. BACKGROUND

[0002] At present, the plant (especially legume) phenotype analysis technology is generally based on single organ identification analysis, and the overall analysis of the whole plant still needs to rely on manual measurement.

[0003] This manual measurement method has the problems of low efficiency, strong subjectivity and poor consistency, and is difficult to meet the demand of high-throughput phenotype analysis. SUMMARY

[0004] Therefore, the present application aims to provide a legume plant phenotype analysis method and system based on images to solve or partially solve the above technical problems.

[0005] To achieve the above purpose, the present application provides a legume plant phenotype analysis method based on images, comprising:

[0006] obtaining a side image of a legume plant;

[0007] processing the side image by using a pre-trained legume fruit detection model to obtain a legume fruit boundary detection result;

[0008] detecting nodes of the side image by using a pre-trained node detection model to obtain a main stem node detection result;

[0009] converting the main stem node detection result into a main stem node sequence corresponding to the spatial structure of the main stem of the legume plant;

[0010] determining a plant phenotype parameter of the legume plant according to the main stem node sequence;

[0011] processing the side image for legume fruit segmentation by using the legume fruit boundary detection result as a prompt to obtain a segmentation mask of each legume fruit;

[0012] determining a legume fruit phenotype parameter based on the segmentation mask of each legume fruit;

[0013] outputting the plant phenotype parameter and the legume fruit phenotype parameter as a phenotype analysis result of the legume plant.

[0014] Based on the same inventive concept, the present application also provides an electronic system comprising a memory, a processor and a computer program stored on the memory and executable by the processor, wherein the processor implements the method as described above when executing the computer program.

[0015] Based on the same inventive concept, the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method described above.

[0016] As can be seen from the above, the image-based legume plant phenotype analysis method and system provided by the present application can obtain a side image of a legume plant, so that a legume fruit detection model can be used to detect the side image to obtain an accurate legume fruit boundary detection result. In addition, a node detection model can also be used to detect the side image to obtain an accurate main stem node detection result. Then, the main stem node detection result can be converted into a main stem node sequence that is more consistent with the spatial structure, and then the main stem node sequence can be used to determine a plant phenotype parameter of the legume plant, which can represent the overall phenotype of the legume plant. Then, the obtained legume fruit boundary detection result can be used as a prompt to perform legume fruit segmentation processing on the side image, which can improve the accuracy and generalization ability of the segmentation processing, and then the segmentation mask of each legume fruit of the legume plant can be obtained. In this way, the accurate legume fruit phenotype parameter can be determined based on the segmentation mask of each legume fruit. Finally, the obtained plant phenotype parameter and legume fruit phenotype parameter are both output as the phenotype analysis result of the legume plant, thereby improving the efficiency and consistency of the phenotype analysis of the legume plant. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application or related art, the following will briefly introduce the drawings needed to be used in the embodiments or related art descriptions. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0018] Figure 1 A flowchart of the image-based legume plant phenotype analysis method of the embodiments of the present application;

[0019] Figure 2 A schematic diagram of the side image of the legume plant of the embodiments of the present application;

[0020] Figure 3 A schematic diagram of the main stem node detection result of the embodiments of the present application;

[0021] Figure 4 A distribution diagram of the main stem node of the embodiments of the present application;

[0022] Figure 5 A schematic diagram of the weighted connected graph of the embodiments of the present application;

[0023] Figure 6 A schematic diagram of a main stem node sequence of an embodiment of the present application;

[0024] Figure 7 A schematic diagram of a main stem constituted by main stem nodes of an embodiment of the present application;

[0025] Figure 8 A schematic diagram of a phenotypic analysis result of an embodiment of the present application;

[0026] Figure 9 A structural block diagram of an image-based legume plant phenotypic analysis device of an embodiment of the present application;

[0027] Figure 10 A structural schematic diagram of an electronic system of an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed explanations will be given below with reference to specific embodiments and the accompanying drawings.

[0029] It should be noted that, unless otherwise defined, technical terms or scientific terms used in the embodiments of the present application should be understood as their common meanings to those skilled in the art to which the present application belongs. The terms “first”, “second”, and similar terms used in the embodiments of the present application do not represent any order, number, or importance, but are only used to distinguish different components. The terms “include”, “contain”, and similar terms mean that the components or objects before the terms encompass the components or objects listed after the terms and their equivalents, without excluding other components or objects. The terms “connect” or “connected” and similar terms do not mean only physical or mechanical connections, but can also include electrical connections, whether direct or indirect. The terms “upper”, “lower”, “left”, “right”, and the like only represent relative positional relationships, which can change when the absolute positions of the described objects change.

[0030] In related technologies, deep learning techniques have been introduced into plant phenotypic analysis, but there are still obvious limitations:

[0031] First, most deep learning methods only focus on the detection of a single organ of a plant (such as a pod or a stem of a soybean plant), lack an understanding of the overall structure of the plant, and cannot automatically construct the topology of the main stem, resulting in the need for manual measurement of structural parameters such as plant height and internode distance.

[0032] Second, traditional segmentation models based on CNN (Convolutional Neural Network) (such as UNet and Mask R-CNN) require a large amount of precisely labeled data for training, and the generalization ability of traditional segmentation models is limited, which seriously affects the accuracy of pod counting and morphological parameter extraction.

[0033] The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0034] The image-based phenotypic analysis method for legumes proposed in this application refers to plants that can produce legume fruits, preferably soybean plants.

[0035] like Figure 1 As shown, the method includes:

[0036] Step 101: Obtain a side view image of the legume plant.

[0037] In practice, an RGB camera (i.e., a red-green-blue camera) is used to capture images of the side of legumes, thus obtaining an initial side image. This initial side image is then preprocessed to obtain an accurate side image (e.g., ...). Figure 2 (As shown).

[0038] Step 102: Using the pre-trained bean fruit detection model, process the side image to obtain the bean fruit boundary detection result.

[0039] In practice, the training process of the legume fruit detection model is as follows: a YOLOv11 architecture is pre-constructed, and the YOLOv11 architecture is trained using a large number of side image samples of legume plants with real legume fruit boundary detection results to obtain the legume fruit detection model.

[0040] The side view image is then input into the legume fruit detection model for legume fruit boundary detection processing, thereby obtaining accurate legume fruit boundary detection results (as shown). The legume fruit boundary detection results are a set of bounding boxes for each legume fruit (e.g., pod).

[0041] Step 103: Using the pre-trained node detection model, perform node detection on the side image to obtain the main stem node detection result.

[0042] In practice, the training process of the main stem node detection model is as follows: a YOLOv11 architecture is pre-constructed, and the YOLOv11 architecture is trained using a large number of side image samples of legumes with real main stem node detection results to obtain the main stem node detection model.

[0043] Then, the side image is input into the main stem node detection model for node detection processing, thereby obtaining accurate main stem node detection results (such as...). Figure 3 As shown in the figure). The main stem node detection result is the set of coordinates of the main stem node (e.g., ...). Figure 4 (As shown).

[0044] Step 104: The detection results of the main stem nodes are converted into a main stem node sequence corresponding to the spatial structure of the main stem of the legume.

[0045] In practice, based on the detection results of the main stem node, a weighted connected graph corresponding to it is constructed (e.g., ...). Figure 5 As shown in the figure, the corresponding main stem node sequence is generated based on the growth path of legumes.

[0046] Step 105: Determine the plant phenotypic parameters of the legume plant based on the main stem node sequence.

[0047] In practice, the overall phenotypic characteristics of legume plants can be determined based on the main stem node sequence, thereby obtaining accurate plant phenotypic parameters. These parameters include plant height and the average internode spacing.

[0048] Step 106: Using the results of the bean fruit boundary detection as a prompt, perform bean fruit segmentation processing on the side image to obtain the segmentation mask for each bean fruit.

[0049] In practice, since the above-mentioned results of the detection of the boundaries of the legume fruits are relatively accurate, these results can be used as a prompt to perform legume fruit segmentation on the side image. This process segments each legume fruit in the side image and obtains the segmentation mask for each legume fruit.

[0050] Specifically, the segmentation model is used to segment the bean fruit boundary detection results and side images obtained in the above steps to obtain an accurate segmentation mask for each bean fruit.

[0051] For example, the segmentation model is a pre-trained SAM (Segment Anything Model, a general artificial intelligence model). Based on the cue-based segmentation capability of the SAM model, the bean fruit boundary detection results (i.e., bounding boxes) obtained in the previous steps are used as cue signals and input into the SAM model along with the side view image. In this way, the bean fruit boundary detection results can guide the SAM model to automatically generate an accurate bean fruit segmentation mask corresponding to each bounding box.

[0052] Step 107: Determine the phenotypic parameters of the legume fruit based on the segmentation mask of each legume fruit.

[0053] In practice, after obtaining the segmentation mask for each legume fruit, phenotypic analysis of the legume fruit can be performed to determine its corresponding phenotypic parameters.

[0054] Step 108: Output the plant phenotypic parameters and the legume fruit phenotypic parameters as the phenotypic analysis results of the legume plants.

[0055] In practice, by integrating the plant phenotypic parameters and legume fruit phenotypic parameters obtained above, accurate phenotypic analysis results of legume plants can be obtained (e.g., Figure 8 As shown in the figure, this allows for output display, enabling users to obtain the phenotypic analysis results of legume plants.

[0056] The above scheme allows for the acquisition of side views of legume plants. A legume fruit detection model can then be used to detect these side views, yielding accurate fruit boundary detection results. Additionally, a node detection model is employed to detect nodes within the side views, resulting in accurate main stem node detection results. These main stem node detection results are then converted into a more spatially consistent main stem node sequence. This sequence is used to determine the plant phenotypic parameters of the legume, which characterize the overall phenotypic profile. The obtained fruit boundary detection results are then used as a cue to segment the side views into legume fruits, improving segmentation accuracy and generalization ability. This yields a segmentation mask for each legume fruit, allowing for the determination of accurate fruit phenotypic parameters. Finally, both the plant phenotypic parameters and the legume fruit phenotypic parameters are output as the phenotypic analysis results for the legume plant, thus improving the efficiency and consistency of the phenotypic analysis.

[0057] In some embodiments, the preprocessing procedure in step 101 includes:

[0058] Normalize each pixel in the side image. The original range of the pixel (0 to 255) is normalized to a floating-point range of [0,1], thus obtaining the normalized side image.

[0059] The normalized side image is scaled to a fixed size to meet the input requirements of subsequent models, and the final side image is used to replace the side image to perform subsequent steps 102 to 108.

[0060] In some embodiments, step 102 includes:

[0061] Step 1021: The side image is resized according to multiple scaling ratios to obtain a transformed image corresponding to each scaling ratio, and all transformed images are combined to obtain a transformed image set.

[0062] In practice, the side image I is scaled using R different scaling ratios. Perform size transformation on the image set. .

[0063] Step 1022: Input each transformed image in the transformed image set into the legume fruit detection model to obtain the undetermined legume fruit detection bounding box corresponding to each transformed image, and the confidence level corresponding to each undetermined legume fruit detection bounding box.

[0064] In practice, the image set will be transformed. In this process, each transformed image is input into the bean fruit detection model for inference, and the bounding box of the bean fruit to be detected corresponding to the transformed image and its corresponding confidence score are obtained and output.

[0065] Step 1023: Map each of the undetermined bean fruit detection bounding boxes back to the side image according to the corresponding scaling ratio to obtain candidate boxes, and combine all candidate boxes to obtain a candidate box set.

[0066] In practice, the bounding box of each undetermined bean fruit detection is scaled according to its corresponding scaling ratio r. q Determine the mapping ratio 1 / r q Mapping back to the coordinate system corresponding to the side view image yields the candidate bounding box. , where r q ∈ q is the sequence number.

[0067] The final set of candidate boxes .

[0068] Step 1024: Sort each candidate box in the candidate box set in descending order according to its corresponding confidence level to obtain sorted candidate boxes.

[0069] Step 1025: The candidate sorting boxes are filtered using generalized intersection-union ratio to obtain the final bean fruit boundary detection results.

[0070] In practice:

[0071] 1) Select the candidate box G with the highest confidence from the sorted candidate boxes and add it to the final bean fruit boundary detection result. Calculate the set of candidate box G and the remaining candidate boxes. Generalized intersection :

[0072] .

[0073] in, Candidate boxes bounding box region and For other candidate box sets The bounding box region of any remaining candidate box in the list. For inclusion and The minimum convex closure region, Let be the intersection-union ratio of A and B.

[0074] 2) Will satisfy Candidate boxes are removed, among which This is the preset suppression threshold.

[0075] 3) Repeat steps 1) to 2) above until the final sorting candidate boxes are empty, to obtain the final bean fruit boundary detection results:

[0076] , where m is the number of legume fruits in the final legume fruit boundary detection result.

[0077] = Where i is the sequence number of the legume fruit, The coordinates of the bounding box center are These represent the pixel width and height of the bounding box, respectively. Let be the confidence level of the i-th bean fruit.

[0078] The above method ensures higher accuracy in the final detection results of legume fruit boundaries.

[0079] In some embodiments, the node detection model in step 103 performs node detection on the side image to obtain the main stem node detection result, which is the set of coordinates of the main stem nodes: , of which One main stem node The pixel coordinates of the main stem node in the image plane are defined.

[0080] In some embodiments, step 104 includes:

[0081] Step 1041: Take each main stem node in the main stem node detection result as a vertex, and combine all vertices to obtain a vertex set.

[0082] Where the vertex set V = .

[0083] Step 1042: Based on every two vertices in the vertex set, construct an undirected edge between the two vertices, and combine all the undirected edges between two vertices to obtain a boundary set.

[0084] In practice, for any two vertices and An undirected edge is established between each of them. That is, the boundary set. .

[0085] Step 1043: Determine the weight corresponding to each undirected edge in the boundary set, and combine all weights to obtain the weight set.

[0086] In some embodiments, step 1043 includes:

[0087] Perform the following for each undirected edge:

[0088] Step 10431: Determine the Euclidean distance between the two vertices corresponding to the undirected edge, and determine the spatial proximity component based on the Euclidean distance.

[0089] In practice, the two vertices are calculated. = and = The Euclidean distance between them, and the normalized Euclidean distance formula is:

[0090] Where H and W are the pixel height and pixel width, respectively.

[0091] Step 10432: Based on the connection direction of the two vertices corresponding to the undirected edge, determine the first vertex and the second vertex respectively, and determine whether the second vertex is above the first vertex. If yes, determine that the direction constraint component is 0; otherwise, determine that the direction constraint component is a predetermined penalty weight.

[0092] In practical implementation, the first vertex = The corresponding second vertex = If the second vertex is above the first vertex (i.e. If the origin of the image coordinate system is at the top left corner, then the connection method of the undirected edge can be considered correct and no penalty needs to be applied. The corresponding directional constraint component is 0. Otherwise, a penalty needs to be applied. The corresponding directional constraint component is a predetermined penalty weight, which can prevent the connection direction from growing downward.

[0093] Directional constraint components for: Where F is a predetermined penalty constant (e.g., F=100). For indicator functions, when the condition The value is 1 if the condition is met, and 0 otherwise.

[0094] Step 10433: Determine the connection vector based on the connection direction of the two vertices corresponding to the undirected edge, determine the angle between the connection vector and the vertical upward vector, and determine the vertical directional component based on the angle.

[0095] In practical implementation, the formula for the vertical tendency component is: Where θ is the connection vector. The angle between the component and the vertical upward vector (0, -1), ranging from [0°, 90°]. λ is a preset penalty coefficient used to adjust the proportion of this component in the total weight. When λ = 0, the angle effect is ignored; the larger λ is, the heavier the penalty for non-vertical connections.

[0096] Step 10434: Add the spatial proximity component, the directional constraint component, and the vertical tendency component to obtain the weight corresponding to each undirected edge.

[0097] The formula is: .

[0098] Perform the following for all undirected edges:

[0099] Step 10435: Integrate the weights corresponding to all undirected edges to obtain the weight set W.

[0100] Where W={ }

[0101] Step 1044: Integrate the vertex set, the boundary set, and the weight set to construct a weighted connected graph G(V, E, W).

[0102] Step 1045: Perform main path filtering on the vertices in the weighted connected graph, and remove vertices that are not part of the main path to obtain the main stem node sequence.

[0103] In some embodiments, step 1045 includes:

[0104] Step 10451: Based on the weighted connected graph, determine the vertex with the lowest spatial position as the base node. .

[0105] Step 10452, the basic node As the main stem node, it is added to the initialization sequence of the main stem node. The other vertices in the weighted connected graph, excluding the main stem node, are combined as a candidate set.

[0106] Step 10453, the basic node As the current node .

[0107] Specifically, candidate nodes in the candidate set Must be located in the current node Above, that is, satisfying: Therefore, for nodes not located in the current node... Delete the candidate nodes above to obtain the adjusted candidate set. (The specific process is as follows: step 10454).

[0108] Step 10454: Delete the vertices located below the current node in the candidate set to obtain an adjusted candidate set. Select at least one candidate undirected edge connected to the current node from the adjusted candidate set. Select the undirected edge with the smallest weight from the at least one candidate undirected edge. Place the vertex at the other end of the undirected edge with the smallest weight relative to the current node as the next stem node into the initial stem node sequence to complete one stem node selection.

[0109] Specifically, candidate nodes in the candidate set Must be located in the current node Above, that is, satisfying: Therefore, for nodes not located in the current node... Delete the candidate nodes above to obtain the adjusted candidate set. .

[0110] Then, select the undirected edge with the smallest weight (i.e., the undirected edge with the lowest overall connection cost) among at least one candidate undirected edge connecting to the current node, and use the vertex at the other end of this undirected edge as the next stem node. ,Will Add it to the initialization sequence of the main stem node.

[0111] Step 10455: Remove the next main stem node from the adjusted candidate set to obtain a new candidate set. Then, use the next main stem node as the current node. Repeat step 10454, the main stem node selection process, in this new candidate set until the candidate set is empty. The corresponding process for selecting and constructing the main stem node sequence is as follows: Figure 6 As shown.

[0112] Step 10456: Use the final obtained initial stem node sequence as the stem node sequence. , where L is the number of main stem nodes. The main stems formed by the main stem nodes in this main stem node sequence are as follows: Figure 7 As shown, the matching between the main stem node and the side image is as follows.

[0113] The above method can accurately determine the main stem node sequence, which facilitates the subsequent determination of plant phenotypic parameters based on the main stem node sequence.

[0114] In some embodiments, step 105 includes:

[0115] Step 1051: The main stem node sequence is used to generate the main stem centerline curve by interpolation.

[0116] In practice, cubic spline interpolation is used to generate a smooth, parameterized main stem centerline curve. ,in Let the curve parameters be normalized and satisfy the following conditions: (Root of the curve) (Top of the curve). Wherein, the x-coordinate of the curve at the center line of the main stem... y-axis .

[0117] Step 1052: Based on the main stem centerline curve, determine the height between two adjacent interpolation points sequentially, and sum the heights between the sequentially determined adjacent interpolation points to obtain the plant height of the legume. .

[0118] In practical implementation, the main stem centerline curve is derived from the parameters. arrive The arc length. Its discretization calculation formula is:

[0119]

[0120] in, It is the first on the curve The coordinates of interpolation points with equal parameter intervals This represents the total number of interpolation points.

[0121] Step 1053: Determine the Euclidean distance between all adjacent main stem nodes in the main stem node sequence, and use this distance as the internode spacing to obtain the internode spacing sequence. ,in .

[0122] Step 1054: Determine the average internode spacing based on the internode spacing sequence. The formula is: .

[0123] Step 1055: The plant height and the average internode distance of the legume are used as the plant phenotypic parameters of the legume.

[0124] The above method can obtain accurate plant height and average internode distance of legumes, thereby obtaining accurate plant phenotypic parameters of legumes, which facilitates the overall analysis of the plant.

[0125] In some embodiments, step 106 includes:

[0126] Step 1061: Perform spatial location processing based on the results of the bean fruit boundary detection to generate a prompt embedding feature.

[0127] In practice, for each bounding box in the boundary detection results of legume fruits As a spatial location cue, it is encoded into a cue embedding feature using the encoder in the SAM model. . ,in, This represents the encoder of the SAM model.

[0128] Step 1062: Based on the cue embedding features, the side image is processed to segment the beans to obtain an initial segmentation mask for each bean.

[0129] In practice, the side image I and the cue are embedded in the features. The input is fed into the SAM model (i.e., the segmentation model), and the initial segmentation mask is obtained through forward inference. This process can be formally represented as:

[0130] ,in, This is the initial segmentation mask for the output.

[0131] Step 1062: Perform hole-filling processing on the initial segmentation mask of each bean fruit to obtain the mask after hole filling for each bean fruit.

[0132] Step 1062: Smooth the mask after filling the holes of each bean fruit to obtain the segmentation mask for each bean fruit. .

[0133] In practice, ,in This indicates a hole-filling operation. Morphological smoothing operations include opening and closing operations to remove noise and smooth boundaries.

[0134] Obtain the segmentation mask set , where m is the number of legume fruits.

[0135] The above method can obtain an accurate segmentation mask for each bean fruit, providing an accurate pixel-level segmentation mask for subsequent morphological analysis.

[0136] In some embodiments, step 107 includes:

[0137] Step A1: Determine the fruit outline of each bean fruit based on the segmentation mask of each bean fruit.

[0138] In practice, this is applied to the segmentation mask set. Determine the segmentation mask M for each legume fruit k. k The outline of the fruit Y k , where k represents the serial number of the legume fruit.

[0139] For each legume fruit k, the fruit profile Y k implement:

[0140] Step A2: Determine the total number of pixels in the fruit outline, and determine the fruit area based on the total number of pixels. The formula is: .

[0141] Step A3: Determine the minimum bounding rectangle based on the fruit's outline, and determine the length based on the minimum bounding rectangle. and width .

[0142] The formula is: .

[0143] Step A4: Determine the ratio of the length to the width.

[0144] The formula is: .

[0145] Step A5: Determine multiple sets of three consecutive points in the outline of the fruit, determine the discrete curvature based on the three consecutive points, and calculate the average value of all the discrete curvatures to obtain the overall curvature of the fruit.

[0146] In practice, the corresponding three consecutive points are At the midpoint Discrete curvature at Defined as a vector and The supplementary angle of the included angle between them is:

[0147] .

[0148] Overall curvature The formula is: .

[0149] Step A6: Determine the area of ​​the smallest circumcircle of the fruit outline, and divide the fruit area by the area of ​​the smallest circumcircle to obtain the filling degree.

[0150] In practice, the radius of the smallest circumcircle of the fruit's outline is determined. Then fill degree for: .

[0151] Step A7: The fruit area, length and width, ratio, overall curvature, and filling degree are used as the phenotypic parameters of the legume fruit.

[0152] The above method can accurately determine the phenotypic parameters of legume fruits with various geometric shapes.

[0153] In some embodiments, step 107 includes:

[0154] Step B1: Use the segmentation mask of each bean fruit to perform foreground segmentation on the side image to obtain the foreground region of the bean fruit.

[0155] The corresponding foreground area is .

[0156] Step B2, determine the average intensity of red pixels within the foreground region. Average intensity of green pixels and the average intensity of blue pixels .

[0157] The corresponding formula is:

[0158] ;in, The red pixel image represents the foreground area.

[0159] ;in, The green pixel image represents the foreground area.

[0160] ;in, The image shows the blue pixels in the foreground area.

[0161] Step B3: Determine the mean hue, mean saturation, and mean brightness within the foreground area.

[0162] In practice, the RGB color space of the foreground area image is converted to the HSV color space to obtain the corresponding hue (H), saturation (S), and lightness (V) values, as shown in the formula:

[0163] ;in, The color tone of the foreground area;

[0164] ;in, The saturation of the foreground region;

[0165] ;in, This represents the brightness value of the foreground area.

[0166] Step B4: The average intensity of red pixels, green pixels, and blue pixels, as well as the average hue, saturation, and brightness, are used as phenotypic parameters of legume fruits.

[0167] The above method can accurately determine the phenotypic parameters of legume fruits based on various color characteristics.

[0168] Furthermore, the number of legume fruits (e.g., the number of pods) can be determined based on the number of mask m of legume fruits obtained above. .

[0169] Based on the main stem node sequence , where L is the number of main stem nodes, which allows us to obtain the accurate number of main stem nodes L.

[0170] The accurate plant height can be obtained by following the above process. .

[0171] The above process allows us to obtain the accurate average internode spacing of the main stem nodes. .

[0172] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0173] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0174] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides an image-based legume phenotypic analysis device.

[0175] refer to Figure 9 The device includes:

[0176] Image acquisition module 201 is configured to acquire a side view image of a legume plant;

[0177] The boundary detection module 202 is configured to process the side image using a pre-trained bean fruit detection model to obtain bean fruit boundary detection results.

[0178] Node detection module 203 is configured to use a pre-trained node detection model to perform node detection on the side image and obtain the main stem node detection result;

[0179] The sequence conversion module 204 is configured to convert the main stem node detection result into a main stem node sequence corresponding to the spatial structure of the main stem of the legume.

[0180] The plant phenotypic parameter determination module 205 is configured to determine the plant phenotypic parameters of legumes based on the main stem node sequence.

[0181] The segmentation processing module 206 is configured to use the boundary detection results of the bean fruits as a prompt to perform bean fruit segmentation processing on the side image to obtain a segmentation mask for each bean fruit.

[0182] The legume fruit phenotypic parameter determination module 207 is configured to determine the legume fruit phenotypic parameters based on the segmentation mask of each legume fruit.

[0183] The phenotypic parameter output module 208 is configured to output the plant phenotypic parameters and the legume fruit phenotypic parameters as phenotypic analysis results of the legume plant.

[0184] In some embodiments, the boundary detection module 202 is specifically configured as follows:

[0185] The side image is resized according to multiple scaling ratios to obtain a transformed image corresponding to each scaling ratio. All transformed images are combined to obtain a set of transformed images.

[0186] Each transformed image in the transformed image set is input into the legume fruit detection model to obtain the undetermined legume fruit detection bounding box corresponding to each transformed image, and the confidence level corresponding to each undetermined legume fruit detection bounding box.

[0187] Each of the undetermined bean fruit detection bounding boxes is mapped back to the side image according to the corresponding scaling ratio to obtain candidate boxes, and all candidate boxes are combined to obtain a candidate box set.

[0188] Each candidate box in the candidate box set is sorted in descending order according to its corresponding confidence level to obtain the sorted candidate boxes;

[0189] The candidate sorting boxes are filtered using generalized intersection-union ratio (GUCR) ​​to obtain the final results of bean fruit boundary detection.

[0190] In some embodiments, the sequence conversion module 204 is specifically configured as follows:

[0191] Each main stem node in the main stem node detection result is taken as a vertex, and all vertices are combined to obtain a vertex set;

[0192] Based on each pair of vertices in the vertex set, construct an undirected edge between the two vertices, and combine all the undirected edges between two vertices to obtain the boundary set.

[0193] Determine the weight corresponding to each undirected edge in the boundary set, and combine all weights to obtain the weight set;

[0194] The vertex set, the boundary set, and the weight set are integrated to construct a weighted connected graph;

[0195] The vertices in the weighted connected graph are filtered for main paths, and vertices that are not on the main path are removed to obtain the main stem node sequence.

[0196] In some embodiments, the sequence conversion module 204 is further configured to:

[0197] Perform the following for each undirected edge:

[0198] Determine the Euclidean distance between the two vertices corresponding to the undirected edge, and determine the spatial proximity component based on the Euclidean distance;

[0199] The first vertex and the second vertex are determined according to the connection direction of the two vertices corresponding to the undirected edge. It is then determined whether the second vertex is above the first vertex. If it is, the direction constraint component is determined to be 0; otherwise, the direction constraint component is determined to be a predetermined penalty weight.

[0200] Based on the connection direction of the two vertices corresponding to the undirected edge, determine the connection vector, determine the angle between the connection vector and the vertical upward vector, and determine the vertical directional component based on the angle.

[0201] The spatial proximity component, the directional constraint component, and the vertical directional component are added together to obtain the weight corresponding to each undirected edge.

[0202] Perform the following for all undirected edges:

[0203] The weights corresponding to all undirected edges are integrated to obtain a weight set.

[0204] In some embodiments, the sequence conversion module 204 is further configured to:

[0205] Based on the weighted connected graph, the vertex with the lowest spatial position is determined as the basic node;

[0206] The basic node is placed as the stem node into the initial stem node sequence, and the other vertex combinations in the weighted connected graph, excluding the stem node, are used as the candidate set.

[0207] Set the base node as the current node;

[0208] Delete the vertices below the current node in the candidate set to obtain an adjusted candidate set. Select at least one undirected edge connected to the current node from the adjusted candidate set. Select the undirected edge with the smallest weight from the at least one undirected edge. Place the vertex at the other end of the undirected edge with the smallest weight relative to the current node as the next stem node into the initial stem node sequence to complete one stem node selection.

[0209] The next main stem node is removed from the adjusted candidate set to obtain a new candidate set. The next main stem node is then used as the current node. This new candidate set replaces the candidate set, and the main stem node screening process is repeated until the candidate set is empty.

[0210] The final initialized stem node sequence is used as the stem node sequence.

[0211] In some embodiments, the plant phenotypic parameter determination module 205 is specifically configured as follows:

[0212] The main stem node sequence is used to generate the main stem centerline curve using interpolation.

[0213] Based on the main stem centerline curve, the height between two adjacent interpolation points is determined sequentially. The heights between the two adjacent interpolation points are then accumulated to obtain the plant height of the legume.

[0214] Determine the Euclidean distance between all adjacent main stem nodes in the main stem node sequence as the internode spacing to obtain the internode spacing sequence;

[0215] The average internode spacing is determined based on the internode spacing sequence;

[0216] The plant height and the average internode distance of the legume are used as the plant phenotypic parameters of the legume.

[0217] In some embodiments, the segmentation processing module 206 is specifically configured as follows:

[0218] Based on the results of the legume fruit boundary detection, spatial location processing is performed to generate hint embedding features;

[0219] The side image is segmented into legumes based on the cue embedding features to obtain an initial segmentation mask for each legume.

[0220] The initial segmentation mask of each bean fruit is filled with holes to obtain the mask after filling the holes of each bean fruit.

[0221] The mask after filling the holes of each bean fruit is smoothed to obtain the segmentation mask of each bean fruit.

[0222] In some embodiments, the legume fruit phenotypic parameter determination module 207 is specifically configured as follows:

[0223] The fruit outline of each bean fruit is determined based on the segmentation mask of each bean fruit;

[0224] Perform the following for each legume fruit outline:

[0225] Determine the total number of pixels in the fruit outline, and determine the fruit area based on the total number of pixels;

[0226] The minimum bounding rectangle is determined based on the fruit's outline, and the length and width are determined based on the minimum bounding rectangle.

[0227] Determine the ratio of the length to the width;

[0228] Determine multiple sets of three consecutive points in the outline of the fruit, determine discrete curvature based on the three consecutive points, and calculate the average value of all the discrete curvatures to obtain the overall curvature of the fruit.

[0229] Determine the area of ​​the smallest circumcircle of the fruit outline, and divide the fruit area by the area of ​​the smallest circumcircle to obtain the fill degree;

[0230] The fruit area, length and width, ratio, overall curvature, and filling degree are used as the phenotypic parameters of the legume fruit.

[0231] In some embodiments, the legume fruit phenotypic parameter determination module 207 is further configured to:

[0232] Using the segmentation mask of each bean fruit, the side image is segmented to obtain the foreground region of the bean fruit;

[0233] Determine the average intensity of red pixels, average intensity of green pixels, and average intensity of blue pixels within the foreground region;

[0234] Determine the mean hue, mean saturation, and mean brightness within the foreground region;

[0235] The mean intensity of red, green, and blue pixels, as well as the mean hue, saturation, and brightness, were used as phenotypic parameters of legume fruits.

[0236] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0237] The apparatus of the above embodiments is used to implement the corresponding method in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0238] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides an electronic system, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any of the above embodiments.

[0239] Figure 10 This embodiment illustrates a more specific electronic system hardware structure. The device may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0240] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0241] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0242] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0243] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0244] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0245] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0246] The electronic system described above is used to implement the corresponding method in any of the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0247] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to perform the methods described in any of the above embodiments.

[0248] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0249] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0250] Based on the same concept, corresponding to any of the above embodiments, this application also provides a computer program product, including computer program instructions, which, when run on a computer, cause the computer to perform the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0251] It is understood that before using the technical solutions of the various embodiments in this disclosure, users will be informed of the type, scope of use, and usage scenarios of the personal information involved in an appropriate manner, and user authorization will be obtained.

[0252] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose, based on the prompt message, whether to provide personal information to the software or hardware such as the electronic system, application program, server, or storage medium performing the operations of this disclosed technical solution.

[0253] As an optional but not limited implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic system.

[0254] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0255] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application is limited to these examples; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0256] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0257] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0258] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the claims of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A method for phenotypic analysis of legumes based on images, characterized in that, include: Obtain a side view image of a legume plant; The side image is processed using a pre-trained bean fruit detection model to obtain bean fruit boundary detection results. Using a pre-trained node detection model, node detection is performed on the side image to obtain the main stem node detection result; The detection results of the main stem nodes are converted into a main stem node sequence corresponding to the spatial structure of the main stem of legumes; Based on the main stem node sequence, determine the plant phenotypic parameters of legumes; Using the results of the bean fruit boundary detection as a prompt, the side image is segmented into bean fruits to obtain a segmentation mask for each bean fruit. Based on the segmentation mask of each legume fruit, determine the phenotypic parameters of the legume fruit; The plant phenotypic parameters and the legume fruit phenotypic parameters are output as the phenotypic analysis results of the legume plants.

2. The method according to claim 1, characterized in that, The step of processing the side image using a pre-trained legume fruit detection model to obtain legume fruit boundary detection results includes: The side image is resized according to multiple scaling ratios to obtain a transformed image corresponding to each scaling ratio. All transformed images are combined to obtain a set of transformed images. Each transformed image in the transformed image set is input into the legume fruit detection model to obtain the undetermined legume fruit detection bounding box corresponding to each transformed image, and the confidence level corresponding to each undetermined legume fruit detection bounding box. Each of the undetermined bean fruit detection bounding boxes is mapped back to the side image according to the corresponding scaling ratio to obtain candidate boxes, and all candidate boxes are combined to obtain a candidate box set. Each candidate box in the candidate box set is sorted in descending order according to its corresponding confidence level to obtain the sorted candidate boxes; The candidate sorting boxes are filtered using generalized intersection-union ratio (GUCR) ​​to obtain the final results of bean fruit boundary detection.

3. The method according to claim 1, characterized in that, The detection results of the main stem nodes are converted into a main stem node sequence corresponding to the spatial structure of the main stem of the legume, including: Each main stem node in the main stem node detection result is taken as a vertex, and all vertices are combined to obtain a vertex set; Based on each pair of vertices in the vertex set, construct an undirected edge between the two vertices, and combine all the undirected edges between two vertices to obtain the boundary set. Determine the weight corresponding to each undirected edge in the boundary set, and combine all weights to obtain the weight set; The vertex set, the boundary set, and the weight set are integrated to construct a weighted connected graph; The vertices in the weighted connected graph are filtered for main paths, and vertices that are not on the main path are removed to obtain the main stem node sequence.

4. The method according to claim 3, characterized in that, The step of determining the weight corresponding to each undirected edge in the boundary set, and combining all weights to obtain a weight set, includes: Perform the following for each undirected edge: Determine the Euclidean distance between the two vertices corresponding to the undirected edge, and determine the spatial proximity component based on the Euclidean distance; The first vertex and the second vertex are determined according to the connection direction of the two vertices corresponding to the undirected edge. It is then determined whether the second vertex is above the first vertex. If it is, the direction constraint component is determined to be 0; otherwise, the direction constraint component is determined to be a predetermined penalty weight. Based on the connection direction of the two vertices corresponding to the undirected edge, determine the connection vector, determine the angle between the connection vector and the vertical upward vector, and determine the vertical directional component based on the angle. The spatial proximity component, the directional constraint component, and the vertical directional component are added together to obtain the weight corresponding to each undirected edge. Perform the following for all undirected edges: The weights corresponding to all undirected edges are integrated to obtain a weight set.

5. The method according to claim 3, characterized in that, The step of filtering the vertices in the weighted connected graph by removing vertices that are not part of the main path to obtain the stem node sequence includes: Based on the weighted connected graph, the vertex with the lowest spatial position is determined as the basic node; The basic node is placed as the stem node into the initial stem node sequence, and the other vertex combinations in the weighted connected graph, excluding the stem node, are used as the candidate set. Set the base node as the current node; Delete the vertices below the current node in the candidate set to obtain an adjusted candidate set. Select at least one undirected edge connected to the current node from the adjusted candidate set. Select the undirected edge with the smallest weight from the at least one undirected edge. Place the vertex at the other end of the undirected edge with the smallest weight relative to the current node as the next stem node into the initial stem node sequence to complete one stem node selection. The next main stem node is removed from the adjusted candidate set to obtain a new candidate set. The next main stem node is then used as the current node. This new candidate set replaces the candidate set, and the main stem node screening process is repeated until the candidate set is empty. The final initialized stem node sequence is used as the stem node sequence.

6. The method according to claim 1, characterized in that, The step of determining the plant phenotypic parameters of legumes based on the main stem node sequence includes: The main stem node sequence is used to generate the main stem centerline curve using interpolation. Based on the main stem centerline curve, the height between two adjacent interpolation points is determined sequentially. The heights between the two adjacent interpolation points are then accumulated to obtain the plant height of the legume. Determine the Euclidean distance between all adjacent main stem nodes in the main stem node sequence as the internode spacing to obtain the internode spacing sequence; The average internode spacing is determined based on the internode spacing sequence; The plant height and the average internode distance of the legume are used as the plant phenotypic parameters of the legume.

7. The method according to claim 1, characterized in that, The step of using the boundary detection results of the legume fruits as a prompt to perform legume fruit segmentation processing on the side image to obtain a segmentation mask for each legume fruit includes: Based on the results of the legume fruit boundary detection, spatial location processing is performed to generate hint embedding features; The side image is segmented into legumes based on the cue embedding features to obtain an initial segmentation mask for each legume. The initial segmentation mask of each bean fruit is filled with holes to obtain the mask after filling the holes of each bean fruit. The mask after filling the holes of each bean fruit is smoothed to obtain the segmentation mask of each bean fruit.

8. The method according to claim 1, characterized in that, The determination of phenotypic parameters of legume fruits based on the segmentation mask for each fruit includes: The fruit outline of each bean fruit is determined based on the segmentation mask of each bean fruit; Perform the following for each bean fruit's outline: Determine the total number of pixels in the fruit outline, and determine the fruit area based on the total number of pixels; The minimum bounding rectangle is determined based on the fruit's outline, and the length and width are determined based on the minimum bounding rectangle. Determine the ratio of the length to the width; Determine multiple sets of three consecutive points in the outline of the fruit, determine discrete curvature based on the three consecutive points, and calculate the average value of all the discrete curvatures to obtain the overall curvature of the fruit. Determine the area of ​​the smallest circumcircle of the fruit outline, and divide the fruit area by the area of ​​the smallest circumcircle to obtain the fill degree; The fruit area, length and width, ratio, overall curvature, and filling degree are used as the phenotypic parameters of the legume fruit.

9. The method according to claim 1, characterized in that, The determination of phenotypic parameters of legume fruits based on the segmentation mask for each fruit includes: Using the segmentation mask of each bean fruit, the side image is segmented to obtain the foreground region of the bean fruit; Determine the average intensity of red pixels, average intensity of green pixels, and average intensity of blue pixels within the foreground region; Determine the mean hue, mean saturation, and mean brightness within the foreground region; The mean intensity of red, green, and blue pixels, as well as the mean hue, saturation, and brightness, were used as phenotypic parameters of legume fruits.

10. An electronic system comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 9.

Images