Deep learning-based crop root phenotype detection and index calculation method and device

Abstract

The invention discloses a crop root phenotype detection and index calculation method and device based on deep learning, and the method comprises the steps: inputting an image, carrying out the edge enhancement of a root image through side window filtering, and carrying out the data expansion of the enhanced root image, wherein the root mainly refers to a main root and a side root of a crop; detecting the root system individuals by using the RT-DETR target detection model to obtain a detection frame of the root system individuals; detecting a base part and a stem part in the root system individual image by using an RT-DETR target detection model, and removing a part of base part and stem part areas according to the position of a base part and stem part detection frame in the image; intercepting a root system individual in the image by using a detection frame; segmenting a main root in the image by using an instance segmentation model Mask2form to obtain a contour coordinate point set of the main root; interactively selecting windows with relatively medium lateral root density on the main root; segmenting a lateral root in the window by using an improved Transfiner model to obtain a contour point set of the lateral root in the window; and performing index calculation on the root phenotype in the window by using the contour point set. The method can greatly improve the accuracy of root system segmentation.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision algorithm deployment, and specifically relates to a method and device for crop root phenotype detection and index calculation based on deep learning. Background Art

[0002] As a vital organ for plants to obtain water and nutrients, the root system directly affects their growth and development. In modern agricultural and crop science research, studying the complexity of crop root structure, especially the morphological characteristics of taproots and lateral roots, is of great significance for understanding crop adaptability to the environment, improving crop yields, and optimizing breeding strategies. Instance segmentation technology, applied to root system research, can automatically and accurately segment the contours of individual taproots and lateral roots from images and perform feature analysis. However, instance segmentation technology is currently relatively unavailable in the field of root segmentation. Roots are slender in shape, with large individual differences and many occlusions between individuals. These problems are particularly prominent in lateral roots. Therefore, general instance segmentation methods perform poorly for lateral root segmentation, making it difficult to directly apply them to the calculation of root phenotypic indicators.

[0003] Therefore, this paper utilizes an instance segmentation algorithm combined with agronomic knowledge to develop a deep learning-based method for detecting and calculating crop root phenotyping indicators. Since rice is a crucial crop in our daily lives, this method will be specifically described using rice roots as an example. Summary of the Invention

[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a method and device for crop root phenotype detection and index calculation based on deep learning. By adding snake convolution and KAN modules, the backbone network part of the Transfiner algorithm is improved, the feature extraction ability of the model is enhanced, and the accuracy of the segmentation algorithm in root segmentation is improved.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a method for detecting and calculating crop root phenotypes and indicators based on deep learning, comprising the following steps:

[0007] Input image, use side window filtering to perform edge enhancement on the root system image, and perform data expansion on the enhanced root system image;

[0008] The target detection model RT-DETR is used to detect individual roots in the root system image after data expansion, and the detection frame of the individual roots is obtained. The detection frame is then used to intercept the individual roots in the image.

[0009] The target detection model RT-DETR is used to detect the root base and stem in the root individual image, and the detection frames of the root base and stem are obtained. The base and stem regions are then partially removed according to the positions of the base and stem detection frames in the image, thus extracting the effective region without losing the root area.

[0010] The Mask2former instance segmentation model is used to segment the main root in the image and obtain the segmented contour to obtain the main root contour. A window with a relatively medium density of lateral roots above the main root is interactively selected and the main root contour is intercepted using the window to obtain the main root contour point set within the window.

[0011] The improved Transfiner model is used to segment the lateral roots in the window, obtain the segmented contours, and obtain the contour point set of the lateral roots in the window; the improved Transfiner model uses snake convolution in the backbone network to replace the convolution operation in the first stage of the original backbone network, and uses the KAN module to replace the first convolution operation in the first stage of the original backbone network;

[0012] Based on the contour point set of the main root and the contour point set of the lateral root in the window, the index of the root phenotype in the window is calculated.

[0013] As a preferred technical solution, the edge enhancement of the root system image is performed using side window filtering, specifically:

[0014] During the processing of each pixel, the side window filter evaluates multiple directions in the neighborhood of the current pixel and selects the direction that best suits the pixel structure for filtering;

[0015] The data expansion of the enhanced root system image is specifically performed as follows:

[0016] Randomly select data augmentation methods from Gaussian blur, affine transformation, snow enhancement, horizontal flip, and vertical flip to increase data and improve the generalization of the model.

[0017] As a preferred technical solution, the target detection model RT-DETR is used to detect individual roots in the root image after data expansion, obtain a detection frame of the individual roots, and use the detection frame to intercept the individual roots in the image, specifically:

[0018] The root image is input into the target detection model RT-DETR, which learns the appearance of the root system. The trained weights are then used to detect individual roots in the image, and the bounding box set β of the individual roots is returned. Specifically,

[0019] β={b1,b2,...,b N}

[0020] Where N is the number of detected root individuals, and each bounding box b i For a quadruple b i =(x i1 ,y i1 ,x i2 ,y i2 ), which represent the coordinates of the upper left corner and lower right corner of the bounding box respectively;

[0021] Get the bounding box b of each root individual i =(x i1 ,y i1 ,x i2 ,y i2 ), and then use these bounding boxes to crop the image I and extract the image subregion of each root individual. The specific formula is as follows:

[0022] I i =I[y i1 :y i2 ,x i1 :x i2 ]

[0023] Among them, I i Indicates that according to the bounding box b i The image area corresponding to the i-th root individual is cut out.

[0024] As a preferred technical solution, the target detection model RT-DETR is used to detect the root base and stem in the root individual image, obtain the detection frame of the root base and stem, and remove part of the base and stem area according to the position of the base and stem detection frame in the image, extracting the effective area without losing the root area, specifically:

[0025] The root individual image I is input into the target detection model RT-DETR, the appearance of the root base is learned, and then the trained weights are used to detect the base and stem of the root individual in the image, and the bounding box γ of the base and stem of the root individual is returned. b and γ s , which respectively contain (x b1 ,y b1 ,x b2 ,y b2 ) and (x s1 ,y s1 ,x s2 ,y s2 ), which represent the x- and y-coordinates of the upper left corner of the root individual base bounding box, the x- and y-coordinates of the lower right corner of the root individual base bounding box, the x- and y-coordinates of the upper left corner of the root individual stem bounding box, and the x- and y-coordinates of the lower right corner of the root individual stem bounding box, respectively;

[0026] Get the bounding box of the root individual base and stem (x b1 ,y b1 ,x b2 ,y b2 )、(x s1 ,y s1 ,x s2 ,y s2 ), the bounding box is used to crop the image I, specifically:

[0027] First, the image orientation is determined based on the height and width of the root system image. If the image height is greater than the image width, the image is vertical, otherwise the image is horizontal.

[0028] Then, the bounding box of the base and stem of the root individual γ b and γ s Calculate the center point. The calculation formula of the center point coordinates is as follows:

[0029]

[0030] Among them, (x c ,y c ) represents the center point position;

[0031] If the image is vertical, first determine the center point (x c ,y c ) in the image:

[0032] If (x c ,y c ) is located in the upper half of the image, i.e. Then from the center point (x c ,y c ) starts, keep the upper half of the image; if (x c ,y c ) is located in the lower half of the image, i.e. Then from the center point (x c ,y c ) starts, retaining the lower half of the image;

[0033] If the image is horizontal, first determine the center point (x c ,y c ) in the image:

[0034] If (x c ,y c ) is located in the left half of the image, i.e. Then from the center point (x c ,y c ) starts, retaining the left half of the image;

[0035] If (xc ,y c ) is located in the right half of the image, i.e. Then from the center point (x c ,y c ) starts, retaining the right half of the image;

[0036] Through the above steps, part of the base and stem area of the root system is removed, thereby extracting the effective root area.

[0037] As a preferred technical solution, the instance segmentation model Mask2former is used to segment the main root in the image, and the segmented contour is obtained to obtain the contour of the main root, specifically:

[0038] The root system image I is input into the Mask2Former model, which segments the main root instances in the root system image I and outputs the segmentation mask M;

[0039] Use opencv to extract the contour point coordinates of the segmentation mask M and obtain the main root contour R i The contour point coordinate set RP i .

[0040] As a preferred technical solution, a representative window is selected from the root system individuals, and the outline of the main root is intercepted using the window to obtain a set of main root outline points within the window, specifically:

[0041] Use the interactive graphical interface built with the pyqt library to click and drag to select several representative rectangular windows (x1, y1, x2, y2) on the root system individual image, where x1 and y1 represent the coordinates of the upper left corner of the window, and x2 and y2 represent the coordinates of the lower right corner of the window;

[0042] After getting the window selected by the user, traverse the existing main root contour set C = {C1, C2, ..., C m}, check whether some points of each contour are located within the window, where m represents the total number of contours;

[0043] If the contour C is determined i If part of the image is within the window, it will be cropped. The specific process is as follows:

[0044] First, draw the contour with a fill value of 255 on the image A with all zeros, which is the same size as the root individual image. Then create a cropping region R = [x1, y1, x2, y2]. Then use the cropping region to crop A. Finally, use the OpenCV library to extract the coordinates of the contour points in the area with a value of 255 in the cropping region to obtain the contour C in the window. i The contour point coordinate set P i .

[0045] As a preferred technical solution, the improved instance segmentation model Transfiner is used to segment the lateral roots in the window, and the segmented contours are obtained to obtain the contour point set of the lateral roots in the window, specifically:

[0046] The window area I[y i1 :y i2 ,x i1 :x i2 ] Input the improved instance segmentation model Transfiner, the improved instance segmentation model Transfiner segments the lateral root instances in I and outputs the segmentation mask M, M i represents the binary mask of the i-th instance;

[0047] Use opencv to extract the contour point coordinates of the segmentation mask M to obtain the lateral root contour S i Contour point coordinate set SP i ;

[0048] The improved instance segmentation model Transfiner includes a backbone network, a neck network, a head network, and a mask correction part. The backbone network is used to extract image feature information. The neck network is used to fuse the features extracted by the backbone network, making the features learned by the neck network more diverse. The features are then handed over to the subsequent head network for segmentation, thereby improving network performance. Finally, the mask correction part refines the segmented mask to obtain the final segmentation result. The specific improvements are as follows:

[0049] In the backbone network, snake convolution is used to replace the convolution operation of the Stem in the original backbone network, and KAN is used to replace the first convolution operation in the first stage of the original backbone network to form the KBlock module.

[0050] The snake convolution, due to the continuity constraint design of its convolution kernel, is more free to fit the structural learning features during learning, especially the slender tubular structure. Using snake convolution to replace the first stage convolution layer in the backbone network enables the shallow network to better learn the shape and contour features in the image.

[0051] Compared with the MLP network, the KAN module removes the linear layer, parameterizes the activation function into a learnable one-dimensional spline function, and places it on the edge instead of the node; the KAN module replaces the first convolution operation in the first stage of the original backbone network, and projects the features into the latent space before feature extraction to better extract features.

[0052] As a preferred technical solution, the root phenotype index calculation within the window is performed based on the contour point set of the main root and the contour point set of the lateral root within the window, specifically:

[0053] Based on the contour point set of the main root and the contour point set of the lateral root in the window, the contour perimeter and area of the main root and lateral root are measured using OpenCV. Then, the respective perimeter and area are divided by the number of main roots and lateral roots respectively to obtain the average perimeter and average area indicators of the main root and lateral roots in the window.

[0054] In a second aspect, the present invention provides a crop root phenotype detection and index calculation system based on deep learning, which is applied to the crop root phenotype detection and index calculation method based on deep learning, including an image processing module, a target detection module, an effective area extraction module, a main root segmentation module, a lateral root segmentation module and a root phenotype calculation module;

[0055] The image processing module is used to perform edge enhancement on the root system image using side window filtering, and perform data expansion on the enhanced root system image;

[0056] The target detection module is used to detect individual roots in the root system image after data expansion using the target detection model RT-DETR, obtain a detection frame of the individual roots, and use the detection frame to intercept the individual roots in the image;

[0057] The effective area extraction module uses the target detection model RT-DETR to detect the root base and stem in the root individual image, obtains the detection frame of the root base and stem, and removes part of the base and stem area according to the position of the base and stem detection frame in the image, extracting the effective area without losing the root area;

[0058] The main root segmentation module is used to segment the main root in the image using the instance segmentation model Mask2former and obtain the segmented contour to obtain the main root contour; interactively select a window with a relatively medium density of lateral roots above the main root, and use the window to intercept the main root contour to obtain a set of main root contour points within the window;

[0059] The lateral root segmentation module is used to segment the lateral roots in the window using an improved Transfiner model to obtain the segmented contours and obtain a set of contour points of the lateral roots in the window; the improved Transfiner model uses snake convolution in the backbone network to replace the convolution operation in the first stage of the original backbone network, and uses the KAN module to replace the first convolution operation in the first stage of the original backbone network;

[0060] The root phenotype calculation module is used to calculate the index of the root phenotype in the window based on the outline point set of the main root and the outline point set of the lateral root in the window.

[0061] In a fourth aspect, the present invention provides an electronic device, comprising:

[0062] at least one processor; and,

[0063] a memory communicatively connected to the at least one processor; wherein,

[0064] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the crop root phenotype detection and indicator calculation method based on deep learning.

[0065] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0066] 1. The present invention detects individual roots before segmentation to obtain detection frames of the individual roots, and uses the detection frames to intercept the individual roots in the image; then detects the root base and stem in the image of the individual roots to obtain detection frames of the root base and stem, and removes part of the base and stem areas according to the positions of the base and stem detection frames in the image, extracting effective areas without losing the root area, and then performing segmentation, which can expand the effective range of segmentation and thus improve the accuracy of segmentation.

[0067] 2. In the backbone network of the Transfiner instance segmentation model, the present invention uses snake convolution to replace the convolution operation in the first stage of the original backbone network, and uses KAN to replace the first convolution operation in the first stage of the original backbone network. This enables the model to better learn the appearance characteristics of slender roots and improves the segmentation effect.

[0068] 3. The present invention uses an edge enhancement module and side window filtering to enhance the edges of the root system image. The roots are slender and overlap a lot, so the edges are not obvious, which can improve the accuracy of subsequent mouse detection.

[0069] 4. The present invention adopts a data expansion module and randomly selects data expansion methods from Gaussian blur, affine transformation, snowflake enhancement, horizontal flip, and vertical flip, which can improve the generalization of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0071] Figure 1 This is a flow chart of the root phenotypic index calculation method based on deep learning of the present invention;

[0072] Figure 2 This is a diagram showing the results of individual root system detection performed in an embodiment of the present invention;

[0073] Figure 3 This is a result diagram of root base and stem detection performed by an embodiment of the present invention;

[0074] Figure 4 This is a diagram showing the result of taproot segmentation according to an embodiment of the present invention;

[0075] Figure 5 This is a result diagram of interactively selecting a window according to an embodiment of the present invention;

[0076] Figure 6 This is a result diagram of lateral root segmentation according to an embodiment of the present invention;

[0077] Figure 7 This is a diagram of the overall model of Transfiner according to an embodiment of the present invention;

[0078] Figure 8 This is a diagram of the improved Transfiner model according to an embodiment of the present invention;

[0079] Figure 9 is a structural diagram of an electronic device according to an embodiment of the present invention;

[0080] Figure 10 This is a block diagram of a root phenotypic index calculation system based on deep learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0081] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0082] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0083] See also Figure 1 In one embodiment of the present application, a method for calculating root phenotypic indicators based on deep learning is provided, comprising the following steps:

[0084] (1) Input image and use side window filtering to enhance the edge of the root image;

[0085] Furthermore, the edge enhancement of the image is performed using side window filtering, specifically:

[0086] During the processing of each pixel, the filter evaluates multiple directions in its neighborhood and selects the direction that best suits the pixel structure for filtering. The specific formula is as follows:

[0087]

[0088] Each side window is a local neighborhood that starts from pixel p and extends in a specific direction. L (p) is the left window of pixel p, W R (p) is the right window of pixel p, W U (p) is the upper window of pixel p, W D (p) is the lower window of pixel p, I(q) is the value of pixel q in window w, and I(p) is the value of pixel p.

[0089] (2) Perform data expansion on the image and randomly select data expansion methods from Gaussian blur, affine transformation, snowflake enhancement, horizontal flip, and vertical flip to increase data and improve the generalization of the model.

[0090] Furthermore, the image data is expanded, and the specific formula is as follows:

[0091] S=RandomSelect({a1,a2,a3,a4,a5})

[0092] Among them, S is a randomly selected data augmentation method, a1, a2, a3, a4, and a5 are Gaussian blur, affine transformation, snowflake enhancement, horizontal flip, and vertical flip, respectively.

[0093] Gaussian blur applies a convolution operation to blur the image. The specific formula is as follows:

[0094] I ′ =I*G(σ)

[0095] Where I represents the input image, I ′ represents the output image, G(σ) is a two-dimensional Gaussian kernel with a standard deviation of σ, and “*” represents the convolution operation.

[0096] Affine transformation is a linear transformation that includes operations such as translation, scaling, rotation, and shearing of images. The specific formula is as follows:

[0097] I ′ =A*I+b

[0098] Where I represents the input image, I ′ Represents the output image, A is an affine transformation matrix, representing rotation, scaling and shearing transformations; b is a translation vector, representing the displacement of the image.

[0099] Snow enhancement simulates visual interference by adding white noise to the image. The specific formula is as follows:

[0100] I ′ =I+N(x)

[0101] Among them, I represents the input image, I ′ Represents the output image, N(x) is a noise generating function that simulates the snowflake effect; x is a parameter that controls the noise intensity.

[0102] Horizontal flip and vertical flip are operations to flip the image left and right and up and down respectively. The specific formulas are as follows:

[0103] I ′ =Flip h (I)

[0104] I ′ =Flip v (I)

[0105] Where I represents the input image, I ′ Indicates the output image, Flip h 、Flip v Represents the flip operation in the horizontal direction and the flip operation in the vertical direction respectively.

[0106] (3) Please refer to Figure 2 , use the RT-DETR target detection model to detect the root individuals in the image, obtain the detection frame of the root individuals, and use the detection frame to intercept the root individuals in the image.

[0107] Furthermore, the RT-DETR target detection model is used to detect individual roots, obtain a detection frame of the individual roots, and use the detection frame to capture the individual roots in the image, specifically:

[0108] The root image is input into the target detection model RT-DETR, which learns the appearance of the root system. The trained weights are then used to detect individual roots in the image, and the bounding box set β of the individual roots is returned. Specifically,

[0109] β={b1,b2,...,b N}

[0110] Where N is the number of detected root individuals, and each bounding box b i For a quadruple b i =(x i1 ,y i1 ,x i2 ,y i2 ), which represent the coordinates of the upper left corner and lower right corner of the bounding box respectively.

[0111] Get the bounding box b of each root individual i =(x i1 ,y i1 ,x i2 ,y i2 ), we can use these bounding boxes to crop the image I and extract the image subregion of each root individual. The specific formula is as follows:

[0112] I i =I[y i1 :y i2 ,x i1 :x i2 ]

[0113] Among them, I i Indicates that according to the bounding box b i The image area corresponding to the i-th root individual is cut out.

[0114] (4) Please refer to Figure 3 , the RT-DETR target detection model is used to detect the root base and stem in the root individual image, and the detection frames of the root base and stem are obtained. Then, part of the base and stem area is removed according to the position of the base and stem detection frames in the image, and the effective area is extracted without losing the root area.

[0115] Furthermore, the RT-DETR target detection model is used to detect the root base and stem in the root individual image, and the detection frames of the root base and stem are obtained. Based on the position of the base and stem detection frames in the image, part of the base and stem area is removed to extract the effective area without losing the root area. Specifically:

[0116] The root individual image I is input into the target detection model RT-DETR, the appearance of the root base is learned, and then the trained weights are used to detect the base and stem of the root individual in the image, and the bounding box γ of the base and stem of the root individual is returned. b and γ s , which respectively contain (x b1 ,y b1 ,x b2 ,y b2 ) and (x s1 ,y s1 ,x s2 ,y s2 ), which represent the x- and y-coordinates of the upper left corner of the root individual base bounding box, the x- and y-coordinates of the lower right corner of the root individual base bounding box, the x- and y-coordinates of the upper left corner of the root individual stem bounding box, and the x- and y-coordinates of the lower right corner of the root individual stem bounding box, respectively.

[0117] Get the bounding box of the root individual base and stem (x b1 ,y b1 ,x b2 ,y b2 )、(x s1 ,y s1 ,x s2 ,y s2 ), we use the bounding box to crop the image I as follows:

[0118] First, determine the orientation of the image based on its height and width. If the image's height is greater than its width, the image is vertical; otherwise, the image is horizontal.

[0119] Then, the bounding box of the base and stem of the root individual γ b and γ s Calculate the center point. The calculation formula of the center point coordinates is as follows:

[0120]

[0121] Among them, (x c ,y c ) indicates the center point position.

[0122] If the image is vertical, first determine the center point (x c ,y c ) in the image:

[0123] If (x c ,y c ) is located in the upper half of the image (i.e. ), then from the center point (x c ,y c ) and retain the upper half of the image. The specific cropping formula is:

[0124] R=(0,0,W,y c )

[0125] Where R represents the image area to be retained.

[0126] If (x c ,y c ) is located in the lower half of the image (i.e. ), then from the center point (x c ,y c ) and retain the lower half of the image. The specific cropping formula is:

[0127] R=(0,y c ,W,H)

[0128] If the image is horizontal, first determine the center point (x c ,y c ) in the image:

[0129] If (x c ,y c ) is located in the left half of the image (i.e. ), then from the center point (x c ,y c ) and retain the left half of the image. The specific cropping formula is:

[0130] R=(0,0,x c ,H)

[0131] If (x c ,y c ) is located in the right half of the image (i.e. ), then from the center point (x c ,y c ) and retain the right half of the image. The specific cropping formula is:

[0132] R=(x c ,0,W,H)

[0133] Through the above steps, part of the base and stem area of the root system is removed, thereby extracting the effective root area.

[0134] (5) Please refer to Figure 4, use the instance segmentation model Mask2former to segment the main root in the image, and obtain the segmented contour to obtain the contour of the main root.

[0135] Furthermore, the instance segmentation model Mask2former is used to segment the main root in the image and obtain the segmented contour to obtain the contour of the main root, specifically:

[0136] The root system image I is input into the Mask2Former model, which segments the main root instances in I and outputs the segmentation mask M, M i Represents the binary mask of the i-th instance.

[0137] Use opencv to extract the contour point coordinates of the segmentation mask M and obtain the main root contour R i The contour point coordinate set RP i .

[0138] (6) Please refer to Figure 5 , interactively select a representative window among the root individuals, and use the window to intercept the outline of the main root to obtain the main root contour point set within the window.

[0139] Furthermore, the interactive selection of a representative window among the root individuals is performed, and the outline of the main root is intercepted using the window to obtain a set of main root outline points within the window, specifically:

[0140] The user uses the interactive graphical interface built with the pyqt library to click and drag to interactively select several representative rectangular windows (x1, y1, x2, y2) on the root system individual image, where x1 and y1 represent the coordinates of the upper left corner of the window, and x2 and y2 represent the coordinates of the lower right corner of the window.

[0141] After getting the window selected by the user, traverse the existing main root contour set C = {C1, C2, ..., C m}, check whether some points of each contour are located within the window, where m represents the total number of contours.

[0142] If the contour C is determined i If part of the image is within the window, it will be cropped. The specific process is as follows:

[0143] First, draw the contour with a fill value of 255 on the image A with all zeros, which is the same size as the root individual image. Then create a cropping region R = [x1, y1, x2, y2]. Then use the cropping region to crop A. Finally, use the OpenCV library to extract the coordinates of the contour points in the area with a value of 255 in the cropping region to obtain the contour C in the window. i The contour point coordinate set P i .

[0144] (7) Please refer to Figure 6 、 Figure 7 、 Figure 8 , use the improved Transfiner model to segment the lateral roots in the window, obtain the segmented contours, and obtain the contour point set of the lateral roots in the window; the improved Transfiner model uses snake convolution in the backbone network Backbone to replace the convolution operation of the first stage in the original backbone network Backbone, and uses KAN to replace the first convolution operation of stage 2 in the original backbone network Backbone.

[0145] Furthermore, the improved instance segmentation model Transfiner is used to segment the lateral roots in the window, and the segmented contours are obtained to obtain the contour point set of the lateral roots in the window, specifically:

[0146] The window area I[y i1 :y i2 ,x i1 :x i2 ] Input the improved instance segmentation model Transfiner, the model segments the lateral root instances in I and outputs the segmentation mask M, M i Represents the binary mask of the i-th instance.

[0147] Use opencv to extract the contour point coordinates of the segmentation mask M to obtain the lateral root contour S i Contour point coordinate set SP i .

[0148] The improved instance segmentation model Transfiner includes a backbone network, a neck network, a head network, and a mask correction part. The backbone network is used to extract image feature information. The neck network is used to fuse the features extracted by the backbone network, making the features learned by the neck network more diverse. The features are then handed over to the subsequent head network for segmentation, thereby improving network performance. Finally, the mask correction part refines the segmented mask to obtain the final segmentation result. The specific improvements are as follows:

[0149] In the backbone network Backbone, snake convolution is used to replace the convolution operation of Stem in the original backbone network, and KAN is used to replace the first convolution operation of the first stage in the original backbone network to form the KBlock module.

[0150] Because roots are elongated and their sizes vary widely, ordinary convolutions struggle to accurately fit their appearance. However, snake convolutions, with their kernels constrained by continuity, can more freely adapt to structural learning features, especially for elongated tubular structures.

[0151] The calculation formula of snake convolution is as follows:

[0152]

[0153] K=∑ K‘ B(K′,K)·K′

[0154] Among them, Δ={δ|δ∈[-1,1]} is K i+1 Compared with K i Added offset.

[0155] Using snake convolution to replace the first-stage convolution layer in the backbone network can enable the shallow network to better learn the shape and contour features in the image.

[0156] Compared to the MLP network, the KAN module removes the linear layer and parameterizes the activation function as a learnable one-dimensional spline function, placing it on the edges rather than the nodes. Compared to the MLP network, the KAN module has higher fitting accuracy. The specific formula is as follows:

[0157]

[0158] Among them, φ q,p :[0,1]->R,Φ q :R->R.

[0159] Here, the KAN module is used to replace the first convolution operation in the first stage of the original backbone network. The features are projected into the latent space before feature extraction, which can better extract features.

[0160] (8) Using the set of contour points of the main root and lateral roots within the window, OpenCV was used to measure the contour perimeter and area of the main root and lateral roots. The respective perimeters and areas were then divided by the number of main roots and lateral roots, respectively, to obtain the average perimeter and average area of the main root and lateral roots within the window.

[0161] Furthermore, the root phenotype index calculation within the window is performed by using the contour point set of the main root and lateral root within the window, specifically:

[0162] The perimeter and area of the main root and lateral root outlines in the window are measured by OpenCV, and then divided by the number of main roots and the number of lateral roots respectively to obtain the average perimeter and average area of the main root and lateral roots in the window. The specific formula is as follows:

[0163]

[0164] Among them, L avg,rootbone , L avg,sideroot represents the average circumference of the main root and lateral roots, A avg,rootbone 、A avg,sideroot represents the average area of main root and lateral root, L rootbone [i], L sideroot [j] represents the perimeter of the main root i and lateral root j measured by OpenCV, A rootbone [i]、A sideroot [j] represents the area of main root i and lateral root j measured by OpenCV, N rootbone Indicates the number of main roots in the window, N sideroot Indicates the number of lateral roots within the window.

[0165] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0166] Based on the same concept as the deep learning-based crop root phenotyping and index calculation method in the above-mentioned embodiment, the present invention also provides a deep learning-based crop root phenotyping and index calculation system, which can be used to execute the above-mentioned deep learning-based crop root phenotyping and index calculation method. For ease of explanation, the structural diagram of the embodiment of the deep learning-based crop root phenotyping and index calculation system only shows the parts related to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and it may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0167] See also Figure 9 In another embodiment of the present application, a crop root phenotype detection and index calculation system 100 based on deep learning is provided, which includes an image processing module 101, a target detection module 102, an effective area extraction module 103, a main root segmentation module 104, a lateral root segmentation module 105 and a root phenotype calculation module 106;

[0168] The image processing module 101 is used to perform edge enhancement on the root system image using side window filtering, and perform data expansion on the enhanced root system image;

[0169] The target detection module 102 is used to detect individual roots in the root image after data expansion using the target detection model RT-DETR, obtain a detection frame of the individual roots, and use the detection frame to intercept the individual roots in the image;

[0170] The effective region extraction module 103 detects the root base and stem in the root individual image using the target detection model RT-DETR to obtain the detection frames of the root base and stem, and removes part of the base and stem regions according to the positions of the base and stem detection frames in the image, thereby extracting the effective region without losing the root region;

[0171] The main root segmentation module 104 is used to segment the main root in the image using the instance segmentation model Mask2former and obtain the segmented contour to obtain the main root contour; interactively select a window with a relatively medium density of lateral roots above the main root, and use the window to intercept the main root contour to obtain a set of main root contour points within the window;

[0172] The lateral root segmentation module 105 is used to segment the lateral roots in the window using an improved Transfiner model to obtain the segmented contours and obtain a set of contour points of the lateral roots in the window; the improved Transfiner model uses a snake convolution in the backbone network to replace the convolution operation in the first stage of the original backbone network, and uses a KAN module to replace the first convolution operation in the first stage of the original backbone network;

[0173] The root phenotype calculation module 106 is configured to calculate an index of the root phenotype within the window based on the main root contour point set and the lateral root contour point set within the window.

[0174] It should be noted that the crop root phenotype detection and index calculation system based on deep learning of the present invention corresponds one-to-one to the crop root phenotype detection and index calculation method based on deep learning of the present invention. The technical features and beneficial effects described in the above-mentioned embodiments of the crop root phenotype detection and index calculation method based on deep learning are all applicable to the embodiments of the crop root phenotype detection and index calculation method based on deep learning. For specific contents, please refer to the description in the embodiments of the method of the present invention. No further details will be given here. This is hereby declared.

[0175] In addition, in the implementation of the crop root phenotype detection and index calculation system based on deep learning in the above embodiment, the logical division of each program module is only an example. In actual application, the above functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the crop root phenotype detection and index calculation system based on deep learning is divided into different program modules to complete all or part of the functions described above.

[0176] See also Figure 10In one embodiment, an electronic device for implementing a method for crop root phenotype detection and index calculation based on deep learning is provided. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a crop root phenotype detection and index calculation program 203 based on deep learning.

[0177] The first memory 202 includes at least one type of readable storage medium, including flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the first memory 202 may be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 may also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 200. Furthermore, the first memory 202 may include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can be used not only to store application software installed on the electronic device 200 and various types of data, such as the code of the deep learning-based crop root phenotyping and index calculation program 203, but also to temporarily store data that has been output or is about to be output.

[0178] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the first memory 202, as well as calling data stored in the first memory 202, to perform various functions of the electronic device 200 and process data.

[0179] Figure 10 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 10The structure shown does not constitute a limitation on the electronic device 200 , and the electronic device 200 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0180] The deep learning-based crop root phenotype detection and index calculation program 203 stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When running in the first processor 201, it can achieve the following:

[0181] Use side window filtering to enhance the edge of the root system image, and perform data expansion on the enhanced root system image;

[0182] The target detection model RT-DETR is used to detect individual roots in the root system image after data expansion, and the detection frame of the individual roots is obtained. The detection frame is then used to intercept the individual roots in the image.

[0183] The target detection model RT-DETR is used to detect the root base and stem in the root individual image, and the detection frames of the root base and stem are obtained. The base and stem regions are then partially removed according to the positions of the base and stem detection frames in the image, thus extracting the effective region without losing the root area.

[0184] The Mask2former instance segmentation model is used to segment the main root in the image and obtain the segmented contour to obtain the main root contour. A window with a relatively medium density of lateral roots above the main root is interactively selected and the main root contour is intercepted using the window to obtain the main root contour point set within the window.

[0185] The improved Transfiner model is used to segment the lateral roots in the window, obtain the segmented contours, and obtain the contour point set of the lateral roots in the window; the improved Transfiner model uses snake convolution in the backbone network to replace the convolution operation in the first stage of the original backbone network, and uses the KAN module to replace the first convolution operation in the first stage of the original backbone network;

[0186] Based on the contour point set of the main root and the contour point set of the lateral root in the window, the index of the root phenotype in the window is calculated.

[0187] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0188] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0189] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0190] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for detecting and calculating crop root phenotypes and indicators based on deep learning, characterized in that: The steps include: Input image, use side window filtering to perform edge enhancement on the root system image, and perform data expansion on the enhanced root system image; The target detection model RT-DETR is used to detect individual roots in the root system image after data expansion, and the detection frame of the individual roots is obtained. The detection frame is then used to intercept the individual roots in the image. The target detection model RT-DETR is used to detect the root base and stem in the root individual image, and the detection frames of the root base and stem are obtained. The base and stem regions are then partially removed according to the positions of the base and stem detection frames in the image, thus extracting the effective region without losing the root area. The Mask2former instance segmentation model is used to segment the main root in the image and obtain the segmented contour to obtain the main root contour. A window with a relatively medium density of lateral roots above the main root is interactively selected and the main root contour is intercepted using the window to obtain the main root contour point set within the window. The improved Transfiner model is used to segment the lateral roots in the window, obtain the segmented contours, and obtain the contour point set of the lateral roots in the window; the improved Transfiner model uses snake convolution in the backbone network to replace the convolution operation in the first stage of the original backbone network, and uses the KAN module to replace the first convolution operation in the first stage of the original backbone network; Based on the contour point set of the main root and the contour point set of the lateral root in the window, the index of the root phenotype in the window is calculated.

2. The crop root phenotype detection and index calculation method based on deep learning according to claim 1, characterized in that: The edge enhancement of the root system image is performed using side window filtering, specifically: During the processing of each pixel, the side window filter evaluates multiple directions in the neighborhood of the current pixel and selects the direction that best suits the pixel structure for filtering; The data expansion of the enhanced root system image is specifically performed as follows: Randomly select data augmentation methods from Gaussian blur, affine transformation, snow enhancement, horizontal flip, and vertical flip to increase data and improve the generalization of the model.

3. The method for calculating crop root phenotypic indicators according to claim 1, wherein: The target detection model RT-DETR is used to detect individual roots in the root image after data expansion, to obtain a detection frame of the individual roots, and to use the detection frame to intercept the individual roots in the image, specifically: The root image is input into the target detection model RT-DETR, which learns the appearance of the root system. The trained weights are then used to detect individual roots in the image, and the bounding box set β of the individual roots is returned. Specifically, β={b1,b2,...,b N } Where N is the number of detected root individuals, and each bounding box b i For a quadruple b i =(x i1 ,y i1 ,x i2 ,y i2 ), which represent the coordinates of the upper left corner and lower right corner of the bounding box respectively; Get the bounding box b of each root individual i =(x i1 ,y i1 ,x i2 ,y i2 ), and then use these bounding boxes to crop the image I and extract the image sub-region of each root individual. The specific formula is as follows: I i =I[y i1 :y i2 ,x i1 :x i2 ] Among them, I i Indicates that according to the bounding box b i The image area corresponding to the i-th root individual is cut out.

4. The method for calculating crop root phenotypic indicators according to claim 3, wherein: The target detection model RT-DETR is used to detect the root base and stem in the root individual image, obtain the detection frame of the root base and stem, and remove part of the base and stem area according to the position of the base and stem detection frame in the image, extracting the effective area without losing the root area, specifically: The root individual image I is input into the target detection model RT-DETR, the appearance of the root base is learned, and then the trained weights are used to detect the base and stem of the root individual in the image, and the bounding box γ of the base and stem of the root individual is returned. b and γ s , which respectively contain (x b1 ,y b1 ,x b2 ,y b2 ) and (x s1 ,y s1 ,x s2 ,y s2 ), which represent the x- and y-coordinates of the upper left corner of the root individual base bounding box, the x- and y-coordinates of the lower right corner of the root individual base bounding box, the x- and y-coordinates of the upper left corner of the root individual stem bounding box, and the x- and y-coordinates of the lower right corner of the root individual stem bounding box, respectively; Get the bounding box of the root individual base and stem (x b1 ,y b1 ,x b2 ,y b2 )、(x s1 ,y s1 ,x s2 ,y s2 ), the bounding box is used to crop the image I, specifically: First, the image orientation is determined based on the height and width of the root system image. If the image height is greater than the image width, the image is vertical, otherwise the image is horizontal. Then, the bounding box of the base and stem of the root individual γ b and γ s Calculate the center point. The calculation formula of the center point coordinates is as follows: Among them, (x c ,y c ) represents the center point position; If the image is vertical, first determine the center point (x c ,y c ) in the image: If (x c ,y c ) is located in the upper half of the image, i.e. Then from the center point (x c ,y c ) starts, keep the upper half of the image; if (x c ,y c ) is located in the lower half of the image, i.e. Then from the center point (x c ,y c ) starts, retaining the lower half of the image; If the image is horizontal, first determine the center point (x c ,y c ) in the image: If (x c ,y c ) is located in the left half of the image, i.e. Then from the center point (x c ,y c ) starts, retaining the left half of the image; If (x c ,y c ) is located in the right half of the image, i.e. Then from the center point (x c ,y c ) starts, retaining the right half of the image; Through the above steps, part of the base and stem area of the root system is removed, thereby extracting the effective root area.

5. The method for detecting and calculating crop root phenotypes and indices based on deep learning according to claim 1, characterized in that: The instance segmentation model Mask2former is used to segment the main root in the image and obtain the segmented contour to obtain the contour of the main root, specifically: The root system image I is input into the Mask2Former model, which segments the main root instances in the root system image I and outputs the segmentation mask M; Use opencv to extract the contour point coordinates of the segmentation mask M and obtain the main root contour R i The contour point coordinate set RP i .

6. The method for detecting and calculating crop root phenotypes and indices based on deep learning according to claim 1, characterized in that: The method further comprises: selecting a representative window in the root system individual, and using the window to intercept the outline of the main root to obtain a main root outline point set within the window, specifically: Use the interactive graphical interface built with the pyqt library to click and drag to select several representative rectangular windows (x1, y1, x2, y2) on the root system individual image, where x1 and y1 represent the coordinates of the upper left corner of the window, and x2 and y2 represent the coordinates of the lower right corner of the window; After getting the window selected by the user, traverse the existing main root contour set C = {C1, C2, ..., C m }, check whether some points of each contour are located within the window, where m represents the total number of contours; If the contour C is determined i If part of the image is within the window, it will be cropped. The specific process is as follows: First, draw the contour with a fill value of 255 on the image A with all zeros, which is the same size as the root individual image. Then create a cropping region R = [x1, y1, x2, y2]. Then use the cropping region to crop A. Finally, use the OpenCV library to extract the coordinates of the contour points in the area with a value of 255 in the cropping region to obtain the contour C in the window. i The contour point coordinate set P i .

7. The method for detecting and calculating crop root phenotypes and indices based on deep learning according to claim 1, characterized in that: The improved instance segmentation model Transfiner is used to segment the lateral roots in the window, and the segmented contours are obtained to obtain the contour point set of the lateral roots in the window, specifically: The window area I[y i1 :y i2 ,x i1 :x i2 ] Input the improved instance segmentation model Transfiner, the improved instance segmentation model Transfiner segments the lateral root instances in I and outputs the segmentation mask M, M i represents the binary mask of the i-th instance; Use opencv to extract the contour point coordinates of the segmentation mask M to obtain the lateral root contour S i Contour point coordinate set SP i ; The improved instance segmentation model Transfiner includes a backbone network, a neck network, a head network, and a mask correction part. The backbone network is used to extract image feature information. The neck network is used to fuse the features extracted by the backbone network, making the features learned by the neck network more diverse. The features are then handed over to the subsequent head network for segmentation, thereby improving network performance. Finally, the mask correction part refines the segmented mask to obtain the final segmentation result. The specific improvements are as follows: In the backbone network, snake convolution is used to replace the convolution operation of the Stem in the original backbone network, and KAN is used to replace the first convolution operation in the first stage of the original backbone network to form the KBlock module. The snake convolution, due to the continuity constraint design of its convolution kernel, is more free to fit the structural learning features during learning, especially the slender tubular structure. Using snake convolution to replace the first stage convolution layer in the backbone network enables the shallow network to better learn the shape and contour features in the image. Compared with the MLP network, the KAN module removes the linear layer, parameterizes the activation function into a learnable one-dimensional spline function, and places it on the edge instead of the node; the KAN module replaces the first convolution operation in the first stage of the original backbone network, and projects the features into the latent space before feature extraction to better extract features.

8. The crop root phenotype detection and index calculation method based on deep learning according to claim 1, characterized in that: The root phenotype index calculation within the window is performed based on the main root contour point set and the lateral root contour point set within the window, specifically: Based on the contour point set of the main root and the contour point set of the lateral root in the window, the contour perimeter and area of the main root and lateral root are measured using OpenCV. Then, the respective perimeter and area are divided by the number of main roots and lateral roots respectively to obtain the average perimeter and average area indicators of the main root and lateral roots in the window.

9. A crop root phenotype detection and index calculation system based on deep learning, characterized by: A crop root phenotype detection and index calculation method based on deep learning applied to any one of claims 1-8, comprising an image processing module, a target detection module, an effective area extraction module, a main root segmentation module, a lateral root segmentation module, and a root phenotype calculation module; The image processing module is used to perform edge enhancement on the root system image using side window filtering, and perform data expansion on the enhanced root system image; The target detection module is used to detect individual roots in the root system image after data expansion using the target detection model RT-DETR, obtain a detection frame of the individual roots, and use the detection frame to intercept the individual roots in the image; The effective area extraction module uses the target detection model RT-DETR to detect the root base and stem in the root individual image, obtains the detection frame of the root base and stem, and removes part of the base and stem area according to the position of the base and stem detection frame in the image, extracting the effective area without losing the root area; The main root segmentation module is used to segment the main root in the image using the instance segmentation model Mask2former and obtain the segmented contour to obtain the main root contour; interactively select a window with a relatively medium density of lateral roots above the main root, and use the window to intercept the main root contour to obtain a set of main root contour points within the window; The lateral root segmentation module is used to segment the lateral roots in the window using an improved Transfiner model to obtain the segmented contours and obtain a set of contour points of the lateral roots in the window; the improved Transfiner model uses snake convolution in the backbone network to replace the convolution operation in the first stage of the original backbone network, and uses the KAN module to replace the first convolution operation in the first stage of the original backbone network; The root phenotype calculation module is used to calculate the index of the root phenotype in the window based on the outline point set of the main root and the outline point set of the lateral root in the window.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the crop root phenotype detection and indicator calculation method based on deep learning as described in any one of claims 1 to 7.

Images