Image restoration method for soybean leaf phenotype analysis

Abstract

The invention discloses an image restoration method for soybean leaf phenotype analysis. The method comprises the following steps: firstly, complementing missing edge information of soybean leaves through an edge generation network; then, extracting a coarse-to-fine layered feature map from the complemented edge information by using a structural auto-encoder, and inputting the layered feature map as a guide feature into an image restoration network to assist a restoration process; the soybean leaf structure information is accurately repaired based on the guiding characteristics; and finally, according to the completely repaired leaf image, obtaining phenotypic parameters such as a leaf area index, a structure texture and a color of the soybean plant. By means of the mode, gradient structured edge information and hierarchical feature guidance are introduced, the repairing effect of the soybean leaf picture can be improved, and therefore support is better provided for accurate and complete analysis of soybean leaf phenotype data.

Description

Technical Field

[0001] The invention relates to the field of computer vision technology, and in particular to an image restoration method for soybean leaf phenotype analysis. Background Art

[0002] In soybean breeding, soybean phenotyping technology plays a vital role. Through the precise measurement and analysis of soybean leaf phenotypic characteristics, such as leaf area index, leaf length, leaf width, leaf area, leaf shape, leaf color, vein distribution and texture, key traits such as soybean growth characteristics, stress resistance, environmental adaptability and yield potential can be comprehensively evaluated. For example, the leaf area index can be reflected within a certain range, and the yield of crops increases with the increase of leaf area index. The above phenotypic information of soybean leaves can not only provide scientific guidance for breeding work, but also play an important role in crop growth monitoring, precision agriculture and crop model optimization. However, how to efficiently obtain these key parameters is still one of the difficulties and challenges of current research.

[0003] It is particularly difficult to obtain phenotypic data of soybean leaves in a natural environment. This is because soybean leaves often show a complex spatial distribution during growth, and there is significant occlusion and overlap between leaves, which makes the traditional contact phenotypic data collection method inefficient and easy to cause irreversible damage to the plants. In addition, even if non-contact image acquisition equipment is used, due to the existence of occlusion problems, it is impossible to directly obtain complete leaf morphological information. The occluded leaf area usually leads to incomplete phenotypic data, which affects the accuracy and comprehensiveness of the phenotypic analysis. These problems not only affect the evaluation of soybean growth characteristics, but also have a direct negative impact on breeding efficiency.

[0004] In order to solve the above problems, image restoration technology provides a solution for reconstructing leaf morphological information in occluded areas. Through computer vision and deep learning technology, key features can be extracted from existing image data, and the occluded leaf areas can be reasonably inferred and repaired to generate a complete leaf image. Such a technological breakthrough will provide strong technical support for the efficient and lossless acquisition of soybean phenotypic parameters, and also provide new research directions and application prospects for promoting soybean breeding work towards intelligence and precision. However, the existing image restoration methods such as classic image restoration methods and deep learning-based image restoration methods still have poor restoration effects on soybean leaf images, and there is room for improvement in the quality of image restoration, which in turn affects the accurate analysis of soybean leaf phenotypic data. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the deficiencies in the prior art, the present invention provides an image restoration method for soybean leaf phenotype analysis, which can solve the above technical problems.

[0007] (II) Technical solution

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions: an image restoration method for soybean leaf phenotype analysis, comprising the following steps:

[0009] S1. Take a picture of a soybean plant to obtain a picture of soybean leaves;

[0010] S2, constructing an edge generation network, using the soybean leaf image and the edge generation network to generate a predicted edge output graph, wherein the edge output graph is a complete edge image;

[0011] S3, constructing a structural autoencoder, using the edge output map and the structural autoencoder to obtain a hierarchical feature map; constructing an image restoration network, using the hierarchical feature map and the image restoration network to generate a predicted restoration output map, wherein the restoration output map is a restored complete leaf image;

[0012] S4. Obtain the phenotypic parameters of soybean leaves according to the repair output map.

[0013] Preferably, in step S1, photographing the soybean plants is specifically performed by photographing the soybean plants perpendicularly to the ground using a photographing device to obtain a picture of the soybean plants.

[0014] Preferably, step S1 further comprises: using a target detection model to identify each leaf in the soybean plant image; and using an image segmentation model to segment each leaf in the soybean plant image to obtain a segmented soybean leaf image.

[0015] Preferably, the target detection model is a yolov8 model; and the image segmentation model is a SAM segmentation model.

[0016] Preferably, step S2 specifically includes: obtaining a real edge map and a mask map of the soybean leaf image; further, performing Hadamard product on the real edge map and the mask map to obtain an incomplete edge map; inputting the incomplete edge map and the mask map into an edge generation network to generate a predicted edge output map.

[0017] Preferably, the structural autoencoder includes 3 layers of downsampling convolution, 3 layers of residual blocks and 3 layers of upsampling convolution.

[0018] Preferably, step S3 specifically includes: performing Hadamard product on the soybean leaf image and the mask image to obtain a defective leaf image; inputting the defective leaf image, the mask image and the hierarchical feature map into an image restoration network to generate a predicted restoration output image.

[0019] Preferably, both the edge generation network and the image restoration network adopt the u-net architecture.

[0020] Preferably, step S4 specifically includes: calculating the leaf area index according to the total leaf area and the leaf projection area of ​​the repaired output image.

[0021] (III) Beneficial effects

[0022] Compared with the prior art, the present invention provides an image restoration method for soybean leaf phenotypic analysis, which has the following beneficial effects: the present invention first completes the missing edge information of soybean leaves through an edge generation network; then uses a structural autoencoder to extract a hierarchical feature map from coarse to fine from the completed edge information, and inputs it into the image restoration network as a guiding feature to assist the restoration process; based on the guiding feature, accurate restoration of the soybean leaf structural information is achieved; finally, based on the restored complete leaf image, phenotypic parameters such as leaf area index, structural texture, and color of the soybean plant are obtained. In the above manner, the present invention can improve the restoration effect of soybean leaf images by introducing gradient structured edge information and hierarchical feature guidance, thereby better providing support for the accurate and complete analysis of soybean leaf phenotypic data. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of the steps of an image restoration method for soybean leaf phenotype analysis according to the present invention;

[0024] Figure 2 The architecture diagram of the edge generation network, structural autoencoder and image restoration network of the present invention (the dotted arrow in the figure indicates that it is only used in the restoration stage, that is, only used in the image restoration network);

[0025] Figure 3 Flowchart for calculating leaf area index. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The present invention provides an image restoration method for soybean leaf phenotype analysis, comprising the following steps:

[0028] S1. Take pictures of soybean plants to obtain pictures of soybean leaves.

[0029] Preferably, in step S1, the soybean plant is photographed by a mobile phone or a camera or other shooting device in a single perspective vertically to the ground to obtain a soybean plant picture. Further, step S1 also includes: using a target detection model to identify each leaf in the soybean plant picture, and extracting some soybean plant pictures for annotation labels to train the target detection model for leaves in the soybean plant picture; further, using a picture segmentation model to segment each leaf in the soybean plant picture to obtain each segmented soybean leaf picture. It can be understood that each soybean leaf picture corresponds to a soybean leaf, and most of the soybean leaf pictures are missing due to factors such as occlusion, that is, the soybean leaf picture is an incomplete soybean leaf picture. The present invention is used to repair the soybean leaf picture to make its soybean leaf complete. Preferably, the above-mentioned target detection model is a yolov8 model; the picture segmentation model is a SAM segmentation model, and other target detection models and picture segmentation models of the prior art can also be used. In addition, after the above segmentation is completed, preferably, the soybean leaf picture is further filled and proportionally scaled to a resolution of 256*256 for subsequent data sets.

[0030] The above step S1 corresponds to the data collection and preprocessing stage of the present invention, which belongs to the scope of data set production. Finally, after data enhancement, a training set of 10,968 soybean leaf images and a test set of 1,932 soybean leaf images are obtained. In addition, the edge generation network of step S2, the structured autoencoder of step S3, and the image restoration network based on guided features all belong to the model structure scope. Step S4 is to analyze the soybean leaf phenotypic parameters based on the restored complete leaf image.

[0031] S2. Construct an edge generation network and use the soybean leaf image and the edge generation network to generate a predicted edge output graph.

[0032] Preferably, the edge generation network of the present invention is designed using a u-net architecture (network), and step S2 specifically includes: obtaining a real edge map and a mask map of a soybean leaf image; further, performing a Hadamard product on the real edge map and the mask map to obtain an incomplete edge image; inputting the incomplete edge image and the mask map into the edge generation network to generate a predicted edge output map. It can be understood that the edge output map is a predicted complete edge image of the soybean leaf.

[0033] Specifically, for soybean leaf image I m ∈R C×H×W Use the sobel operator to detect and get the real edge map E gt ∈R C ×H×W ; Soybean leaf picture I m ∈R C×H×WPerform mask processing to obtain a mask map M∈R C×H×W , the mask image can be a binary image: the pixels of the area to be repaired are 1, and the pixels of the background are 0.

[0034] Furthermore, for the real edge graph E gt Perform Hadamard product with the mask image M to obtain the incomplete edge image E input ∈R C×H×W =E gt ⊙(1-M); where C represents the number of channels, H represents the height of the image, W represents the width of the image, and the mask map M is used as a precondition; ⊙ represents the Hadamard product, which is a matrix operation that multiplies elements by position. The output of the edge generation network (i.e., the edge output map) E pred ∈R C×H×W From the following formula (1), we get:

[0035] E pred =G1(E input ,M) (1)

[0036] Among them, G1 is the edge generation network, using the incomplete edge image E input and mask map M as the input of the edge generation network.

[0037] Specifically, the edge generation network G1 includes the following modules:

[0038] Broken Edge Image E input First, a 3*3 convolution is performed to obtain a feature map with richer information. The number of channels is 48, and then it is sent as input to the u-net network. The u-net network contains seven Transformer blocks, of which the first three Transformer blocks are used as encoder layers, the middle Transformer block is used as the bottleneck layer (Bottleneck), and the last three Transformer blocks are used as decoder layers.

[0039] The Transformer block consists of a MultiHead Transposed Attention with Gate mechanism (MTAG) and a Gated-Dconv feed-forward network (GDFN) implemented by gated deep convolution. The gate mechanism can be seen as a neural network layer whose output is the product of two linear transformation components of the input. The gate mechanism plays an important role in the field of image restoration: it can guide effective information to participate in the restoration and suppress useless information, thereby helping to learn and restore the local image structure of the soybean leaf image. The output of the MultiHead Transposed Attention with Gate mechanism (MTAG) is X ～ ∈R c`×H`×W` It can be written as the following formula (2):

[0040] X ～ =X^+X

[0041] X^=A⊙GA=MDTA(X) G=φ(W d X) (2)

[0042] Among them, MDTA is the multi-head transposed attention mechanism, φ is the GELU activation function, and W d are learnable parameters, A and G are the outputs of the two linear transformation components, and ⊙ represents the Hadamard product.

[0043] Consistent with the classic u-net, there is a skip connection between each Transformer block of the encoder and the corresponding block of the decoder, which is used to pass the high-resolution feature map in the encoder directly to the decoder, which helps to maintain the details of the image and enables the decoder to use the features of the encoder for more refined restoration. For the first three Transformer blocks in the u-net network, each block is followed by an MPD block. The MPD block (mask-aware pixel-shuffle downsampling module) can effectively retain the visible information extracted from the damaged image of the soybean leaf picture, while ensuring that the model can fully utilize high-level useful information in the inference stage (image restoration network). Output of the MPD block Receive the output of the previous Transformer block And the mask image M i (i∈1, 2, 3) as input, where M i Depending on the position of the Transformer block, corresponding downsampling is performed to achieve With the same H and W. For the first four Transformer blocks, it is formulated as follows (3):

[0044]

[0045] Among them, MPD(·) represents MPD block, and TB(·) represents Transformer block.

[0046] In the edge generation network, after the u-net network, there is another Transformer block as a refinement module to further optimize the output of the u-net network, and finally a 3*3 convolution is performed to obtain the final output: Edge output diagram E pred .

[0047] In addition, preferably, the loss function selection for the edge generation network includes a combination of the following prior art loss functions: L1 loss (L1loss), which is used to make the restoration result reasonable in context; feature matching loss L fm (featurematching loss), used to stabilize training; structural similarity loss L ssim (SSIM loss, LSSIM), which is used to optimize the structural similarity of the image and reduce the distortion in the restoration process, which is beneficial to the restoration of low-level features such as veins, contours and boundaries; and adversarial loss L adv，1 (adversarial loss) is used to improve the overall output quality. The final loss function of the edge generation network is expressed as follows (4):

[0048] L total (E pred , E gt )=λ 11 L1+λ 21 L fm +λ 31 L ssim +λ 41 L adv，1 (4)

[0049] Among them, λ 11 ,λ 21 ,λ 31 ,λ 41 They are 1, 10, 0.1, and 0.01 respectively. In addition, the edge generation network can also use other loss functions in the prior art.

[0050] At this point, the edge generation network part is over. By training the edge generation network, the predicted edge output graph E can be generated. pred .

[0051] S3, construct a structural autoencoder, use the edge output map and the structural autoencoder to obtain a hierarchical feature map; construct an image restoration network, and use the hierarchical feature map and the image restoration network to generate a predicted restoration output map. It can be understood that the above restoration output map is the restoration of the complete leaf image.

[0052] The structured autoencoder of the present invention is mainly used to extract the edge generation network output E pred hierarchical features; specifically, the structural autoencoder includes 3 layers of downsample convolution as an encoder, 3 layers of residual blocks (ResNetBlock) as the middle layer and 3 layers of upsample convolution as a decoder.

[0053] Specifically, step S3 obtains the final output of the middle layer of the structured autoencoder and the output of the decoder layer 3, that is, the above hierarchical feature map includes 4 feature maps from coarse to fine. i i∈(1, 2, 3, 4), these 4 feature maps are embedded and fused with the 4-layer input (encoder and middle layer) of the u-net in the following image restoration network, which plays the role of guiding the restoration features, so that the image restoration network can more effectively learn image details and structural information, thereby better restoring the soybean leaf image. The above feature map f i It can be expressed as the following formula (5):

[0054] f1, f2, f3, f4 = EFC (E pred ) (5)

[0055] Among them, EFC stands for Edge Feature autoencoder.

[0056] The main architecture of the image restoration network of the present invention is similar to the above-mentioned edge generation network, and is also designed using the u-net architecture. The image restoration network processing process of step S3 specifically includes: first, similarly, for the soybean leaf image I m ∈R C ×H×W With the mask map M∈R C×H×W Perform Hadamard product to obtain the incomplete leaf image I input ∈R C×H×W =I m ⊙(1-M); Next, the incomplete leaf image I input , mask map M and the 4 feature maps f obtained by the above structure autoencoder i The hierarchical feature map of is input into the image restoration network to generate the predicted restoration output map. The output of the image restoration network (restoration output map) I pred ∈R C×H×WIt can be obtained from the following formula (6):

[0057] I pred =G2(I input , M, f i )i∈(1,2,3,4) (6)

[0058] Among them, G2 is an image restoration network (the image restoration network is based on the hierarchical feature map as the guide feature), using the incomplete leaf image I input , mask map M and feature map f i As input to the image restoration network.

[0059] Different from the above-mentioned edge generation network, the present invention adds a feature map f in front of the three-layer encoder and the middle layer in the image restoration network u-net architecture. i , to guide the restoration. In the image restoration network, for the first four Transformer blocks, the above formula (3) is changed to the following formula (7) to adapt to the image restoration network:

[0060]

[0061] Among them, MPD(·) represents MPD block, TB(·) represents Transformer block, Same reason broken leaves picture I input Obtained through 3*3 convolution.

[0062] In addition, preferably, the loss function selection for the image restoration network includes a combination of the following prior art loss functions: L1 loss, which is used to ensure the consistency of image reconstruction in context; style loss L style (styleloss), used to measure the difference in style; perceptual loss L perc (perceptual loss), used to compare high-level perceptual features extracted from the pre-trained network; and adversarial loss L adv，2 (adversarial loss) is used to improve the overall output quality. The final loss function of the image restoration network is expressed as follows (8):

[0063] L total (I pred , I gt )=λ 12 L1+λ 22 L style +λ 32 L perc +λ 42 L adv，2 (8)

[0064] Among them, Igt represents the real image of soybean leaves, λ 12 ,λ 22 ,λ 32 ,λ 42 They are 1, 250, 0.1, and 0.01 respectively.

[0065] At this point, the image restoration network based on the guided feature is completed. By training the image restoration network, the final predicted restoration output image I can be generated. pred .

[0066] S4. Output graph L according to the above repair pred That is, the complete leaf image is repaired to obtain the leaf area index, structural texture, color and other phenotypic parameters of the soybean leaf; preferably, the leaf area index is calculated according to the total leaf area and leaf projection area of ​​the repaired output image. The leaf area index calculation formula is the ratio of the total leaf area to the area of ​​the leaf projected on the ground (i.e., the leaf projection area), which can be expressed as the following formula (9):

[0067] LAI=A GL / A PGL (9)

[0068] Among them, A GL represents the total leaf area, A PGL Represents the leaf projection area. According to the definition of this pair of LAI values, due to the shading effect between leaves, the projection area A PGL Usually smaller than the actual green leaf area A GL , so the value of LAI is usually greater than or equal to 1. In Formula 9, the total area of ​​soybean leaves in the soybean plant image taken from a bird's-eye view is the projected area A PGL , the total area of ​​all complete leaves after repairing the occluded leaf area is A GL In addition, other methods in the prior art may also be used to calculate and obtain the phenotypic parameters of soybean leaves, and no excessive restrictions are imposed herein.

[0069] The following table 1 compares the image restoration experimental results of the method proposed by the present invention and the existing methods (Ours refers to the present invention):

[0070] Method Resolution PSNR↑ SSIM↑ FID↓ LPIPS↓ L1↓ ctsdg 256*256 27.5026 0.9210 23.7304 0.0529 2.4014 HINT 256*256 27.9738 0.9260 5.3208 0.0428 2.2630 T-former 256*256 28.1113 0.9241 6.1753 0.0420 2.3007 Ours 256*256 28.5241 0.9286 5.5295 0.0384 2.0746

[0071] It can be seen from Table 1 above that the present invention has excellent performance in terms of peak signal-to-noise ratio (PSNR), structural similarity (SSIM), similarity index (F ID, LPIPS), L1 loss index, etc.

[0072] It can be understood that the present invention is based on image restoration technology, combined with core theories and algorithms such as Transformer, u-net structural framework, attention mechanism, gating mechanism and autoencoder, and deeply explores image restoration methods for soybean leaf phenotypic analysis. The present invention specifically uses the edge information of soybean leaves as an additional guiding feature to improve the restoration effect. The precise restoration of the present invention includes three stages: first, the missing edge information of soybean leaves is completed through an edge generation network; then, the completed edge information is extracted from a coarse to fine hierarchical feature map using a structural autoencoder, and input into the image restoration network as a guiding feature to assist the restoration process; finally, based on the guiding features and the missing leaves, the precise restoration of the soybean leaf structural information is achieved.

[0073] The present invention is based on a deep learning network architecture and is used to repair the contour and texture of leaves in the occluded area of ​​a soybean leaf image. Compared with the existing repair methods, the present invention has the following beneficial effects: (1) In view of the problems of inaccurate structural contours and artifacts in the leaf repair process of the existing image repair model, the present invention introduces gradient structured edge information through u-net and hierarchical feature guidance through a structural autoencoder, thereby improving the repair effect. The present invention can effectively handle the common occlusion and noise problems of soybean leaf images in the natural environment, thereby providing more accurate and complete phenotypic data such as soybean plant leaf area index, leaf length, leaf width, etc., and better realizes the acquisition of soybean plant and leaf phenotypic data in a non-contact and convenient manner; (2) The present invention proposes a method for extracting edge output graphs using a structural autoencoder Hierarchical feature maps are embedded into the encoder and middle layer of the U-Net network from coarse to fine layer by layer, achieving accurate guidance of the restoration features, thereby improving the restoration quality of soybean leaf images; (3) The present invention introduces a multi-head transposed attention mechanism based on a gating mechanism, which can dynamically distinguish between effective information and useless information used for soybean leaf image restoration during the model training process; at the same time, useless and erroneous edge information is discarded after being extracted and introduced into the image restoration network, thereby improving the accuracy and stability of the image restoration effect; (4) According to the model, the complete leaf image is restored to obtain the phenotypic parameters of the soybean plant, such as leaf area index, structural texture, and color, so as to comprehensively evaluate the key traits of soybean, such as growth characteristics, environmental adaptability, and yield potential, and provide scientific guidance for breeding work.

[0074] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0075] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An image restoration method for soybean leaf phenotyping, characterized in that: The following steps are involved: S1. Take a picture of a soybean plant to obtain a picture of soybean leaves; S2, constructing an edge generation network, and using the soybean leaf image and the edge generation network to generate a predicted edge output graph, wherein the edge output graph is a complete edge image; S3, constructing a structural autoencoder, and using the edge output map and the structural autoencoder to obtain a hierarchical feature map; constructing an image restoration network, and using the hierarchical feature map and the image restoration network to generate a predicted restoration output map, wherein the restoration output map is a restored complete leaf image; S4. Obtaining phenotypic parameters of soybean leaves according to the repair output graph.

2. The image restoration method for soybean leaf phenotyping according to claim 1, characterized in that: In the step S1, the photographing of the soybean plants is specifically performed by photographing the soybean plants perpendicularly to the ground using a photographing device to obtain a picture of the soybean plants.

3. The image restoration method for soybean leaf phenotyping according to claim 2, characterized in that: The step S1 also includes: using a target detection model to identify each leaf in the soybean plant image; using an image segmentation model to segment each leaf in the soybean plant image to obtain the segmented soybean leaf image.

4. The image restoration method for soybean leaf phenotyping according to claim 3, characterized in that: The target detection model is the yolov8 model; the image segmentation model is the SAM segmentation model.

5. The image restoration method for soybean leaf phenotyping according to claim 1, characterized in that: The step S2 specifically includes: obtaining a real edge map and a mask map of the soybean leaf image; further, performing a Hadamard product on the real edge map and the mask map to obtain an incomplete edge map; and inputting the incomplete edge map and the mask map into the edge generation network to generate the predicted edge output map.

6. The image restoration method for soybean leaf phenotyping according to claim 1, characterized in that: The structural autoencoder includes 3 layers of downsampling convolution, 3 layers of residual blocks and 3 layers of upsampling convolution.

7. The image restoration method for soybean leaf phenotyping according to claim 5, characterized in that: The step S3 specifically includes: performing Hadamard product on the soybean leaf image and the mask image to obtain a defective leaf image; inputting the defective leaf image, the mask image and the hierarchical feature map into the image restoration network to generate the predicted restoration output image.

8. The image restoration method for soybean leaf phenotyping according to claim 1, characterized in that: The edge generation network and the image restoration network both adopt the u-net architecture.

9. The image restoration method for soybean leaf phenotyping according to claim 1, characterized in that: The step S4 specifically includes: calculating the leaf area index according to the total leaf area and the leaf projection area of ​​the repaired output image.

Images