A method, device and system for obtaining leaf phenotypes of living crops

Abstract

The present invention provides a method, device, system and application for obtaining leaf phenotypes of living crops, and the method comprises the following steps: S1, obtaining a phenotypic image of a leaf of a living crop; S2, drawing a leaf mask image of the phenotypic image; S3, enhancing the leaf mask and obtaining a training set and a test set; S4, training a Segformer neural network model using the training set; S5, testing the test set using the trained Segformer neural network model, and measuring the segmentation mask of the leaf; S6, determining the number of pixels of the segmentation mask; S7, calculating the leaf area according to the number of pixels and the preset pixel area. The present invention can achieve the purpose of on-site leaf phenotype measurement and image drawing of living crops, avoids the problem of inaccurate measurement caused by leaf dehydration and curling in vitro, improves the accuracy of measurement, and provides conditions for subsequent research and judgment on the type of deficiency and the severity of the disease.

Description

Technical Field

[0001] The present invention belongs to the technical field of plant phenotype detection, and in particular relates to a method, device and system for acquiring the phenotype of leaves of living crops. Background Art

[0002] Plant phenotyping plays an important role in agricultural science and plant breeding because it can reveal the relationship between plant morphological characteristics (such as leaf shape, size and color) and growth and development, as well as environmental adaptability. These morphological characteristics not only affect the photosynthesis efficiency of plants, but also have a direct impact on the yield and quality of crops. Among them, the phenotypic parameters of plants, such as leaf shape and area, not only determine the efficiency of plant utilization of light and nutrients, but also reflect the health of plants. Therefore, it is of great significance to develop efficient and accurate plant leaf phenotyping technology.

[0003] Currently, most plant leaf phenotypic measurements rely on traditional manual field measurement methods or use phenotypic instruments for measurement after sampling. Manual field measurements are affected by complex environments, measuring instruments, and the professional quality of personnel, resulting in large measurement errors. Compared with manual measurements, the use of phenotypic instruments for measurement has improved efficiency and accuracy to a certain extent. However, the leaf curling problem caused by the rapid loss of water after the plant leaves are sampled and separated from the body will lead to significantly smaller measured values ​​in the subsequent leaf area measurement step, and incomplete extraction of phenotypic information in the leaf image acquisition step. Flattening the curled leaves is time-consuming and labor-intensive, and the inability to complete phenotypic extraction in time when the sample size is large will lead to the loss of some important phenotypic characteristics of the subsequent sample leaves. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method, device and system for obtaining the phenotype of leaves of living crops, which can improve the accuracy of the phenotypic data of the leaves of living crops.

[0005] The present invention is implemented as follows: a method for obtaining leaf phenotypes of living crops, comprising the following steps:

[0006] S1. Obtain phenotypic images of living crop leaves;

[0007] S2, drawing a leaf mask image of the phenotypic image;

[0008] S3, performing enhancement processing on the leaf mask to obtain a training set and a test set;

[0009] S4, training the Segformer neural network model using the training set;

[0010] S5. Testing the test set using the trained Segformer neural network model to obtain a segmentation mask of the leaf;

[0011] S6, determining the number of pixels of the segmentation mask;

[0012] S7. Calculate the leaf area according to the number of pixel points and a preset pixel area.

[0013] Furthermore, the method of obtaining the phenotypic image of the leaves of the living crop comprises the following steps:

[0014] S101, calculating the displacement of the scanning unit according to the moving speed of the scanning unit;

[0015] S102, calculating the splicing position of each line array image according to the displacement of the scanning unit;

[0016] S103, stitching all the line array images into a complete phenotype image according to the stitching position of each line array image.

[0017] Furthermore, the calculation formula of the displacement of the scanning unit is:

[0018] ;

[0019] In the above formula, Indicates the moving speed of the scanning unit, Indicates time, Indicates scanning i Linear image The displacement of the scanning unit after

[0020] The calculation formula for the position of each linear image after stitching is:

[0021] ;

[0022] The stitching formula for the complete phenotype image is:

[0023] ;

[0024] In the above formula, N represents the number of line array images, y It is the frame width perpendicular to the scanning direction and is a constant.

[0025] Furthermore, the step of enhancing the leaf mask and obtaining a training set and a test set comprises the following steps:

[0026] S301, setting a small-angle flip operator;

[0027] S302, performing multiple flipping of the mask parts of the leaf mask image and the phenotype image at small angles to obtain a flipped mask image and a flipped phenotype image, and calculating pixel coordinates of the flipped mask image and the flipped phenotype image according to the flipping operator;

[0028] S303 , dividing the flipped mask image and the flipped phenotype image into a training set and a test set respectively according to the ratio.

[0029] Furthermore, the calculation formula of the flip operator is:

[0030] ;

[0031] In the above formula, Indicates the flip angle, which is a random value within ±30°.

[0032] Furthermore, the method for calculating the preset pixel area includes the following steps:

[0033] S701, setting scanning parameters and the area of ​​the reference calibrator;

[0034] S702, obtaining a reference marker image containing a preset background;

[0035] S703, marking a mask of the reference calibrator in the reference calibrator image, and calculating the number of pixels thereof according to the mask;

[0036] S704: Calculate the preset pixel area according to the number of pixels and a pixel area calculation formula.

[0037] The present invention also provides a system based on the above-mentioned method for obtaining leaf phenotypes of living crops, comprising:

[0038] An acquisition module, used for acquiring phenotypic images of leaves of living crops;

[0039] A drawing module, used for drawing a leaf mask image of the phenotypic image;

[0040] A processing module, used for enhancing the leaf mask and obtaining a training set and a test set;

[0041] A training module, used for training a Segformer neural network model using the training set;

[0042] A testing module, used to test the test set using a trained Segformer neural network model to measure the segmentation mask of the leaf;

[0043] A determination module, used to determine the number of pixels of the segmentation mask;

[0044] A calculation module is used to calculate the leaf area according to the number of pixel points and a preset pixel area.

[0045] The present invention also provides a device for acquiring leaf phenotypes of living crops, comprising: a main control unit, wherein the main control unit comprises a processing module and a storage module, wherein the storage module stores a computer program, and when the computer program is executed by the processing module, the steps of the above method are implemented.

[0046] The beneficial effects brought by the present invention are:

[0047] 1. The present invention can achieve the purpose of on-site measurement of living crop leaf phenotypes and image drawing, avoids the problem of inaccurate measurement caused by leaf dehydration and curling, improves the accuracy of measurement, and provides conditions for subsequent research and judgment on the type of deficiency and the severity of the disease.

[0048] 2. The present invention can meet the requirements of leaf phenotyping of different types and sizes, and expand the scale of the data set by adding processing technology, thereby improving the reliability of model training.

[0049] 3. The present invention improves the recognition of blade scanning in an angle deflection state through a small angle flip operator, thereby improving the accuracy of the calculation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of the method of the present invention;

[0051] Figure 2 for Figure 1 Schematic diagram of phenotypic image stitching in the method shown;

[0052] Figure 3 A 1:1 diagram of leaf area calculated using the present invention and manual annotation methods;

[0053] Figure 4 is a structural block diagram of the system in the present invention;

[0054] Figure 5 It is a structural stereogram of the device in the present invention;

[0055] Figure 6 A top view of the structure of the device of the present invention;

[0056] Figure 7 for Figure 6 Structural cross-section view along the AA axis;

[0057] Figure 8 for Figure 7 Structural cross-section view along the BB direction;

[0058] Fig. 9 The figure is a diagram showing the internal structure of the scanning body in the device of the present invention;

[0059] Fig.10 for Fig. 9A partial enlarged view of point I in the middle;

[0060] Fig.11 for Fig. 9 A partial enlarged view of position II in the middle;

[0061] Fig.12 It is a structural block diagram of the device in the present invention.

[0062] Figure numerals: 1. substrate; 2. optical axis; 3. scanning body; 4. optical axis seat; 5. rack; 6. pressure plate; 7. speed measuring wheel; 8. reference calibration mark image; 9. transparent cover; 10. upper shell cover; 11. lower shell; 12. display unit; 13. button unit; 14. power supply unit; 15. main control unit; 16. interface unit; 17. motor; 18. encoder; 19. scanning unit; 20. front sensing switch; 21. rear sensing switch; 22. first gear; 23. second gear; 24. third gear; 25. linear bearing. DETAILED DESCRIPTION

[0063] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention. Example

[0064] Take rice leaves as the test object, such as Figures 1 to 3 As shown, a method for obtaining leaf phenotypes of living crops comprises the following steps:

[0065] S1. Obtain 100 phenotypic images of rice leaves as needed. Figure 2 As shown, in this step, the phenotype image is spliced ​​by N linear array images, which includes the following steps:

[0066] S101, calculating the displacement of the scanning unit according to the moving speed of the scanning unit. Specifically, the calculation formula of the displacement of the scanning unit is:

[0067] ;

[0068] In the above formula, Indicates the moving speed of the scanning unit, Indicates time, Indicates scanning i Linear image The displacement of the scanning unit after.

[0069] S102, calculating the splicing position of each line array image according to the displacement of the scanning unit. Specifically, the calculation formula of the position of each line array image after splicing is: That is to say, the coordinates of the first line array image are , the displacement of the scanning unit after scanning the first linear array image is , so the coordinates of the second line array image are , ; The coordinates of the third line array image are , ..., the coordinates of the Nth line array image are .

[0070] S103, stitching all the line array images into a complete phenotype image according to the stitching position of each line array image. Specifically, the stitching formula of the complete phenotype image is:

[0071] ;

[0072] In the above formula, N represents the number of line array images, y It is the frame width perpendicular to the scanning direction and is a constant.

[0073] S2. Use the SAM model for auxiliary annotation to draw a leaf mask image of the phenotypic image.

[0074] S3, the blade mask is enhanced and a training set and a test set are obtained. Since the blade is not scanned at a completely vertical angle in actual working conditions, that is, there is a small angle between the blade centerline and the displacement direction of the scanning unit. Data enhancement processing is performed on this basis. This step includes the following steps:

[0075] S301, set small angle flip operator Specifically, the calculation formula of the flip operator is:

[0076] ;

[0077] In the above formula, Indicates the flip angle, which is a random value within ±30°.

[0078] When rotated 30° clockwise: ;

[0079] When rotated 30° counterclockwise: .

[0080] Taking the mask image as an example, the coordinates of a pixel point in the original mask are , the rotated coordinates for ,but:

[0081] ;

[0082] .

[0083] The above process image operations are completed using the opencv library.

[0084] S302, the mask parts of the leaf mask image and the phenotype image are flipped multiple times at a small angle to obtain a flipped mask image and a flipped phenotype image, and the pixel coordinates of the flipped mask image and the flipped phenotype image are calculated according to the flip operator. In this example, the enhancement factor is set to 3, so each leaf phenotype image is flipped 3 times at a small angle, and a total of 300 leaf phenotype images are obtained.

[0085] S303, dividing the multiplied flip mask image and flip phenotype image into a training set and a test set according to a ratio of 7:3, that is, 210 images are used for training and the remaining images are used for testing.

[0086] S4. Use the training set to train the Segformer neural network model. The training uses cross entropy loss as the loss function and the AdamW algorithm to obtain the following model parameters: mIoU is 0.9549, mAc: 0.9774mFscore is 0.9767, and mRecall is 0.9774.

[0087] S5. Use the trained Segformer neural network model to test the test set and obtain the segmentation mask of the leaf, that is, the binary image.

[0088] S6. Use Python program to analyze the number of pixels in the binary image.

[0089] S7, calculating the leaf area according to the number of pixels and the preset pixel area. Specifically, the method for calculating the preset pixel area includes the following steps:

[0090] S701 . Set a scanning parameter of a scanning unit and an area of ​​a reference marker, where the scanning parameter is a DPI of the scanning unit.

[0091] S702: Acquire a reference marker image 8 containing a preset background.

[0092] S703 , marking the mask of the reference calibrator in the reference calibrator image 8 , and calculating the number of pixels thereof according to the mask.

[0093] S704: Calculate the preset pixel area according to the number of pixels and the pixel area calculation formula. The calculation formula is: ,

[0094] In the above formula, is the area of ​​the reference marker image, is the number of pixels in the reference marker image.

[0095] The calculation formula for leaf area is: , is the number of pixels in the binary image.

[0096] The above method can be deployed in terminal devices such as microcontrollers, cloud servers, smart phones, and computers. Example

[0097] like Figure 4 As shown, the present invention also provides a system for obtaining leaf phenotypes of living crops, comprising:

[0098] An acquisition module, used for acquiring phenotypic images of leaves of living crops;

[0099] A drawing module, used for drawing a leaf mask image of the phenotypic image;

[0100] A processing module, used for enhancing the leaf mask and obtaining a training set and a test set;

[0101] A training module, used for training a Segformer neural network model using the training set;

[0102] A testing module, used to test the test set using a trained Segformer neural network model to measure the segmentation mask of the leaf;

[0103] A determination module, used to determine the number of pixels of the segmentation mask;

[0104] A calculation module is used to calculate the leaf area according to the number of pixel points and a preset pixel area. Example

[0105] like Figures 5 to 12 As shown, the present invention also provides a device for acquiring leaf phenotypes of living crops, comprising a substrate 1, an optical axis 2 and a scanning body 3.

[0106] The substrate 1 is a rectangular plate, length×width=40cm×30cm, with a flat surface and a solid color, so that the blades can be distinguished and used as a background plate. Preferably, the scanning area of ​​the substrate is white, and the scanning area is provided with a reference calibrator image 8, preferably, the reference calibrator image 8 is provided in the lower right corner of the scanning area.

[0107] Optical axis seats 4 are fixedly mounted on the four corners of the substrate 1 . Therefore, two optical axis seats 4 opposite to each other in the length direction are fixedly mounted with an optical axis 2 of model M8, so that the optical axis 2 is respectively fixed on the left and right sides of the substrate 1 .

[0108] The scanning body 3 is mounted on the optical axis 2 via a linear bearing 25 so as to be able to pass through the scanning area along a straight line to complete the scanning of the blade and the reference marker image 8 .

[0109] The scanning body 3 includes a scanning unit 19, an interface unit 16, a speed measuring unit, a power supply unit 14, a main control unit 15, a display unit 12 and a key unit 13. Obviously, the scanning body 3 also includes a shell to accommodate and fix the aforementioned units. The shell includes a lower shell 11 and an upper shell cover 10, and the two are connected by M3 screws. The lower shell 11 and the upper shell are both provided with structures for installing components, such as grooves or cavities, etc. This is a conventional means, so it is not repeated. The scanning unit 19, the speed measuring unit, the interface unit 16, the power supply unit 14, the display unit 12 and the key unit 13 are electrically connected to the main control unit 15 respectively.

[0110] The display unit 12 and the key unit 13 are fixed on the upper shell cover 10. The display unit 12 is a 1.3-inch ST7789P3 color LCD module. The key unit 13 is a 3PIN jog key, with a total of 5 keys, which respectively realize the functions of power on / off, confirmation, cancellation, and selection. The key unit 13 cooperates with the display unit 12 to realize the input of commands and the viewing of data.

[0111] The scanning unit 19, the interface unit 16, the speed measuring unit, the power supply unit 14 and the main control unit 15 are fixed in the lower housing 11. The scanning unit 19 is located at the bottom of the lower housing 11, so as to obtain a continuous linear array image of the object being measured when scanning. The scanning unit 19 is a CIS image sensor with a resolution of 100 dpi-600 dpi and a maximum scanning image size of A4. The interface unit 16 is a USB interface module or a Type-C interface module, which is arranged on the front / rear side wall of the lower housing 11, and is used for data transmission between the scanning body 3 and other terminal devices and charging with an external power supply.

[0112] The speed measuring unit includes a speed measuring wheel 7 and an encoder 18. The speed measuring wheel 7 is connected to the encoder 18. The rim of the speed measuring wheel 7 extends out of the bottom of the lower housing 11 and cooperates with the substrate 1 or a component on the substrate 1 to obtain the scanning speed. That is to say, when the scanning body 3 is manually driven to work, it is difficult to scan at a uniform speed, so it is necessary to calculate the actual displacement of the scanning unit 19 according to a formula to achieve the splicing of the linear array images.

[0113] In order to prevent the speed measuring wheel 7 from sliding when rolling and affecting the speed measuring result, the speed measuring wheel 7 is selected to be a gear, and a rack 5 is set on the surface of the substrate 1. The engagement of the two realizes the non-slip rolling of the speed measuring wheel 7, thereby improving the accuracy of the speed measuring result.

[0114] The power supply unit 14 is a lithium battery with a capacity of 3000 mAh and can be charged through the interface unit 16 .

[0115] The main control unit 15 is a single chip microcomputer module, which includes a processing module and a storage module. The processing module uses an FPGA chip, and the storage module stores a computer program. When the computer program is executed by the processing module, the method of the first embodiment is implemented.

[0116] As a preferred example, the scanning body 3 can also scan automatically, and the scanning speed is uniform, which further improves the accuracy of the calculation results. The scanning body 3 also includes a driving unit and a sensing unit, and the driving unit and the sensing unit are electrically connected to the main control unit 15 respectively. The driving unit includes a motor 17 and a gear mechanism, and the motor 17 is connected to the speed measuring wheel 7 through the gear mechanism, so that the speed measuring wheel 7 rolls along the rack 5 to realize the movement of the scanning body 3. As a preferred example, two speed measuring wheels 7 are provided, and they are symmetrically arranged on both sides of the lower shell 11. One of the speed measuring wheels 7 is connected to the encoder 18 through a gear mechanism, and one of the speed measuring wheels 7 is connected to the motor 17 through a gear mechanism, and the two gear mechanisms are connected through a transmission shaft. In other words, the gear mechanism includes a first gear 22, a second gear 23 and a third gear 24. The first gear 22 is installed on the main shaft of the motor 17 and the main shaft of the encoder 18, the second gear 23 is installed on the rotating shaft of the speed measuring wheel 7, and the third gear 24 is installed on the transmission shaft, and the first gear 22 is meshed with the second gear 23 and the third gear 24 respectively.

[0117] The sensing unit includes a front sensing switch 20 and a rear sensing switch 21. The front sensing switch 20 is arranged on the front side wall of the lower housing 11, and the rear sensing switch 21 is arranged on the rear side wall of the lower housing 11. When the scanning body 3 moves to the limit position, the sensing switch will be triggered by the optical axis seat 4 to generate an electrical signal, thereby controlling the scanning body 3 to stop moving. In order to prevent the situation where one of the sensing switches fails and cannot stop, two front sensing switches 20 and two rear sensing switches 21 are provided, which are used to be triggered by the optical axis seats 4 on the left and right sides respectively. In other words, the scanning body 3 can be stopped after any sensing switch is triggered.

[0118] As a preferred example, the device also includes a transparent cover plate 9, which is arranged on the surface of the substrate 1. The function of the transparent cover plate 9 is to press the object to be measured, so as to prevent the object to be measured from moving during scanning and affecting the scanning results. On the other hand, the transparent cover plate 9 can also play a role in flattening the object to be measured, so that the object to be measured is completely flattened. One short side of the transparent cover plate 9 is fixed to the substrate 1 by the pressing plate 6, and the other side is not fixed. Therefore, when placing the object to be measured, the unfixed side can be lifted up, and the transparent cover plate 9 can be released after the object to be measured is placed. At this time, the transparent cover plate 9 will press the object to be measured on the substrate 1 under the action of its own elasticity.

[0119] The application of the device for acquiring the phenotype of leaves of living crops in the third embodiment in acquiring the phenotype of leaves of living crops is as follows:

[0120] Before using this device, it is necessary to pre-set the scanning parameters and move the scanning body 3 to the initial position, that is, the end of the transparent cover 9 that is not fixed. If automatic scanning is selected, the scanning speed must also be set. After the settings are completed, lift up one side of the transparent cover 9, then put the leaves in, and finally put down the transparent cover 9 to press the leaves. Then start the scanning body 3 to complete the acquisition of the leaf phenotypic image. The leaf phenotypic image is then processed and calculated by the main control unit 15, and the results are finally output through the display unit 12. During the entire process, the leaves do not need to be separated from the body, which ensures the integrity and activity of the leaves and avoids the influence of factors such as water loss and curling. The device is easy to carry and can complete the sampling and measurement of leaves outdoors. The output results can be exported to other terminal devices via a data cable for secondary viewing and storage.

[0121] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for obtaining leaf phenotypes of living crops, characterized in that: The steps include: S1, obtaining a phenotypic image of a leaf of a living crop; the obtaining of a phenotypic image of a leaf of a living crop comprises the following steps: S101, calculating the displacement of the scanning unit according to the moving speed of the scanning unit; the calculation formula of the displacement of the scanning unit is: ； In the above formula, Indicates the moving speed of the scanning unit, Indicates time, Indicates scanning the i-th line array image The displacement of the scanning unit after S102, calculating the splicing position of each line array image according to the displacement of the scanning unit; the calculation formula of the position of each line array image after splicing is: ； S103, stitching all the line array images into a complete phenotype image according to the stitching position of each line array image; the stitching formula of the complete phenotype image is: ; In the above formula, N represents the number of linear images, and y is the frame width perpendicular to the scanning direction, which is a constant; S2, drawing a leaf mask image of the phenotypic image; S3, performing enhancement processing on the leaf mask to obtain a training set and a test set; S4, training a Segformer neural network model using the training set; S5. Testing the test set using the trained Segformer neural network model to obtain a segmentation mask of the leaf; S6, determining the number of pixels of the segmentation mask; S7. Calculate the leaf area according to the number of pixel points and a preset pixel area.

2. The method for obtaining leaf phenotypes of living crops according to claim 1, characterized in that: The step of enhancing the leaf mask and obtaining a training set and a test set comprises the following steps: S301, setting a small-angle flip operator; S302, performing multiple flipping of the mask parts of the leaf mask image and the phenotype image at small angles to obtain a flipped mask image and a flipped phenotype image, and calculating pixel coordinates of the flipped mask image and the flipped phenotype image according to the flipping operator; S303 , dividing the flipped mask image and the flipped phenotype image into a training set and a test set respectively according to the ratio.

3. The method for obtaining leaf phenotypes of living crops according to claim 2, characterized in that: The calculation formula of the flip operator is: ； In the above formula, Indicates the flip angle, which is a random value within ±30°.

4. The method for obtaining leaf phenotypes of living crops according to claim 1, characterized in that: The method for calculating the preset pixel area comprises the following steps: S701, setting scanning parameters and the area of ​​the reference calibrator; S702, obtaining a reference marker image containing a preset background (8); S703, marking the mask of the reference calibrator in the reference calibrator image (8), and calculating the number of pixels thereof according to the mask; S704: Calculate the preset pixel area according to the number of pixels and a pixel area calculation formula.

5. A system based on the method for obtaining leaf phenotypes of living crops as claimed in claim 1, characterized in that: include: An acquisition module, used for acquiring phenotypic images of leaves of living crops; A drawing module, used for drawing a leaf mask image of the phenotypic image; A processing module, used for enhancing the leaf mask and obtaining a training set and a test set; A training module, used for training a Segformer neural network model using the training set; A testing module, used to test the test set using a trained Segformer neural network model to measure the segmentation mask of the leaf; A determination module, used to determine the number of pixels of the segmentation mask; A calculation module is used to calculate the leaf area according to the number of pixel points and a preset pixel area.

6. A device for acquiring leaf phenotypes of living crops, characterized in that: include: A main control unit (15), the main control unit (15) comprising a processing module and a storage module, the storage module storing a computer program, and when the computer program is executed by the processing module, the steps of the method for obtaining leaf phenotypes of living crops according to claim 1 are implemented.

Images