Multimodal Image-Based Chlorophyll Fluorescence Data Analysis Method, Device, and Equipment

Abstract

The present application provides a method, apparatus, and device for analyzing chlorophyll fluorescence data based on multimodal images. For the multimodal images of the chlorophyll fluorescence data of plants, the height type of the plants is determined by height analysis for the target modal images among them. In this way, the most suitable segmentation method can be determined according to the height type, making the foreground mask obtained by segmentation more accurate. Moreover, clustering denoising processing is performed on the foreground mask to avoid the interference of some noises in the foreground mask and optimize the foreground mask, so that the obtained optimized foreground mask more conforms to the foreground contour characteristics of the plants. Then, according to this optimized foreground mask, other modal images are segmented to obtain a more accurate foreground image of the plant. Thus, the photosynthetic parameters determined according to this foreground image will also be more accurate and can adapt to various scenarios where the plants are located.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and particularly to a method, device and equipment for analyzing chlorophyll fluorescence data based on multi-modal images. Background Art

[0002] The analysis of chlorophyll fluorescence data is of great significance for evaluating the health status of plants. Traditional analysis methods mainly rely on basic chlorophyll fluorescence data for foreground extraction, but in complex scenarios, such as the coexistence of plants and weeds, large differences in plant height, etc., it is difficult to accurately extract the foreground automatically. Summary of the Invention

[0003] In view of this, the purpose of this application is to propose a method, device and equipment for analyzing chlorophyll fluorescence data based on multi-modal images to solve or partially solve the above technical problems.

[0004] Based on the above purpose, this application proposes a method for analyzing chlorophyll fluorescence data based on multi-modal images, including:

[0005] Obtain multi-modal images of plant chlorophyll fluorescence data, and determine the target modal image in the multi-modal images;

[0006] Determine the depth histogram of the target modal image, and determine the height type of the plant according to the depth histogram;

[0007] Determine the segmentation method corresponding to the height type, and segment the target modal image according to the segmentation method to obtain the foreground mask of the plant;

[0008] Perform clustering denoising processing on the foreground mask to obtain an optimized foreground mask;

[0009] Use the optimized foreground mask to segment at least one image measured based on photosynthesis in the multi-modal image to obtain the foreground image corresponding to the plant, and determine the photosynthetic parameters of the plant according to the foreground image.

[0010] Based on the same concept, this application also proposes a device for analyzing chlorophyll fluorescence data based on multi-modal images, including:

[0011] An image acquisition module, configured to obtain multi-modal images of plant chlorophyll fluorescence data, and determine the target modal image in the multi-modal images;

[0012] A plant height determination module, configured to determine the depth histogram of the target modal image, and determine the height type of the plant according to the depth histogram;

[0013] A segmentation module, configured to determine a segmentation method corresponding to the height type, and segment the target modal image according to the segmentation method to obtain a foreground mask of the plant;

[0014] A clustering optimization module, configured to perform clustering denoising processing on the foreground mask to obtain an optimized foreground mask;

[0015] A photosynthetic parameter determination module, configured to use the optimized foreground mask to segment at least one image obtained by photosynthesis measurement in the multi-modal image to obtain a foreground image corresponding to the plant, and determine the photosynthetic parameters of the plant according to the foreground image.

[0016] Based on the same concept, the present application also proposes an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the above-mentioned method is implemented.

[0017] As can be seen from the above, the method, device, and equipment for chlorophyll fluorescence data analysis based on multi-modal images provided by the present application will, for the multi-modal image of the chlorophyll fluorescence data of the plant, determine the height type of the plant through height analysis of the target modal image therein, so that the most suitable segmentation method can be determined according to the height type, making the segmented foreground mask more accurate. And it will also perform clustering denoising processing on the foreground mask to avoid the interference of some noises in the foreground mask, optimize the foreground mask, making the obtained optimized foreground mask more in line with the foreground contour characteristics of the plant. Then, segment and process other modal images according to the optimized foreground mask to obtain a more accurate foreground image of the plant. In this way, the photosynthetic parameters determined according to the foreground image will also be more accurate and can adapt to various scenarios where the plant is located. Description of the Drawings

[0018] In order to more clearly illustrate the technical solutions in the present application or related technologies, the following will briefly introduce the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained according to these drawings without creative efforts.

[0019] Figure 1 It is a flowchart of the method for chlorophyll fluorescence data analysis based on multi-modal images according to the embodiment of the present application;

[0020] Figure 2 It is a graphical schematic diagram of the method for chlorophyll fluorescence data analysis based on multi-modal images according to the embodiment of the present application;

[0021] Figure 3 For Figure 2Depth distribution diagram corresponding to tall plants;

[0022] Figure 4 For Figure 2 Depth distribution diagram corresponding to short plants;

[0023] Figure 5 Schematic diagram of segmentation by the recursive multi-threshold segmentation method of this application embodiment;

[0024] Figure 6 Schematic diagram of segmentation by the depth weight allocation segmentation method of this application embodiment;

[0025] Figure 7 For Figure 6 Schematic diagram of the curve of the normalized depth value and the corresponding depth weight value under different gain parameters k;

[0026] Figure 8 Schematic diagram of clustering processing according to contour area in this application embodiment;

[0027] Figure 9 Schematic diagram of comparison between the foreground mask obtained by the recursive multi-threshold segmentation method and the mask obtained by the traditional method in this application embodiment;

[0028] Figure 10 For Figure 9 Pixel distribution diagram corresponding to Fv / Fm;

[0029] Figure 11 For Figure 10 Distribution density diagram of Fv / Fm corresponding to;

[0030] Figure 12 For Figure 9 Pixel distribution diagram corresponding to NDVI;

[0031] Figure 13 For Figure 12 Distribution density diagram of NDVI corresponding to;

[0032] Figure 14 Schematic diagram of comparison between the foreground mask obtained by the depth weight allocation segmentation method and the mask obtained by the traditional method in this application embodiment;

[0033] Figure 15 For Figure 14 Pixel distribution diagram corresponding to Fv / Fm;

[0034] Figure 16 For Figure 15 Distribution density diagram of Fv / Fm corresponding to;

[0035] Figure 17 For Figure 14Pixel distribution map corresponding to NDVI;

[0036] Figure 18 For Figure 17 Schematic diagram of the distribution density of the corresponding NDVI;

[0037] Figure 19 Chlorophyll fluorescence images recorded at different times in the embodiments of the present application;

[0038] Figure 20 Structural block diagram of the chlorophyll fluorescence data analysis device based on multi-modal images in the embodiments of the present application;

[0039] Figure 21 Structural schematic diagram of the electronic device in the embodiments of the present application. Detailed implementation manners

[0040] It can be understood that the data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the corresponding laws, regulations and related provisions.

[0041] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and then implement the present application, rather than limiting the scope of the present application in any way. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to be able to convey the scope of the present application completely to those skilled in the art.

[0042] In this article, it should be understood that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction and does not have any limiting meaning.

[0043] Glossary:

[0044] INF: Device INFormation File, a file used to describe data information such as devices or files under the Windows operating system.

[0045] DAT: Data, a data file with the extension ".dat".

[0046] OJIP: O-J-I-P Fluorescence Induction Curve, a rapid chlorophyll fluorescence induction kinetic curve, which is one of the most classic fluorescence kinetic curves.

[0047] PAM: Pulse Amplitude Modulated Fluorescence, a technique for fluorescence measurement by pulsed modulated light.

[0048] GFP: Green Fluorescent Protein, green fluorescent protein.

[0049] RFP: Red Fluorescent Protein, a fluorescent protein that can be used for biological labeling and real-time imaging.

[0050] Otsu: Maximum inter-class variance method, proposed by Japanese scholar Nobuyuki Otsu in 1979, is a method for automatically obtaining thresholds that is self-suitable for bimodal situations, also known as the Otsu method, abbreviated as Otsu.

[0051] ChlF: Chlorophyll fluorescence imaging.

[0052] Nir: Near Infrared, near-infrared imaging.

[0053] The embodiments of the present application provide a method for analyzing chlorophyll fluorescence data based on multi-modal images, as Figure 1 shown, the method includes:

[0054] Step 101, obtain a multi-modal image of the chlorophyll fluorescence data of the plant, and determine the target modal image in the multi-modal image.

[0055] Specifically, the multi-modal image refers to multiple types of images obtained when different imaging technologies and sensors are used for the same scene or object. It can provide complementary information from different angles and can more comprehensively understand and analyze the target object. The multi-modal image includes chlorophyll fluorescence images (such as Figure 19 shown), near-infrared grayscale images, depth maps, etc.

[0056] The chlorophyll fluorescence image can reflect the physiological state and photosynthesis efficiency of the plant by detecting the fluorescence characteristics of plant chlorophyll; the near-infrared grayscale image uses the characteristics of near-infrared light and can provide information different from visible light images, such as the water content or tissue structure of the plant; the depth map provides the spatial structure information of the scene, can reflect the three-dimensional shape and distance information of the object, and has strong robustness to light changes.

[0057] The chlorophyll fluorescence data file includes INF file types for storing metadata information such as measurement frequency and DAT file types for storing data of different measurement programs, including one or more of file formats for various measurement types such as photosynthesis measurement (such as OJIP and PAM methods), chlorophyll fluorescence, reflection spectrum (near-infrared, red edge), fluorescent protein (GFP and RFP), and visible light images.

[0058] Step 102: Determine the depth histogram of the target modal image, and determine the height type of the plant according to the depth histogram.

[0059] In specific implementation, use a plant height classifier based on the depth histogram to process the depth histogram, thereby accurately estimating the height of the plant and determining the height type of the plant. The height type at least includes: tall plants and short plants.

[0060] Step 103: Determine the segmentation method corresponding to the height type, and segment the target modal image according to the segmentation method to obtain the foreground mask of the plant.

[0061] In specific implementation, different height types correspond to different segmentation methods. In this way, determining the most suitable segmentation according to the height type in a targeted manner makes the obtained foreground mask more in line with the image characteristics of the plant and more accurate.

[0062] Step 104: Perform clustering denoising processing on the foreground mask to obtain an optimized foreground mask.

[0063] In specific implementation, since the obtained foreground mask will have noise, which affects its accuracy, it is necessary to optimize it through clustering denoising to make the optimized foreground mask more accurate.

[0064] Step 105: Use the optimized foreground mask to segment at least one frame of the image obtained based on photosynthesis measurement in the multi-modal image to obtain the foreground image corresponding to the plant, and determine the photosynthetic parameters of the plant according to the foreground image.

[0065] In specific implementation, the corresponding photosynthetic parameters include: maximum quantum yield ( ), actual photochemical quantum yield ( , that is, ), (non-photochemical quenching), (light absorption index), (electron transport rate), (chlorophyll index), (normalized difference vegetation index) and other photosynthetic parameters. For example,

[0066]

[0067]

[0068]

[0069] Among them, is the initial fluorescence image, is the maximum fluorescence image, is the steady-state fluorescence image. Is the maximum fluorescence image under light adaptation.

[0070]

[0071]

[0072]

[0073]

[0074] Among them, , , , respectively represent the reflectance images of the red light, near-infrared, far-infrared, and red-edge bands, Is the proportion of PSII. PSII (Photosystem II) is an important component in the conversion of light energy in photosynthesis. It is the second photochemical system in photosynthesis and is mainly responsible for the photolysis of water.

[0075] Through the above solution, for the multi-modal images of the chlorophyll fluorescence data of the plant, the height type of the plant is determined by analyzing the height of the target modal image among them. In this way, the most suitable segmentation method can be determined according to the height type, making the foreground mask obtained by segmentation more accurate. In addition, clustering and denoising processing will be performed on the foreground mask to avoid the interference of some noises in the foreground mask and optimize the foreground mask, making the obtained optimized foreground mask more in line with the foreground contour characteristics of the plant. Then, segmenting other modal images according to the optimized foreground mask to obtain a more accurate foreground image of the plant. In this way, the photosynthetic parameters determined according to this foreground image will also be more accurate and can adapt to various scenarios where the plant is located.

[0076] In some embodiments, step 102 includes:

[0077] Step 1021, extracting the depth information of each pixel from the target modal image.

[0078] Specifically, in implementation, through a monocular depth estimation model, the depth information of each pixel is extracted from the chlorophyll fluorescence image of the target modal image Among them , where Is the depth value corresponding to the pixel point at position In the image , generating a depth map .

[0079] Step 1022, generating a depth histogram according to the depth information.

[0080] Specifically, in implementation, the generated depth map Map to the depth value range Between them, a depth map is generated. The frequency of each depth value is counted to generate a normalized depth histogram (as Figure 2 shown), and its formula is:

[0081]

[0082] Among them, count is the number of occurrences of the depth value in the image, is the normalized depth histogram, and . is the number of occurrences of the depth value in the depth map .

[0083] Step 1023, determine the graphic features of the plant according to the depth histogram, compare the graphic features with the feature threshold, and determine the height type of the plant (as Figure 2 shown).

[0084] For plants of different heights, the features such as the foreground of the plant and ground noise in the near-infrared grayscale image are very different and cannot be applied to the same image segmentation method. Therefore, before applying image segmentation, it is necessary to use monocular depth estimation to classify the plant height type to determine the plant height type, which is convenient to match different segmentation methods according to the height type.

[0085] Through the above scheme, the graphic features of the plant determined by means of the depth histogram are more matched with the height of the plant, and thus the height type determined based on the graphic features is more accurate.

[0086] In some embodiments, step 1023 includes:

[0087] Step 10231, determine the rectangular measure R, skewness measure and kurtosis measure of the depth histogram.

[0088] Specifically, the rectangular measure reflects the degree of approximation of the shape of the depth histogram to an ideal rectangle. The formula is:

[0089]

[0090] Among them, is the area under the depth histogram graph, is the area of the rectangular region enclosed by the depth histogram. The closer the rectangular measure value R of the depth histogram is to 1, the closer its shape is to a rectangle.

[0091] Skewness measure Indicates the asymmetry of the depth histogram distribution, and the formula is:

[0092]

[0093] where is the mean of the depth histogram, is the standard deviation, is the total number of depth values in the depth histogram. The skewness measure is used to judge the tail direction and shape of the depth histogram.

[0094] Kurtosis measure Indicates the sharpness of the depth histogram distribution. Its calculation formula is:

[0095]

[0096] where is the kurtosis measure, indicating the "peak" degree of the depth histogram. A higher kurtosis measure usually corresponds to the histogram characteristics of tall plants, showing a concentrated sharp peak.

[0097] Step 10232, obtain the rectangle threshold , skewness threshold and kurtosis threshold .

[0098] Step 10233, in response to and and satisfying the first type of condition, determine that the plant is a tall plant (as shown in Figure 3 , the depth distribution diagram corresponding to the tall plant).

[0099] Alternatively, step 10234, in response to and and satisfying the second type of condition, determine that the plant is a short plant (as shown in Figure 4 , the depth distribution diagram corresponding to the short plant);

[0100] Alternatively, step 10235, in response to the rectangle measure R, skewness measure and kurtosis measure satisfying other conditions except the first type of condition and the second type of condition, discard the target modal image of the plant.

[0101] Through the above solution, according to the comparison between the rectangle measure R and , the comparison between the skewness measure and , and the comparison between the kurtosis measure and By comparing them, the height of the plants can be accurately determined, tall plants can be accurately distinguished from short plants, and then the corresponding segmentation methods can be accurately allocated.

[0102] In some embodiments, step 103 includes:

[0103] Step 1031, in response to determining that the height type is a tall plant, use a recursive multi-threshold segmentation method to segment the target modal image to obtain a foreground mask of the plant.

[0104] Alternatively, step 1032, in response to determining that the height type is a short plant, use a depth-weighted allocation segmentation method to segment the target modal image to obtain a foreground mask of the plant.

[0105] Through the above solution, for tall plants, due to the large difference in their image depth and small ground noise, it is easy to form a stratification of the depth signal intensity. Therefore, using a recursive multi-threshold segmentation method can accurately segment the target modal image, and then obtain an accurate foreground mask; for short plants, due to their small image depth difference and large ground noise, a depth-weighted allocation segmentation method is needed to reduce the ground noise of the target modal image, and then ensure that an accurate foreground mask can be obtained.

[0106] In some embodiments, step 1031 is as Figure 5 shown and includes:

[0107] Step 10311, determine the near-infrared grayscale image in the target modal image.

[0108] Specifically, the gray level range of this near-infrared grayscale image is , and the size is pixels.

[0109] Step 10312, use the maximum inter-class variance (Otsu) to determine the segmentation threshold of the pixels, and perform the segmentation process: use the segmentation threshold to perform foreground segmentation on the near-infrared grayscale image to obtain a preliminary foreground mask, perform a bitwise OR operation on the preliminary foreground mask and the previous foreground mask to obtain the current foreground mask, determine the current background region according to the current foreground mask, and obtain the number of pixels in the current background region.

[0110] Specifically, the determination process of the segmentation threshold is:

[0111] The inter-class variance The calculation formula is as follows:

[0112]

[0113] Wherein, and They are the weights of Class 1 and Class 2 respectively. is the gray value, defined as:

[0114] .

[0115] Among them, is the probability of pixels with gray level in the image, and are the average gray values of Class 1 and Class 2, and the calculation formula is:

[0116]

[0117] Finally, select the gray value corresponding to the maximum τ as the optimal segmentation threshold.

[0118] After performing a bitwise OR operation on the preliminary foreground mask and the previous foreground mask, the current foreground mask is obtained, and the background area is , where M is the near-infrared gray image. After obtaining the background area, the number of pixels in the background area will be counted .

[0119] Step 10313, in response to determining that the number of pixels is greater than or equal to the quantity threshold, re-determine the segmentation threshold, and iteratively execute the segmentation process until the number of pixels is less than the quantity threshold.

[0120] In specific implementation, if the number of pixels in the background area , where m is the number of horizontal pixels of the near-infrared gray image and n is the number of vertical pixels of the near-infrared gray image; it proves that the background area has not been segmented well and needs to be segmented again, then the segmentation threshold will be re-determined according to the above determination process of the segmentation threshold , and the background area will be repeatedly segmented according to this segmentation threshold until the condition is met.

[0121] Step 10314, in response to determining that the number of pixels is less than the quantity threshold, use the current foreground mask as the foreground mask of the plant (such as the final foreground mask shown in Figure 5 ).

[0122] Through the above solution, the recursive multi-threshold segmentation method is the Otsu segmentation method based on recursive multi-thresholds, which is used for the near-infrared grayscale images of tall plants. Due to the large depth difference in the near-infrared grayscale images, a large amount of ground noise in the chlorophyll fluorescence images is effectively avoided. However, the near-infrared signal intensities of the plant leaves at different depths form stratifications. Therefore, the segmentation of the entire near-infrared grayscale image is divided into multiple levels, and the optimal threshold for each level is automatically calculated. By performing refined segmentation on the background region in each level of recursion, automatic and multi-level image segmentation is achieved, and an accurate foreground mask is obtained.

[0123] Specifically, as Figure 9 shown, the results obtained by traditional single-modal mask processing for images of different modalities (e.g., ChlF chlorophyll fluorescence imaging, Fv / Fm maximum quantum yield imaging, Nir near-infrared imaging, Depth depth map), compared with the foreground mask results obtained by this application for tall plants (i.e., Figure 9 the mask and the pseudo-color map of photosynthetic parameters optimized based on recursive multi-threshold segmentation in Figure 10 and 11 ), combined with the schematic diagram of Fv / Fm corresponding to Figure 12 and 13 the schematic diagram of NDVI corresponding to

[0124] In some embodiments, step 1032 is as Figure 6 shown and includes:

[0125] Step 10321, determining the chlorophyll fluorescence image in the target modal image.

[0126] Step 10322, performing monocular depth estimation on the chlorophyll fluorescence image to obtain a depth map.

[0127] Specifically in implementation, a monocular depth estimation model is used to generate a depth map for the chlorophyll fluorescence image, and this depth map can describe the depth information of each region in the image.

[0128] Step 10323, performing normalization processing on the depth map to obtain a normalized depth map.

[0129] Specifically in implementation, the normalization formula is as follows:

[0130]

[0131] where is the depth value in the normalized depth map, is the depth value in the depth map before normalization, is the minimum depth value in the depth map, is the maximum depth value in the depth map.

[0132] Step 10324: Determine the depth weight value of each pixel point according to the normalized depth map.

[0133] During specific implementation, as Figure 7 shown, the determined normalized depth map is (i.e., the normalized depth value corresponding to the abscissa in Figure 7 ), i is the position sorting of the pixel points, and the calculation formula for the depth weight value of each pixel point (i.e., the ordinate weight corresponding to Figure 7 in) is:

[0134]

[0135] where k is a gain parameter, and this gain parameter is used to control the steepness of the curve; is a set constant, usually set to 0.5 (or the median of the normalized depth value).

[0136] Step 10325: Determine the near-infrared grayscale image in the target modal image, and use the depth weight value of each pixel point to transform the near-infrared grayscale image to obtain an adjusted near-infrared grayscale image.

[0137] Step 10326: Perform segmentation processing on the adjusted near-infrared grayscale image to obtain a foreground mask of the plant.

[0138] During specific implementation, the calculated weight coefficient is applied to the near-infrared grayscale image (NIR) to adjust the grayscale value of the near-infrared grayscale image, enhance its foreground area, and weaken the influence of the background area. The adjusted near-infrared grayscale image The calculation formula of is:

[0139]

[0140] where is the grayscale value of the near-infrared grayscale image, is the depth weight value calculated from the depth map.

[0141] Through the above solution, the foreground and background of the plant can be accurately separated, and then the ground noise can be effectively removed, making the foreground mask obtained after segmentation processing more accurate. Specifically, as Figure 14 shown are the results obtained by traditional single-modal mask processing for images of different modalities (for example, ChlF chlorophyll fluorescence imaging, Fv / Fm maximum quantum yield imaging, Nir near-infrared imaging, Depth depth map), and the foreground mask results obtained by this application for short plants (i.e., Figure 14 the mask and the photosynthetic parameter false color map part optimized by depth weight allocation segmentation in), combined with Figure 15and 16 Schematic diagram of the corresponding Fv / Fm, and Figure 17 and 18 Schematic diagram of the corresponding NDVI. Obviously, the solution of this application has better effects.

[0142] In some embodiments, step 104 is as Figure 8 shown and includes:

[0143] Step 1041: Extract at least one contour from the foreground mask.

[0144] Determine the foreground mask Use an image processing library such as OpenCV. Through a contour detection algorithm (such as cv2.findContours), extract contours from the foreground mask to obtain a contour set , where each represents a contour, where i ∈ n and n is the number of contours. Step 1042: Calculate the area of each contour and perform clustering processing on the areas of at least one contour to obtain at least one clustering cluster.

[0145] In specific implementation, for each contour calculate its area : . Obtain a set of contour area sets: .

[0146] Then use the K-Means algorithm to perform clustering processing on the contour area set, clustering it into k clustering clusters, and each clustering cluster contains contours with similar areas.

[0147] Step 1043: Calculate the mean of each clustering cluster and determine the optimization threshold according to the mean of each clustering cluster.

[0148] In specific implementation, for each clustering cluster, calculate the mean of all contour areas within the clustering cluster (i.e., the clustering center). Let the center of the th clustering cluster be , that is, the mean of each clustering cluster:

[0149] ;

[0150] where represents the th clustering cluster, is the number of contours in the clustering cluster .

[0151] For the obtained k means , in order to determine the optimization threshold T of the area, first calculate the median of all clustering means.

[0152] If , the optimization threshold can be directly taken as and 's average value:

[0153] .

[0154] If , then remove the maximum mean value , and calculate the median of the remaining mean values as the optimization threshold T:

[0155] .

[0156] Step 1044: Compare the area of each contour with the optimization threshold respectively, and remove the contours smaller than the optimization threshold.

[0157] For each contour , if its area is smaller than the optimization threshold , then remove this contour:

[0158]

[0159] Step 1045: Combine the remaining contours to form an optimized foreground mask.

[0160] In specific implementation, for each remaining contour that is retained , draw it on the mask image to obtain the optimized foreground mask .

[0161] It should be noted that the method of the embodiment of the present application can be executed by a single device, such as a computer or a server, etc. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In this case of a distributed scenario, one of the multiple devices can only execute one or more steps of the method of the embodiment of the present application, and these multiple devices will interact with each other to complete the described method.

[0162] It should be noted that some embodiments of the present application have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be executed in a different order from those in the above embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0163] Based on the same concept, corresponding to the method for analyzing chlorophyll fluorescence data based on multi-modal images in any of the above embodiments, the present application further provides an apparatus for analyzing chlorophyll fluorescence data based on multi-modal images.

[0164] Reference Figure 20 , the apparatus includes:

[0165] An image acquisition module 201, configured to acquire a multi-modal image of the chlorophyll fluorescence data of a plant and determine a target modal image in the multi-modal image;

[0166] A plant height determination module 202, configured to determine a depth histogram of the target modal image and determine the height type of the plant according to the depth histogram;

[0167] A segmentation module 203, configured to determine a segmentation method corresponding to the height type and segment the target modal image according to the segmentation method to obtain a foreground mask of the plant;

[0168] A clustering optimization module 204, configured to perform clustering denoising processing on the foreground mask to obtain an optimized foreground mask;

[0169] A photosynthetic parameter determination module 205, configured to use the optimized foreground mask to segment at least one image measured based on photosynthesis in the multi-modal image to obtain a foreground image corresponding to the plant, and determine the photosynthetic parameters of the plant according to the foreground image.

[0170] In some embodiments, the plant height determination module 202 is specifically configured to:

[0171] Extract the depth information of each pixel from the target modal image;

[0172] Generate a depth histogram according to the depth information;

[0173] Determine the graphic features of the plant according to the depth histogram, compare the graphic features with the feature threshold, and determine the height type of the plant.

[0174] In some embodiments, the plant height determination module 202 is specifically configured to:

[0175] Determine the rectangular metric R, skewness metric and kurtosis metric ;

[0176] Obtain the rectangular threshold , skewness threshold and kurtosis threshold ;

[0177] In response to And And If the first type of condition is satisfied, determine that the plant is a tall plant; or,

[0178] In response to And And If the second type of condition is satisfied, determine that the plant is a short plant; or,

[0179] In response to the rectangular metric R, the skewness metric And the kurtosis metric If other conditions except the first type of condition and the second type of condition are satisfied, discard the target modal image of the plant.

[0180] In some embodiments, the segmentation module 203 is specifically configured to:

[0181] In response to determining that the height type is a tall plant, perform segmentation processing on the target modal image by using a recursive multi-threshold segmentation method to obtain a foreground mask of the plant; or,

[0182] In response to determining that the height type is a short plant, perform segmentation processing on the target modal image by using a depth weight assignment segmentation method to obtain a foreground mask of the plant.

[0183] In some embodiments, the segmentation module 203 is specifically configured to:

[0184] Determine the near-infrared grayscale image in the target modal image;

[0185] Use the maximum inter-class variance to determine the segmentation threshold of the pixels, and perform the segmentation process: perform foreground segmentation on the near-infrared grayscale image by using the segmentation threshold to obtain a preliminary foreground mask, perform a bitwise OR operation on the preliminary foreground mask and the previous foreground mask to obtain the current foreground mask, determine the current background region according to the current foreground mask, and obtain the number of pixels in the current background region;

[0186] In response to determining that the number of pixels is greater than or equal to the number threshold, re-determine the segmentation threshold, and iteratively execute the segmentation process until the number of pixels is less than the number threshold;

[0187] In response to determining that the number of pixels is less than the number threshold, use the current foreground mask as the foreground mask of the plant.

[0188] In some embodiments, the segmentation module 203 is specifically configured to:

[0189] Determine the chlorophyll fluorescence image in the target modal image;

[0190] Perform monocular depth estimation on the chlorophyll fluorescence image to obtain a depth map;

[0191] Normalize the depth map to obtain a normalized depth map;

[0192] Determine the depth weight value of each pixel point according to the normalized depth map;

[0193] Determine the near-infrared grayscale image in the target modality image, and use the depth weight value of each pixel point to transform the near-infrared grayscale image to obtain an adjusted near-infrared grayscale image;

[0194] Perform segmentation processing on the adjusted near-infrared grayscale image to obtain a foreground mask of the plant.

[0195] In some embodiments, the clustering optimization module 204 is specifically configured to:

[0196] Extract at least one contour from the foreground mask;

[0197] Calculate the area of each contour, and perform clustering processing on the areas of at least one contour to obtain at least one clustering cluster;

[0198] Calculate the mean of each clustering cluster, and determine an optimization threshold according to the mean of each clustering cluster;

[0199] Compare the area of each contour with the optimization threshold respectively, and remove the contours smaller than the optimization threshold;

[0200] Combine the remaining contours to form an optimized foreground mask.

[0201] In some embodiments, the multi-modal image refers to multiple types of images obtained when different imaging technologies and sensors are used for the same scene or object.

[0202] For the convenience of description, when describing the above device, various modules are described separately according to their functions. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0203] The device in the above embodiment is used to implement the corresponding method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0204] Based on the same concept, corresponding to the method in any of the above embodiments, the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the method in any of the above embodiments when executing the computer program.

[0205] Figure 21FIG. 0 shows a more specific schematic diagram of the hardware structure of the electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. Among them, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other inside the device through the bus 1050.

[0206] The processor 1010 may be implemented by means of a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0207] The memory 1020 may be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0208] The input / output interface 1030 is used to connect to an input / output module to implement information input and output. The input / output module may be configured as a component in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Among them, the input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0209] The communication interface 1040 is used to connect to a communication module (not shown in the figure) to implement communication interaction between this device and other devices. Among them, the communication module may implement communication in a wired manner (such as USB, network cable, etc.) or in a wireless manner (such as mobile network, WIFI, Bluetooth, etc.).

[0210] The bus 1050 includes a path for transmitting information between various components of the device (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0211] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device may also only include the components necessary to implement the solution of the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0212] The electronic device of the above embodiment is used to implement the corresponding method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0213] Based on the same concept, corresponding to the method of any of the above embodiments, the present application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the method described in any of the foregoing embodiments.

[0214] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0215] The computer instructions stored in the storage medium of the above embodiment are used to cause the computer to execute the method described in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0216] Based on the same concept, corresponding to the method of any of the above embodiments, the present application also provides a computer program product, including computer program instructions, which when running on a computer, cause the computer to execute the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0217] Those of ordinary skill in the art should understand that: the discussion of any of the above embodiments is only exemplary, and is not intended to imply that the scope of the present application (including the claims) is limited to these examples; under the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of brevity.

[0218] In addition, for the sake of simplicity of description and discussion, and in order not to make the embodiments of the present application difficult to understand, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. In addition, the device may be shown in block diagram form to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present application are to be implemented (i.e., these details should be fully within the understanding of those skilled in the art). In the case where specific details (such as circuits) are set forth to describe the exemplary embodiments of the present application, it will be apparent to those skilled in the art that the embodiments of the present application can be implemented without these specific details or with variations of these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0219] Although the present application has been described in connection with specific embodiments of the present application, many alternatives, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other memory architectures (such as DRAM) can be used in the embodiments discussed.

[0220] The embodiments of the present application are intended to cover all such alternatives, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omission, modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present application shall be included in the protection scope of the present application.

Claims

1. A method for analyzing chlorophyll fluorescence data based on multimodal images, characterized in that, Including: Obtain a multimodal image of plant chlorophyll fluorescence data, and determine the target modal image in the multimodal image; Determine the depth histogram of the target modal image, and determine the height type of the plant according to the depth histogram; Determine the segmentation method corresponding to the height type, and segment the target modal image according to the segmentation method to obtain the foreground mask of the plant; Perform clustering denoising processing on the foreground mask to obtain an optimized foreground mask; Use the optimized foreground mask to segment at least one image measured based on photosynthesis in the multimodal image to obtain the foreground image corresponding to the plant, and determine the photosynthetic parameters of the plant according to the foreground image; The determining the segmentation method corresponding to the height type, and segmenting the target modal image according to the segmentation method to obtain the foreground mask of the plant includes: In response to determining that the height type is a tall plant, use a recursive multi-threshold segmentation method to segment the target modal image to obtain the foreground mask of the plant; In response to determining that the height type is a short plant, use a depth-weighted allocation segmentation method to segment the target modal image to obtain the foreground mask of the plant; The using a depth-weighted allocation segmentation method to segment the target modal image to obtain the foreground mask of the plant includes: Determine the chlorophyll fluorescence image in the target modal image; Perform monocular depth estimation on the chlorophyll fluorescence image to obtain a depth map; Perform normalization processing on the depth map to obtain a normalized depth map; Determine the depth weight value of each pixel point according to the normalized depth map; Determine the near-infrared grayscale image in the target modal image, and use the depth weight value of each pixel point to transform the near-infrared grayscale image to obtain an adjusted near-infrared grayscale image; Perform segmentation processing on the adjusted near-infrared grayscale image to obtain the foreground mask of the plant.

2. The method according to claim 1, wherein The determining the depth histogram of the target modal image, and determining the height type of the plant according to the depth histogram includes: Extract the depth information of each pixel from the target modal image; Generate a depth histogram according to the depth information; Determine the graphical features of the plant according to the depth histogram, compare the graphical features with the feature threshold, and determine the height type of the plant.

3. The method according to claim 2, wherein The determining the graphical features of the plant according to the depth histogram, comparing the graphical features with the feature threshold, and determining the height type of the plant includes: Determine the rectangle measure R, skewness measure and kurtosis measure ; Obtain rectangle threshold , skewness threshold and kurtosis threshold ; In response to and and satisfying a first type of condition, determining that the plant is a tall plant; or, In response to and and if a second type of condition is satisfied, determine that the plant is a dwarf plant; or In response to the rectangularity measure R, skewness measure and kurtosis measure satisfying other conditions in addition to the first type of conditions and the second type of conditions, discard the target modal image of the plant.

4. The method according to claim 1, characterized in that The using a recursive multi-threshold segmentation method to segment the target modal image to obtain the foreground mask of the plant includes: Determine the near-infrared grayscale image in the target modal image; Use the maximum between-class variance to determine the segmentation threshold of the pixels, and perform the segmentation process: use the segmentation threshold to perform foreground segmentation on the near-infrared grayscale image to obtain a preliminary foreground mask, perform a bitwise OR operation on the preliminary foreground mask and the previous foreground mask to obtain the current foreground mask, determine the current background region according to the current foreground mask, and obtain the number of pixels in the current background region; In response to determining that the number of pixels is greater than or equal to the quantity threshold, re-determine the segmentation threshold and iteratively execute the segmentation process until the number of pixels is less than the quantity threshold; In response to determining that the number of pixels is less than the quantity threshold, use the current foreground mask as the foreground mask of the plant.

5. The method according to claim 1, wherein The clustering and denoising process for the foreground mask to obtain an optimized foreground mask includes: Extract at least one contour from the foreground mask; Calculate the area of each contour and perform clustering on the areas of at least one contour to obtain at least one clustering cluster; Calculate the mean of each clustering cluster and determine the optimization threshold according to the mean of each clustering cluster; Compare the area of each contour with the optimization threshold respectively, and remove the contours with areas less than the optimization threshold; Combine the remaining contours to form an optimized foreground mask.

6. The method according to claim 1, characterized in that, The multi-modal image refers to multiple types of images obtained when different imaging technologies and sensors are used for the same scene or object.

7. A chlorophyll fluorescence data analysis device based on multimodal images, characterized in that, It includes: An image acquisition module configured to acquire a multi-modal image of the chlorophyll fluorescence data of the plant and determine the target modal image in the multi-modal image; A plant height determination module configured to determine the depth histogram of the target modal image and determine the height type of the plant according to the depth histogram; A segmentation module configured to determine the segmentation method corresponding to the height type and segment the target modal image according to the segmentation method to obtain the foreground mask of the plant; A clustering optimization module configured to perform clustering and denoising processing on the foreground mask to obtain an optimized foreground mask; A photosynthetic parameter determination module configured to use the optimized foreground mask to segment at least one image measured based on photosynthesis in the multi-modal image to obtain the foreground image corresponding to the plant, and determine the photosynthetic parameters of the plant according to the foreground image; The segmentation module is specifically configured to: In response to determining that the height type is a tall plant, perform segmentation processing on the target modal image using a recursive multi-threshold segmentation method to obtain the foreground mask of the plant; In response to determining that the height type is a short plant, perform segmentation processing on the target modal image using a depth weight assignment segmentation method to obtain the foreground mask of the plant; The segmentation module is specifically configured to: Determine the chlorophyll fluorescence image in the target modal image; Perform monocular depth estimation on the chlorophyll fluorescence image to obtain a depth map; Perform normalization processing on the depth map to obtain a normalized depth map; Determine the depth weight value of each pixel point according to the normalized depth map; Determine the near-infrared grayscale image in the target modal image, and convert the near-infrared grayscale image using the depth weight value of each pixel point to obtain an adjusted near-infrared grayscale image; Perform segmentation processing on the adjusted near-infrared grayscale image to obtain the foreground mask of the plant.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 6.

Images