Chlorophyll fluorescence data analysis method, device and equipment based on multi-modal image

Abstract

The invention provides a chlorophyll fluorescence data analysis method, device and equipment based on a multi-modal image. For the multi-modal image of the chlorophyll fluorescence data of the plant, the height type of the plant is determined through the height analysis of the target modal image, so that the most suitable segmentation mode can be determined according to the height type, the foreground mask obtained through segmentation is more accurate, the foreground mask is subjected to clustering denoising processing, and the segmentation accuracy is improved. According to the method, interference of some noise in a foreground mask is avoided, the foreground mask is optimized, so that the obtained optimized foreground mask better conforms to foreground contour features of the plant, segmentation processing is performed on other modal images according to the optimized foreground mask, and a relatively accurate foreground image of the plant is obtained; in this way, the photosynthetic parameters determined according to the foreground image are more accurate and can adapt to various scenes where plants are located.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a chlorophyll fluorescence data analysis method, device and equipment based on multimodal images. Background Art

[0002] The analysis of chlorophyll fluorescence data is of great significance for assessing the health status of plants. Traditional analysis methods mainly rely on basic chlorophyll fluorescence data for foreground extraction, but in complex scenarios, such as when plants coexist with weeds and when there are large differences in plant heights, it is difficult to accurately and automatically extract the foreground. Summary of the invention

[0003] In view of this, the purpose of the present application is to propose a chlorophyll fluorescence data analysis method, device and equipment based on multimodal images to solve or partially solve the above-mentioned technical problems.

[0004] Based on the above objectives, this application proposes a chlorophyll fluorescence data analysis method based on multimodal images, including: Acquire a multimodal image for plant chlorophyll fluorescence data, and determine a target modal image in the multimodal image; Determining a depth histogram of the target modality image, and determining a height type of the plant according to the depth histogram; Determine a segmentation method corresponding to the height type, segment the target modality image according to the segmentation method, and obtain a foreground mask of the plant; Performing clustering denoising processing on the foreground mask to obtain an optimized foreground mask; The optimized foreground mask is used to segment at least one frame of the multimodal image obtained based on photosynthesis measurement to obtain a foreground image corresponding to the plant, and the photosynthetic parameters of the plant are determined according to the foreground image.

[0005] Based on the same concept, the present application also proposes a chlorophyll fluorescence data analysis device based on multimodal images, comprising: An image acquisition module is configured to acquire a multimodal image for plant chlorophyll fluorescence data and determine a target modal image in the multimodal image; a plant height determination module, configured to determine a depth histogram of the target modality image, and determine a height type of the plant according to the depth histogram; a segmentation module, configured to determine a segmentation method corresponding to the height type, and segment the target modality image according to the segmentation method to obtain a foreground mask of the plant; A clustering optimization module is configured to perform clustering denoising on the foreground mask to obtain an optimized foreground mask; The photosynthetic parameter determination module is configured to use the optimized foreground mask to segment at least one frame of the multimodal image obtained based on photosynthesis measurement to obtain a foreground image corresponding to the plant, and determine the photosynthetic parameters of the plant based on the foreground image.

[0006] Based on the same concept, the present application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-described method when executing the computer program.

[0007] From the above, it can be seen that the chlorophyll fluorescence data analysis method, device and equipment based on multimodal images provided by the present application will target the multimodal images of chlorophyll fluorescence data of the plant, and determine the height type of the plant through height analysis of the target modal image therein, so that the most appropriate segmentation method can be determined according to the height type, so that the foreground mask obtained by segmentation is more accurate, and the foreground mask will also be clustered and denoised to avoid interference from some noise in the foreground mask, and the foreground mask will be optimized so that the optimized foreground mask is more in line with the foreground contour characteristics of the plant. In this way, other modal images are segmented according to the optimized foreground mask to obtain a more accurate foreground image of the plant, so that the photosynthetic parameters determined according to the foreground image will also be more accurate and can adapt to various scenes in which the plant is located. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0009] Figure 1 This is a flow chart of a chlorophyll fluorescence data analysis method based on multimodal images according to an embodiment of the present application; Figure 2 A graphical schematic diagram of a chlorophyll fluorescence data analysis method based on multimodal images according to an embodiment of the present application; Figure 3 for Figure 2 The depth distribution map of the corresponding tall plants in the middle; Figure 4 for Figure 2 The depth distribution map of the corresponding short plants; Figure 5 A segmentation schematic diagram of a recursive multi-threshold segmentation method according to an embodiment of the present application; Figure 6 A segmentation schematic diagram of a segmentation method for depth weight allocation in an embodiment of the present application; Figure 7 for Figure 6 Schematic diagram of the curve of the normalized depth value and the corresponding depth weight value under different gain parameters k; Figure 8 A schematic diagram of clustering according to contour area according to an embodiment of the present application; Fig. 9 A schematic diagram comparing a foreground mask obtained by the recursive multi-threshold segmentation method of an embodiment of the present application and a mask obtained by a traditional method; Fig.10 for Fig. 9 Pixel distribution diagram corresponding to Fv / Fm; Fig.11 For Fig.10 Schematic diagram of the corresponding Fv / Fm distribution density; Fig.12 for Fig. 9 Pixel distribution map corresponding to NDVI; Fig.13 For Fig.12 Schematic diagram of the corresponding NDVI distribution density; Fig.14 A schematic diagram comparing a foreground mask obtained by a segmentation method using depth weight allocation according to an embodiment of the present application and a mask obtained by a traditional method; Fig.15 for Fig.14 Pixel distribution diagram corresponding to Fv / Fm; Fig.16 For Fig.15 Schematic diagram of the corresponding Fv / Fm distribution density; Fig.17 for Fig.14 Pixel distribution map corresponding to NDVI; Fig.18 For Fig.17 Schematic diagram of the corresponding NDVI distribution density; Fig.19 Chlorophyll fluorescence images recorded at different times in an embodiment of the present application; Fig. 20 This is a structural block diagram of a chlorophyll fluorescence data analysis device based on multimodal images according to an embodiment of the present application; Fig.21 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0010] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0011] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present application, and are not intended to limit the scope of the present application in any way. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0012] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0013] Glossary: INF: Device INFormation File, a file used to describe data information such as devices or files in the Windows operating system.

[0014] DAT: Data, is a data file with the extension ".dat".

[0015] OJIP: OJIP Fluorescence Induction Curve, the rapid chlorophyll fluorescence induction kinetic curve, is one of the most classic fluorescence kinetic curves.

[0016] PAM: Pulse Amplitude Modulated Fluorescence, is a technique for measuring fluorescence by pulse modulated light.

[0017] GFP: Green Fluorescent Protein, green fluorescent protein. RFP: Red Fluorescent Protein, a fluorescent protein that can be used for biological labeling and real-time imaging.

[0018] Otsu: The maximum inter-class variance method was proposed by Japanese scholar Nobuyuki Otsu in 1979. It is a method for automatically obtaining thresholds that is suitable for bimodal situations. It is also called the Otsu method, or Otsu for short.

[0019] ChlF: Chlorophyll fluorescence imaging.

[0020] Nir: Near Infrared, near infrared imaging.

[0021] The present application embodiment provides a chlorophyll fluorescence data analysis method based on multimodal images, such as Figure 1 As shown, the method includes: Step 101: Acquire a multimodal image of plant chlorophyll fluorescence data and determine a target modal image in the multimodal image.

[0022] In specific implementation, the multimodal image refers to multiple types of images obtained when different imaging technologies and sensors are used for the same scene or object. It can provide complementary information from different angles, and can more comprehensively understand and analyze the target object. Multimodal images include chlorophyll fluorescence images (such as Fig.19 as shown), near-infrared grayscale image, depth image, etc.

[0023] Chlorophyll fluorescence images can reflect the physiological state and photosynthesis efficiency of plants by detecting the fluorescence characteristics of plant chlorophyll; near-infrared grayscale images utilize the characteristics of near-infrared light to provide information different from visible light images, such as the water content or tissue structure of plants; depth maps provide spatial structure information of the scene, can reflect the three-dimensional shape and distance information of objects, and are highly robust to changes in lighting.

[0024] Chlorophyll fluorescence data files include INF file types for storing metadata information such as measurement frequency and DAT file types for storing data from different measurement procedures, including one or more file formats of multiple measurement types such as photosynthesis measurements (such as OJIP and PAM methods), chlorophyll fluorescence, reflectance spectra (near infrared, red edge), fluorescent proteins (GFP and RFP), and visible light images.

[0025] Step 102: determine a depth histogram of the target modality image, and determine the height type of the plant according to the depth histogram.

[0026] In specific implementation, a plant height classifier based on a depth histogram is used to process the depth histogram, thereby accurately estimating the height of the plant and determining the height type of the plant, wherein the height type includes at least: tall plants and short plants.

[0027] Step 103, determining a segmentation method corresponding to the height type, and segmenting the target modality image according to the segmentation method to obtain a foreground mask of the plant.

[0028] In specific implementation, different height types correspond to different segmentation methods, so that the most appropriate segmentation method can be determined in a targeted manner according to the height type, making the foreground mask obtained by segmentation more consistent with the image characteristics of the plant and more accurate.

[0029] Step 104: performing clustering denoising processing on the foreground mask to obtain an optimized foreground mask.

[0030] In specific implementation, since the obtained foreground mask may contain noise, which affects its accuracy, it is necessary to optimize it through clustering denoising to make the optimized foreground mask more accurate.

[0031] Step 105 , using the optimized foreground mask to segment at least one frame of the multimodal image obtained based on photosynthesis measurement to obtain a foreground image corresponding to the plant, and determining the photosynthetic parameters of the plant according to the foreground image.

[0032] In specific implementation, the corresponding photosynthetic parameters include: maximum quantum yield ( ), actual photochemical quantum yield ( ,Right now )、 (non-photochemical quenching), (Light Absorption Index), (electron transfer rate), (chlorophyll index), (Normalized Difference Vegetation Index) and other photosynthetic parameters. For example,

[0033]

[0034]

[0035] in, is the initial fluorescence image, is the maximum fluorescence image, is the steady-state fluorescence image, is the maximum fluorescence image under light adaptation.

[0036]

[0037]

[0038]

[0039]

[0040] in, , , , Represent the reflectance images of red light, near infrared, far infrared, and red edge bands respectively. It is the ratio of PSII. PSII (Photosystem II) is an important component of light energy conversion in photosynthesis. It is the second photochemical system of photosynthesis and is mainly responsible for the photolysis of water.

[0041] Through the above scheme, for the multimodal images of the chlorophyll fluorescence data of the plant, the height type of the plant will be determined by height analysis of the target modal image, so that the most appropriate segmentation method can be determined according to the height type, so that the foreground mask obtained by segmentation is more accurate, and the foreground mask will be clustered and denoised to avoid the interference of some noise in the foreground mask, and the foreground mask will be optimized to make the optimized foreground mask more consistent with the foreground contour characteristics of the plant. In this way, other modal images are segmented according to the optimized foreground mask to obtain a more accurate foreground image of the plant. In this way, the photosynthetic parameters determined according to the foreground image will also be more accurate and can adapt to various scenes in which the plant is located.

[0042] In some embodiments, step 102 includes: Step 1021: extract depth information of each pixel from the target modality image.

[0043] In the specific implementation, the chlorophyll fluorescence image of the target modality image is obtained through a monocular depth estimation model. Extract the depth information of each pixel ,in For images Middle position The depth value corresponding to the pixel point generates a depth map .

[0044] Step 1022: Generate a depth histogram according to the depth information.

[0045] In specific implementation, the generated depth map Mapping to depth value range Count the frequency of each depth value and generate a normalized depth histogram (like Figure 2 The formula is:

[0046] Among them, count is the depth value The number of occurrences in the image, is the normalized depth histogram, and . is the depth value Number of occurrences in the depth map .

[0047] Step 1023, determine the graphic features of the plant according to the depth histogram, compare the graphic features with the feature threshold, and determine the height type of the plant (such as Figure 2 as shown).

[0048] For plants of different heights, the features of plant foreground and ground noise in the near-infrared grayscale images are very different and cannot be applied to the same image segmentation method. Therefore, before applying image segmentation, it is necessary to use monocular depth estimation to classify the plant height type to determine the plant height type, so as to match different segmentation methods according to the height type.

[0049] Through the above solution, the graphic features of the plants determined by means of the depth histogram are more closely matched with the height of the plants, and thus the height type determined based on the graphic features is more accurate.

[0050] In some embodiments, step 1023 includes: Step 10231, determine the rectangular metric R and skewness metric of the depth histogram And the kurtosis measure .

[0051] In specific implementation, the rectangle measurement Reflects how close the shape of the depth histogram is to the ideal rectangle. The formula is:

[0052] in, is the area under the depth histogram graph, is the area of ​​the rectangular region enclosed by the depth histogram. The closer the rectangular measurement value R of the depth histogram is to 1, the closer its shape is to a rectangle.

[0053] Skewness measures Represents the asymmetry of the depth histogram distribution, the formula is:

[0054] in, is the mean of the depth histogram, is the standard deviation, is the total number of depth values ​​in the depth histogram. Skewness measure Used to determine the tail direction and shape of the depth histogram.

[0055] Kurtosis measure Indicates the sharpness of the depth histogram distribution. Its calculation formula is:

[0056] in, is a kurtosis measure that indicates how "peaked" the depth histogram is. A higher kurtosis measure generally corresponds to the histogram characteristics of tall plants, showing concentrated, sharp peaks.

[0057] Step 10232, obtain the rectangle threshold , skewness threshold And the kurtosis threshold .

[0058] Step 10233, in response to and and If the first type of condition is met, the plant is determined to be a tall plant (e.g. Figure 3 (as shown, corresponding to the depth distribution of tall plants).

[0059] Alternatively, step 10234, in response to and and If the second type of condition is met, the plant is determined to be a dwarf plant (e.g. Figure 4 As shown, the depth distribution diagram corresponding to the short plants); Alternatively, step 10235, in response to the rectangular metric R, the skewness metric And the kurtosis measure If other conditions except the first type condition and the second type condition are met, the target modality image of the plant is discarded.

[0060] Through the above scheme, according to the rectangular measurement R and Contrast, skewness measure and Comparison and kurtosis measure and By comparing the height of the plants, we can accurately judge the height of the plants and distinguish between tall plants and short plants, so as to accurately allocate the corresponding division method.

[0061] In some embodiments, step 103 includes: Step 1031 , in response to determining that the height type is a tall plant, segment the target modality image using a recursive multi-threshold segmentation method to obtain a foreground mask of the plant.

[0062] Alternatively, in step 1032, in response to determining that the height type is a short plant, the target modality image is segmented using a depth weight distribution segmentation method to obtain a foreground mask of the plant.

[0063] Through the above scheme, for tall plants, due to their large image depth difference and small ground noise, it is easy to form a stratification of depth signal intensity. Therefore, the recursive multi-threshold segmentation method can be used to accurately segment the target modal image and obtain an accurate foreground mask; for short plants, due to their small image depth difference and large ground noise, it is necessary to use the depth weight distribution segmentation method to reduce the ground noise of the target modal image, thereby ensuring that an accurate foreground mask can be obtained.

[0064] In some embodiments, step 1031 is as follows Figure 5 As shown, including: Step 10311, determine the near infrared grayscale image in the target modality image.

[0065] In specific implementation, the grayscale range of the near-infrared grayscale image is , the size is Pixel.

[0066] Step 10312, use the maximum between-class variance (Otsu) to determine the pixel segmentation threshold, and execute the segmentation process: use the segmentation threshold to perform foreground segmentation on the near-infrared grayscale image to obtain a preliminary foreground mask, perform a bitwise OR operation on the preliminary foreground mask and the previous foreground mask to obtain a current foreground mask, determine the current background area according to the current foreground mask, and obtain the number of pixels in the current background area.

[0067] In specific implementation, the process of determining the segmentation threshold is: Between-class variance The calculation formula is as follows:

[0068] in, and are the weights of class 1 and class 2, respectively. is the gray value, defined as: .

[0069] in, is the gray level of the pixel in the image The pixel probability, and is the grayscale mean of class 1 and class 2, and the calculation formula is:

[0070] The final choice makes The gray value corresponding to the maximum τ as the optimal segmentation threshold.

[0071] After the initial foreground mask and the previous foreground mask are bitwise ORed, the current foreground mask is obtained. , background area Just for , where M is the near-infrared grayscale image. After obtaining the background area, the number of pixels in the background area will be counted .

[0072] Step 10313, in response to determining that the number of pixels is greater than or equal to the number threshold, re-determine the segmentation threshold, and iterate the segmentation process until the number of pixels is less than the number threshold.

[0073] In specific implementation, if the number of pixels in the background area , where m is the number of horizontal pixels of the near-infrared grayscale image, and n is the number of vertical pixels of the near-infrared grayscale image; it proves that the background area has not been segmented well and needs to be segmented again, and the segmentation threshold will be re-determined according to the above segmentation threshold determination process , according to the segmentation threshold, the background area Repeat the segmentation until the Conditions till now.

[0074] Step 10314, in response to determining that the number of pixels is less than the number threshold, using the current foreground mask as the foreground mask of the plant (eg Figure 5 The final foreground mask is shown in ).

[0075] Through the above scheme, the recursive multi-threshold segmentation method is the Otsu segmentation method based on recursive multi-threshold, which is used in the near-infrared grayscale image of tall plants. Due to the large depth difference of the near-infrared grayscale image, a large amount of ground noise in the chlorophyll fluorescence image is effectively avoided, but the near-infrared signal intensity of the leaves of plants at different depths forms a layer. Therefore, the segmentation of the entire near-infrared grayscale image is divided into multiple levels, and the optimal threshold of each level is automatically calculated. By refining the segmentation of the background area in each recursive layer, automatic and multi-level image segmentation is achieved to obtain an accurate foreground mask.

[0076] Specific as Fig. 9 The results of different modal images (e.g., ChlF chlorophyll fluorescence imaging, Fv / Fm maximum quantum yield imaging, Nir near-infrared imaging, Depth depth map) obtained under traditional single-modal mask processing are shown, and the foreground mask results (i.e., Fig. 9 The mask and photosynthetic parameter pseudo-color map based on recursive multi-threshold segmentation optimization in Fig.10 and 11 The corresponding schematic diagram of Fv / Fm, and Fig.12 and 13 Corresponding to the schematic diagram of NDVI, it is obvious that the solution of the present application has better effect.

[0077] In some embodiments, step 1032 is as follows Figure 6 As shown, including: Step 10321, determining the chlorophyll fluorescence image in the target modality image.

[0078] Step 10322: Perform monocular depth estimation on the chlorophyll fluorescence image to obtain a depth map.

[0079] In a specific implementation, a monocular depth estimation model is used to generate a depth map for the chlorophyll fluorescence image, and the depth map can describe the depth information of each area in the image.

[0080] Step 10323: normalize the depth map to obtain a normalized depth map.

[0081] In specific implementation, the normalization formula is as follows:

[0082] in, is the depth value in the normalized depth map, is the depth value in the depth map before normalization, is the minimum depth value in the depth map, is the maximum depth value in the depth map.

[0083] Step 10324, determining the depth weight value of each pixel according to the normalized depth map.

[0084] When implementing it, Figure 7 As shown, the normalized depth map is determined as (That is, corresponding to Figure 7 The horizontal coordinate is the normalized depth value in the middle), i is the position order of the pixels, and the depth weight value of each pixel (i.e., the corresponding Figure 7 The calculation formula of the ordinate weight in is:

[0085] Wherein, k is a gain parameter, which is used to control the steepness of the curve; To set a constant, it is usually set to 0.5 (or the median of the normalized depth values).

[0086] Step 10325, determining a near infrared grayscale image in the target modality image, and converting the near infrared grayscale image using the depth weight value of each pixel to obtain an adjusted near infrared grayscale image.

[0087] Step 10326, segmenting the adjusted near-infrared grayscale image to obtain a foreground mask of the plant.

[0088] In specific implementation, the calculated weight coefficient is applied to the near-infrared grayscale image (NIR) to adjust the grayscale value of the near-infrared grayscale image, enhance its foreground area and weaken the influence of the background area. Adjusted near-infrared grayscale image The calculation formula is:

[0089] in, is the grayscale value of the near-infrared grayscale image, It is the depth weight value calculated by the depth map.

[0090] Through the above scheme, the foreground and background of the plant can be accurately separated, and the ground noise can be effectively removed, making the foreground mask obtained after the segmentation process more accurate. Fig.14 The results of different modal images (e.g., ChlF chlorophyll fluorescence imaging, Fv / Fm maximum quantum yield imaging, Nir near-infrared imaging, Depth depth map) obtained under traditional single-modal mask processing are shown, and the foreground mask results (i.e., Fig.14 The mask and photosynthetic parameter pseudo-color map based on the depth weight allocation segmentation optimization part) combined with Fig.15 and 16 The corresponding schematic diagram of Fv / Fm, and Fig.17 and 18 Corresponding to the schematic diagram of NDVI, it is obvious that the solution of the present application has better effect.

[0091] In some embodiments, step 104 is as follows Figure 8 As shown, including: Step 1041: extract at least one contour from the foreground mask.

[0092] Determine the foreground mask Use image processing libraries such as OpenCV to extract the foreground mask from the foreground mask using contour detection algorithms such as cv2.findContours Extract the contours and get a contour set , where each represents a contour, where i∈n, n is the number of contours. Step 1042, calculate the area of ​​each contour, and perform clustering processing on the area of ​​at least one contour to obtain at least one cluster.

[0093] In specific implementation, for each contour Calculate its area : . Get a set of contour areas: .

[0094] Then the K-Means algorithm is used to cluster the contour area set into k clusters, each of which contains contours with similar areas.

[0095] Step 1043, calculating the mean of each cluster, and determining the optimization threshold according to the mean of each cluster.

[0096] In specific implementation, for each cluster, the mean of all contour areas in the cluster (i.e., the cluster center) is calculated. The center of the cluster is , that is, the mean of each cluster: ; in, Indicates Clusters, It is a cluster The number of contours in .

[0097] For the k means obtained ,In order to determine the optimal threshold T of the area, the median of all cluster means is first calculated.

[0098] like , then the optimization threshold can be directly taken and The average value of: .

[0099] like , then remove the maximum mean , calculate the median of the remaining means as the optimization threshold T: .

[0100] Step 1044, compare the area of ​​each contour with the optimization threshold, and remove contours that are smaller than the optimization threshold.

[0101] Each outline , if its area Less than the optimization threshold , then remove the contour:

[0102] Step 1045, combine the remaining contours to form an optimized foreground mask.

[0103] In the specific implementation, for each remaining contour , draw it onto the mask image to get the optimized foreground mask .

[0104] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0105] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0106] Based on the same concept, corresponding to the chlorophyll fluorescence data analysis method based on multimodal images in any of the above embodiments, the present application also provides a chlorophyll fluorescence data analysis device based on multimodal images.

[0107] refer to Fig. 20 , the device comprises: The image acquisition module 201 is configured to acquire a multimodal image for plant chlorophyll fluorescence data and determine a target modal image in the multimodal image; The plant height determination module 202 is configured to determine a depth histogram of the target modality image, and determine the height type of the plant according to the depth histogram; The segmentation module 203 is configured to determine a segmentation method corresponding to the height type, and segment the target modality image according to the segmentation method to obtain a foreground mask of the plant; The clustering optimization module 204 is configured to perform clustering denoising processing on the foreground mask to obtain an optimized foreground mask; The photosynthetic parameter determination module 205 is configured to segment at least one frame of the multimodal image obtained based on photosynthesis measurement using the optimized foreground mask to obtain a foreground image corresponding to the plant, and determine the photosynthetic parameters of the plant according to the foreground image.

[0108] In some embodiments, the plant height determination module 202 is specifically configured to: Extracting depth information of each pixel from the target modality image; generating a depth histogram according to the depth information; The graphic features of the plant are determined according to the depth histogram, and the graphic features are compared with the feature threshold to determine the height type of the plant.

[0109] In some embodiments, the plant height determination module 202 is specifically configured to: Determine the rectangular metric R and skewness metric of the depth histogram And the kurtosis measure ; Get the rectangle threshold , skewness threshold And the kurtosis threshold ; In response to and and The first type of condition is met, and the plant is determined to be a tall plant; or, In response to and and The plant is determined to be a dwarf plant if the second type of condition is met; or, Responding to the rectangular metric R, the skewness metric And the kurtosis measure If other conditions except the first type condition and the second type condition are met, the target modality image of the plant is discarded.

[0110] In some embodiments, the segmentation module 203 is specifically configured to: In response to determining that the height type is a tall plant, segmenting the target modality image using a recursive multi-threshold segmentation method to obtain a foreground mask of the plant; or, In response to determining that the height type is a short plant, the target modality image is segmented using a depth weight distribution segmentation method to obtain a foreground mask of the plant.

[0111] In some embodiments, the segmentation module 203 is specifically configured to: determining a near infrared grayscale image in a target modality image; The pixel segmentation threshold is determined by maximizing the inter-class variance, and the segmentation process is performed: the near-infrared grayscale image is segmented by the segmentation threshold to obtain a preliminary foreground mask, the preliminary foreground mask is bitwise ORed with the previous foreground mask to obtain a current foreground mask, the current background area is determined according to the current foreground mask, and the number of pixels in the current background area is obtained; In response to determining that the number of pixels is greater than or equal to a number threshold, re-determining a segmentation threshold, and iteratively performing the segmentation process until the number of pixels is less than the number threshold; In response to determining that the number of pixels is less than a number threshold, the current foreground mask is used as a foreground mask of the plant.

[0112] In some embodiments, the segmentation module 203 is specifically configured to: determining a chlorophyll fluorescence image in the target modality image; Performing monocular depth estimation on the chlorophyll fluorescence image to obtain a depth map; Normalizing the depth map to obtain a normalized depth map; Determining a depth weight value of each pixel according to the normalized depth map; Determine a near infrared grayscale image in the target modality image, and convert the near infrared grayscale image using a depth weight value of each pixel to obtain an adjusted near infrared grayscale image; The adjusted near-infrared grayscale image is segmented to obtain a foreground mask of the plant.

[0113] In some embodiments, the cluster optimization module 204 is specifically configured to: extracting at least one contour from the foreground mask; Calculating the area of ​​each contour, and performing clustering processing on the area of ​​at least one contour to obtain at least one cluster; Calculate the mean of each cluster, and determine the optimization threshold according to the mean of each cluster; Compare the area of ​​each contour with the optimization threshold, and remove contours smaller than the optimization threshold; The remaining contours are combined to form the optimized foreground mask.

[0114] In some embodiments, the multimodal image refers to multiple types of images acquired when different imaging technologies and sensors are used for the same scene or object.

[0115] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0116] The device of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0117] Based on the same concept, corresponding to the method of any of the above embodiments, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of any of the above embodiments when executing the computer program.

[0118] Fig.21 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0119] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0120] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0121] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0122] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0123] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0124] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0125] The electronic device of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0126] Based on the same concept, corresponding to any of the above-mentioned embodiments, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in any of the above embodiments.

[0127] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM, Parameter Random Access Memory, parameter random access memory), static random access memory (SRAM, Static Random-Access Memory), dynamic random access memory (DRAM, Dynamic Random Access Memory), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM, Electrically Erasable Programmable read only memory), flash memory or other memory technology, read-only CD-ROM (Compact Disc Read-Only Memory), digital versatile disc (DVD, Digital Video Disc) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0128] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0129] Based on the same concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a computer program product, including computer program instructions. When the computer program instructions are run on a computer, the computer executes the method described in any of the above embodiments, which has the beneficial effects of the corresponding method embodiments and will not be repeated here.

[0130] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0131] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power / ground connections to the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device may be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present application are to be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0132] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., DRAM) may use the embodiments discussed.

[0133] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A chlorophyll fluorescence data analysis method based on multimodal images, characterized in that: include: Acquire a multimodal image for plant chlorophyll fluorescence data, and determine a target modal image in the multimodal image; Determining a depth histogram of the target modality image, and determining a height type of the plant according to the depth histogram; Determine a segmentation method corresponding to the height type, segment the target modality image according to the segmentation method, and obtain a foreground mask of the plant; Performing clustering denoising processing on the foreground mask to obtain an optimized foreground mask; The optimized foreground mask is used to segment at least one frame of the multimodal image obtained based on photosynthesis measurement to obtain a foreground image corresponding to the plant, and the photosynthetic parameters of the plant are determined according to the foreground image.

2. The method according to claim 1, characterized in that The step of determining a depth histogram of the target modality image and determining the height type of the plant according to the depth histogram includes: Extracting depth information of each pixel from the target modality image; generating a depth histogram according to the depth information; The graphic features of the plant are determined according to the depth histogram, and the graphic features are compared with the feature threshold to determine the height type of the plant.

3. The method according to claim 2, characterized in that Determining the graphic features of the plant according to the depth histogram, comparing the graphic features with a feature threshold, and determining the height type of the plant includes: Determine the rectangular metric R and skewness metric of the depth histogram And the kurtosis measure ; Get the rectangle threshold , skewness threshold And the kurtosis threshold ; In response to and and The first type of condition is met, and the plant is determined to be a tall plant; or, In response to and and The plant is determined to be a dwarf plant if the second type of condition is met; or, Responding to the rectangular metric R, the skewness metric And the kurtosis measure If other conditions except the first type condition and the second type condition are met, the target modality image of the plant is discarded.

4. The method according to claim 1, characterized in that: The determining of a segmentation method corresponding to the height type, segmenting the target modality image according to the segmentation method, and obtaining a foreground mask of the plant includes: In response to determining that the height type is a tall plant, segmenting the target modality image using a recursive multi-threshold segmentation method to obtain a foreground mask of the plant; or, In response to determining that the height type is a short plant, the target modality image is segmented using a depth weight distribution segmentation method to obtain a foreground mask of the plant.

5. The method according to claim 4, characterized in that The segmentation process of the target modality image using a recursive multi-threshold segmentation method to obtain a foreground mask of the plant includes: determining a near infrared grayscale image in a target modality image; The pixel segmentation threshold is determined by maximizing the inter-class variance, and the segmentation process is performed: the near-infrared grayscale image is segmented by the segmentation threshold to obtain a preliminary foreground mask, the preliminary foreground mask is bitwise ORed with the previous foreground mask to obtain a current foreground mask, the current background area is determined according to the current foreground mask, and the number of pixels in the current background area is obtained; In response to determining that the number of pixels is greater than or equal to a number threshold, re-determining a segmentation threshold, and iteratively performing the segmentation process until the number of pixels is less than the number threshold; In response to determining that the number of pixels is less than a number threshold, the current foreground mask is used as a foreground mask of the plant.

6. The method according to claim 4, characterized in that The segmentation method using depth weight distribution is used to segment the target modality image to obtain a foreground mask of the plant, including: determining a chlorophyll fluorescence image in the target modality image; Performing monocular depth estimation on the chlorophyll fluorescence image to obtain a depth map; Normalizing the depth map to obtain a normalized depth map; Determining a depth weight value of each pixel according to the normalized depth map; Determine a near infrared grayscale image in the target modality image, and convert the near infrared grayscale image using a depth weight value of each pixel to obtain an adjusted near infrared grayscale image; The adjusted near-infrared grayscale image is segmented to obtain a foreground mask of the plant.

7. The method according to claim 1, characterized in that The performing clustering denoising processing on the foreground mask to obtain an optimized foreground mask includes: extracting at least one contour from the foreground mask; Calculating the area of ​​each contour, and performing clustering processing on the area of ​​at least one contour to obtain at least one cluster; Calculate the mean of each cluster, and determine the optimization threshold according to the mean of each cluster; Compare the area of ​​each contour with the optimization threshold, and remove contours smaller than the optimization threshold; The remaining contours are combined to form the optimized foreground mask.

8. The method according to claim 1, characterized in that The multimodal image refers to multiple types of images acquired when different imaging technologies and sensors are used for the same scene or object.

9. A chlorophyll fluorescence data analysis device based on multimodal images, characterized in that: include: An image acquisition module is configured to acquire a multimodal image for plant chlorophyll fluorescence data and determine a target modal image in the multimodal image; a plant height determination module, configured to determine a depth histogram of the target modality image, and determine a height type of the plant according to the depth histogram; a segmentation module, configured to determine a segmentation method corresponding to the height type, and segment the target modality image according to the segmentation method to obtain a foreground mask of the plant; A clustering optimization module is configured to perform clustering denoising on the foreground mask to obtain an optimized foreground mask; The photosynthetic parameter determination module is configured to use the optimized foreground mask to segment at least one frame of the multimodal image obtained based on photosynthesis measurement to obtain a foreground image corresponding to the plant, and determine the photosynthetic parameters of the plant based on the foreground image.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

Images