Leaf vegetable phenotype parameter measurement method and system based on binocular vision and deep learning

Abstract

The invention relates to the technical field of intelligent agriculture, and discloses a leaf vegetable phenotype parameter measurement method and system based on binocular vision and deep learning. The method comprises the following steps: acquiring an RGB left view and an RGB right view of a leaf vegetable by using a binocular camera, and obtaining a depth map of the leaf vegetable through stereo matching; performing semantic segmentation on the RGB left view by using the leaf vegetable canopy image segmentation model to obtain a segmentation mask; segmenting the RGB left view and the depth map by using a segmentation mask to obtain an RGB segmentation map and a depth segmentation map; based on the RGB segmentation image and the depth segmentation image, geometric parameters of the leaf vegetables are obtained; inputting the RGB segmentation image, the depth segmentation image and the geometric parameters into a pre-trained leaf area regression model of the leaf vegetables, predicting the single-plant leaf area of the leaf vegetables, and outputting phenotypic parameters of the leaf vegetables; the phenotypic parameters comprise plant height, diameter, projection area and single plant leaf area. According to the method, the production condition of the leaf vegetables can be rapidly monitored, the monitoring cost can be reduced, and the monitoring precision and efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of smart agricultural technology, and specifically to a method and system for measuring leafy vegetable phenotypic parameters based on binocular vision and deep learning. Background Art

[0002] Smart agriculture is a new agricultural production method that takes information and knowledge as core elements, deeply integrates with agriculture through modern information technology and intelligent equipment, and realizes information perception, quantitative decision-making, intelligent control, precise input, and personalized services throughout the agricultural production process. As one of the important application areas of smart agriculture, plant factories are an efficient agricultural system that achieves continuous growth of crops throughout the year through high-precision environmental control within the facility. It has the characteristics and advantages of high resource utilization, concentrated technology, controllable environment, and high production quality. It can effectively solve the problems faced by traditional agriculture, such as the aging of the agricultural population, insufficient arable land, pests and diseases, and deterioration of climate and soil environment. In addition, crop phenotypic characteristics monitoring is also an important task in factory crop cultivation. In the plant factory environment, crop planting is highly automated, and condition control systems such as environmental systems, lighting systems, and fertilization systems need to judge the growth status of crops based on the real-time phenotypic parameters of crops and make corresponding adjustments. Therefore, crop phenotypic monitoring is widely used in plant factories.

[0003] Mainstream traditional crop phenotyping methods are time-consuming, labor-intensive, highly subjective, and sometimes destructive. Compared with traditional methods, computer vision technology obtains crop phenotypic parameters in a low-cost, non-destructive, and efficient manner. Computer vision technology usually requires the capture of crop images first, and then further analysis to obtain crop phenotypic parameters by extracting image features. In terms of capturing crop images, depth cameras can obtain depth information of crops, solving the information limitation problem of two-dimensional images. However, commonly used TOF cameras and structured light cameras are expensive. Summary of the invention

[0004] The present application provides a method for measuring leafy vegetable phenotypic parameters based on binocular vision and deep learning to solve the problems in the prior art that traditional crop phenotyping methods are time-consuming, labor-intensive, highly subjective, and destructive, and that commonly used TOF cameras and structured light cameras are expensive.

[0005] Correspondingly, the present application also provides a leafy vegetable phenotypic parameter measurement system based on binocular vision and deep learning, an electronic device, and a computer-readable storage medium to ensure the implementation and application of the above method.

[0006] In order to solve the above technical problems, the present application discloses a method for measuring leafy vegetable phenotypic parameters based on binocular vision and deep learning, the method comprising:

[0007] The RGB left view and RGB right view of the leafy vegetables are collected by using a binocular camera, and the depth map of the leafy vegetables is obtained by stereo matching;

[0008] Use the pre-trained leafy vegetable canopy image segmentation model to perform semantic segmentation on the RGB left view and obtain the segmentation mask;

[0009] The RGB left view and the depth map are segmented using the segmentation mask to obtain an RGB segmentation map and a depth segmentation map;

[0010] Based on the RGB segmentation map and the depth segmentation map, the geometric parameters of the leafy vegetables are obtained;

[0011] The RGB segmentation map, depth segmentation map and geometric parameters are input into the pre-trained leaf area regression model of leafy vegetables to predict the leaf area per plant of leafy vegetables and output the phenotypic parameters of leafy vegetables; the phenotypic parameters include plant height, diameter, projected area and leaf area per plant.

[0012] The present application also discloses a leafy vegetable phenotypic parameter measurement system based on binocular vision and deep learning, the system comprising:

[0013] An image acquisition module is used to acquire an RGB left view and an RGB right view of the leafy vegetables using a binocular camera, and obtain a depth map of the leafy vegetables through stereo matching;

[0014] An image segmentation module is used to perform semantic segmentation on the RGB left view using a pre-trained leafy vegetable canopy image segmentation model to obtain a segmentation mask;

[0015] The image segmentation module is also used to segment the RGB left view and the depth map using the segmentation mask to obtain an RGB segmentation map and a depth segmentation map;

[0016] A geometric parameter acquisition module is used to obtain the geometric parameters of leafy vegetables based on the RGB segmentation map and the depth segmentation map;

[0017] The phenotypic parameter prediction module is used to input the RGB segmentation map, the depth segmentation map and the geometric parameters into the pre-trained leaf area regression model of leafy vegetables, predict the leaf area per plant of leafy vegetables, and output the phenotypic parameters of leafy vegetables; the phenotypic parameters include plant height, diameter, projected area and leaf area per plant.

[0018] The present application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, one or more methods described in the present application are implemented.

[0019] The present application also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, one or more methods described in the present application are implemented.

[0020] In this application, a binocular camera is used to collect leafy vegetable images in a factory environment with high-throughput data requirements, and combined with deep learning technology (leafy vegetable canopy image segmentation model and leafy vegetable leaf area regression model) for analysis, which can reduce monitoring costs and improve monitoring accuracy and efficiency. Among them, after non-destructively measuring the geometric parameters of leafy vegetables, the RGB information, depth information and geometric information of leafy vegetables are fused to predict the single plant leaf area of ​​leafy vegetables, and the destruction of leafy vegetables is avoided while achieving highly automated monitoring. This application can quickly monitor the production status of leafy vegetables, provide necessary leafy vegetable parameters for control systems and production personnel, facilitate plant factory production personnel to monitor leafy vegetable phenotypic characteristics in real time, more intuitively grasp the current growth state of leafy vegetables, and provide necessary data for leafy vegetable production systems, and ultimately achieve low-cost, high-quality production of leafy vegetables in factories.

[0021] Additional aspects and advantages of the present application will be given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A flow chart of a method for measuring leafy vegetable phenotypic parameters based on binocular vision and deep learning provided in an embodiment of the present application;

[0024] Figure 2 A specific flow chart for measuring leafy vegetable phenotypic parameters based on binocular vision and deep learning provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram of correcting and stereo matching images captured by a binocular camera provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of a leafy vegetable leaf area regression model provided in an embodiment of the present application;

[0027] Figure 5 A schematic diagram of the structure of a leafy vegetable phenotypic parameter measurement system based on binocular vision and deep learning is provided for the embodiment of the present application;

[0028] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0030] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0031] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0032] The solution provided in the embodiments of the present application can be executed by any electronic device, such as a terminal device or a server, wherein the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and the present application does not limit this. With regard to the technical problems existing in the prior art, the leafy vegetable phenotypic parameter measurement method and system based on binocular vision and deep learning provided in the present application is intended to solve at least one of the technical problems of the prior art.

[0033] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0034] With the rapid development of artificial intelligence technology, deep learning technology has gradually matured and is widely used in agricultural production. It has shown excellent performance in tasks such as image segmentation and non-destructive measurement. The acquisition of crop image information based on deep learning provides important technical support for the measurement of crop phenotypic parameters in plant factories. In order to achieve the goals of cost savings and high-quality production of leafy vegetables in plant factories, it is necessary to accurately and quickly monitor the phenotypic characteristics of leafy vegetables.

[0035] Therefore, an embodiment of the present application provides a method for measuring leafy vegetable phenotypic parameters based on binocular vision and deep learning. The method can be executed by any electronic device, and optionally, can be executed on a server or terminal device.

[0036] like Figure 1 As shown in , the method may include the following steps:

[0037] Step 101: Use a binocular camera to collect the RGB left view and RGB right view of the leafy vegetables, and obtain a depth map of the leafy vegetables through stereo matching.

[0038] In an embodiment of the present application, the position of the binocular camera is fixed on the top plate above the cultivation platform for leafy vegetable production in the plant factory, and the binocular camera can move with the slide rail to achieve the canopy image capture of individual leafy vegetables one by one.

[0039] Compared with TOF cameras and structured light cameras, binocular cameras have price advantages and are more robust in the lighting environment of plant factories. With binocular vision technology, three-dimensional spatial information of crops can be obtained, providing more data support for the measurement of crop phenotypic parameters, reducing the workload of collecting images and improving monitoring efficiency.

[0040] Step 102: Use the pre-trained leafy vegetable canopy image segmentation model to perform semantic segmentation on the RGB left view to obtain a segmentation mask.

[0041] Step 103: Use the segmentation mask to segment the RGB left view and the depth map to obtain an RGB segmentation map and a depth segmentation map.

[0042] Step 104: obtaining geometric parameters of leafy vegetables based on the RGB segmentation map and the depth segmentation map;

[0043] Step 105, input the RGB segmentation map, the depth segmentation map and the geometric parameters into the pre-trained leaf area regression model of leafy vegetables, predict the leaf area per plant of leafy vegetables, and output the phenotypic parameters of the leafy vegetables; the phenotypic parameters include plant height, diameter, projected area and leaf area per plant.

[0044] After completing the collection and preprocessing of leafy vegetable image data and target geometric parameters, this application uses computer binocular vision technology and deep learning related models to study the phenotypic characteristics of leafy vegetables, obtain phenotypic parameters, and provide an accurate basis for environmental control.

[0045] In the embodiment of the present application, a binocular camera is used to collect leafy vegetable images in a factory environment with high-throughput data requirements, and combined with deep learning technology (leafy vegetable canopy image segmentation model and leafy vegetable leaf area regression model) for analysis, which can reduce monitoring costs and improve monitoring accuracy and efficiency. Among them, after non-destructively measuring the geometric parameters of leafy vegetables, the RGB information, depth information and geometric information of leafy vegetables are fused to predict the single plant leaf area of ​​leafy vegetables, and highly automated monitoring is achieved while avoiding damage to leafy vegetables. The embodiment of the present application can quickly monitor the production status of leafy vegetables, provide necessary leafy vegetable parameters for the control system and production personnel, facilitate plant factory production personnel to monitor leafy vegetable phenotypic characteristics in real time, more intuitively grasp the current growth state of leafy vegetables, and provide necessary data for the leafy vegetable production system, and finally realize low-cost and high-quality production of leafy vegetables in the factory.

[0046] In an optional embodiment, if Figure 2 As shown, a binocular camera is used to collect the RGB left view and RGB right view of the leafy vegetables, and a depth map of the leafy vegetables is obtained through stereo matching, including:

[0047] Step 1.1: Calibrate the binocular camera to obtain its internal and external parameters and distortion coefficients.

[0048] In the embodiment of the present application, the Zhang Zhengyou calibration method in the OpenCV toolbox is used to calibrate the binocular camera, and multiple groups of calibration plate images taken at different angles are used to solve the internal and external parameters (intrinsic parameter matrix, external parameter matrix) and distortion coefficient of the camera.

[0049] Step 1.2: Use a binocular camera to photograph the canopy of leafy vegetables to obtain an RGB left view and an RGB right view of the leafy vegetables.

[0050] For example, after the leafy vegetable seeds are rooted, they are implanted in impregnated rock wool, placed in a dark and low-temperature environment for 1-2 days, and transplanted into a growth cabinet for cultivation after two leaves grow out. A low-cost Intel RealSense D405 binocular camera is used to capture images of the leafy vegetable canopy to obtain distorted RGB left and right views. The hanging height of the binocular camera is not limited in the embodiments of the present application.

[0051] Step 1.3: Use the internal and external parameters and the distortion coefficients to perform distortion correction on the RGB left view and the RGB right view.

[0052] Step 1.4: Use the Bouguet stereo correction algorithm to calibrate the distortion-corrected RGB left view and RGB right view to the same plane (i.e., stereo correction) to obtain the corrected RGB left view and RGB right view.

[0053] Step 1.5: Taking the corrected RGB left view as a reference, stereo matching is performed on the corrected RGB left view and the RGB right view through a semi-global matching algorithm (SGBM) to obtain the disparity of the corrected RGB left view and the RGB right view.

[0054] Step 1.6: Obtain the depth map of the leafy vegetable based on the disparity calculation.

[0055] In the embodiment of the present application, the depth map of the canopy image of the leafy vegetable is calculated based on the triangulated imaging principle. The RGB left view and the RGB right view of the leafy vegetable are subjected to distortion correction and stereo correction, and the RGB left view and the depth map are shown as follows: Figure 3 shown.

[0056] In an optional embodiment, if Figure 2 As shown, the RGB left view is semantically segmented using a pre-trained leafy vegetable canopy image segmentation model. Before obtaining the segmentation mask, the method further includes:

[0057] Step 2.1, obtain the RGB left view and depth map of the leafy vegetable canopy.

[0058] The RGB left view and depth map of the leafy vegetable canopy can be obtained according to the method in steps 1.1-1.6 above. The RGB left view described in this step is a corrected RGB left view.

[0059] Step 2.2: Measure the diameter, plant height and total leaf area of ​​each leafy vegetable in the leafy vegetable canopy.

[0060] In the embodiment of the present application, a vernier caliper is used to measure the diameter and height of a single leaf vegetable, a leaf area meter is used to destructively measure the total leaf area of ​​a single leaf vegetable, and a label is recorded for the plant height, diameter and leaf area of ​​each leaf vegetable.

[0061] Step 2.3: Use Labelme software to segment and label the RGB left view of the leafy vegetable canopy to obtain segmentation region labeling information.

[0062] Step 2.4: Create labels based on the segmented area annotation information and the diameter, plant height and total leaf area of ​​each leafy vegetable.

[0063] Step 2.5: Create a leafy vegetable dataset based on the RGB left view, depth map, and labels of the leafy vegetable canopy.

[0064] Exemplarily, the rectified RGB left view, depth map and label of the leafy vegetable canopy are cropped so that the resolution of the image is 512×512 pixels, and the complete canopy of the leafy vegetable is located in the center of the image. The RGB left view, depth map and label of the cropped leafy vegetable canopy are enhanced by clockwise rotation of 90°, 180°, 270°, horizontal flipping, vertical flipping, and brightness changes (120%, 110%, 90%, 80%) to produce a leafy vegetable dataset.

[0065] Step 2.6: Use the leafy vegetable dataset to train the leafy vegetable canopy image segmentation model.

[0066] In the embodiment of the present application, the leafy vegetable data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:1 for training a leafy vegetable canopy image segmentation model.

[0067] In an optional embodiment, the loss function of the leafy vegetable canopy image segmentation model is a cross entropy loss function.

[0068] The calculation formula of the cross entropy loss function is:

[0069]

[0070] Among them, p represents the true distribution, q represents the predicted distribution, and x represents a sample or event.

[0071] In an optional embodiment, the leafy vegetable canopy image segmentation model uses a DeepLabV3+ network;

[0072] Use the pre-trained leafy vegetable canopy image segmentation model to perform semantic segmentation on the RGB left view and obtain the segmentation mask, including:

[0073] Step 3.1: Input the RGB left view into the feature extraction module of the pre-trained DeepLabV3+ network to extract high-level features and low-level features.

[0074] In the embodiment of the present application, the RGB left view and the depth map are first preprocessed. The preprocessing operation is: for the leafy vegetables whose phenotypic parameters need to be measured, the RGB left view and the depth map are cropped so that the leafy canopy is located in the center of the image as much as possible. The image size after cropping according to the center pixel of the image is 512×512 pixels, and the leafy canopy is located in the center of the image. Then the preprocessed RGB left view is input into the pre-trained DeepLabV3+ network for feature extraction.

[0075] Step 3.2, after upsampling, the high-level features are fused with the low-level features to obtain RGB fusion features;

[0076] Step 3.3: Convolve and upsample the RGB fusion features to obtain a segmentation mask. Optionally, output the segmentation mask of the leafy canopy after 3×3 convolution and upsampling.

[0077] The RGB left view and the depth map are segmented using the segmentation mask to obtain an RGB segmentation map and a depth segmentation map.

[0078] In an optional embodiment, the geometric parameters include plant height, diameter, projected area and volume; Figure 2 As shown, based on the RGB segmentation map and the depth segmentation map, the geometric parameters of the leafy vegetables are obtained, including:

[0079] Step 4.1, determine the minimum depth value of the depth segmentation map, and obtain the plant height of the leafy vegetables by calculating the difference between the known distance from the binocular camera to the leafy vegetable cultivation platform and the minimum depth value.

[0080] It is known that the binocular camera is directly above the leaf vegetable cultivation platform, and the distance is recorded as H 1 ; Find the point with the smallest depth value in the depth segmentation map, and consider this point as the highest point of the leaf vegetable, and record the distance as H 2 , at this time, the calculation formula for the plant height H of leafy vegetables is:

[0081] H=H 1 -H 2

[0082] Step 4.2: Combine the RGB segmentation map and the depth segmentation map to generate a 3D point cloud, project the 3D point cloud along the Z axis, and use 2D point cloud computing to obtain the diameter of the leafy vegetable.

[0083] In the embodiment of the present application, the generated 3D point cloud is projected along the Z direction to obtain a 2D point cloud of the leafy vegetable canopy. All points in the 2D point cloud are found through Graham scanning, and a pair of points with the farthest Euclidean distance in the 2D point cloud are found. The distance at this time is recorded as the diameter of the leafy vegetable, and the calculation formula is:

[0084]

[0085] Where D is the diameter of the leafy vegetable, (x i ,y i ),(x j ,y j ) represents two different projection points in the 2D point cloud.

[0086] Step 4.3: Voxelize the 3D point cloud and calculate the projection area of ​​the leafy vegetable using the number of voxels and the area per unit voxel.

[0087] Specifically, select voxels of appropriate size and voxelize the 3D point cloud generated in step 4.2. Then calculate the number of voxels n. Since the size of the voxel is known, the projection area of ​​the leafy canopy is calculated as:

[0088] S projected =voxel_size 2 ×n

[0089] Among them, S prohected Represents the projection area of ​​the leafy canopy image, vixel_size 2 Represents the area of ​​a unit voxel in cm 2 , n represents the number of voxels.

[0090] Step 4.4, add a virtual point with a Z value of H1 to the 3D point cloud, and voxelize the 3D point cloud after adding the virtual point, and use the distance from each voxel to the virtual point multiplied by the sum of the area of ​​the unit voxel to estimate the volume of the leafy vegetable;

[0091] Among them, each point in the 3D point cloud is represented by an XYZ coordinate in three-dimensional space; the virtual point is located at the center of gravity of the point cloud in the XY plane.

[0092] Since the leafy vegetables below the canopy are blocked during shooting, in order to characterize the volume of the leafy vegetables, a virtual point P0 (X0, Y0, Z0) with a Z value of H1 is added at the center of gravity of the XY plane of the leafy vegetable canopy point cloud. This point is located at the top of the leafy vegetable cultivation platform.

[0093] The Euclidean distance from the virtual point to each voxel point is calculated, and the volume of the leafy vegetable is estimated using these distances per voxel area. The specific calculation formula is:

[0094]

[0095] Where V represents the estimated volume of leafy vegetables, D io Represents the distance from the i-th voxel point to the virtual point P 0 The distance in cm, voxel_size 2 Represents the area of ​​a unit voxel in cm 2 , n represents the number of voxels.

[0096] In an optional embodiment, if Figure 4 As shown, the leafy vegetable leaf area regression model includes a first densely connected convolutional network (DenseNet), a second densely connected convolutional network, a first fully connected network, and a second fully connected network;

[0097] The RGB segmentation map, depth segmentation map and geometric parameters are input into the pre-trained leaf area regression model to predict the leaf area of ​​leafy vegetables and output the phenotypic parameters of leafy vegetables, including:

[0098] Step 5.1: Input the RGB segmentation map and the depth segmentation map into the first densely connected convolutional network and the second densely connected convolutional network respectively to obtain the corresponding RGB high-level features and depth high-level features.

[0099] In an embodiment of the present application, the first densely connected convolutional network and the second densely connected convolutional network have the same structure.

[0100] Step 5.2: Combine the plant height, diameter, projected area and volume of the leafy vegetables into a four-dimensional vector, and input the four-dimensional vector into the first fully connected neural network to obtain the geometric features of the leafy vegetables.

[0101] Optionally, the first fully convolutional network has 5 layers, including 1 input layer, 3 hidden layers and 1 output layer, and each layer of neurons uses ReLU as the activation function.

[0102] Step 5.3: Concatenate RGB high-level features, deep high-level features, and geometric features to obtain multimodal fusion features.

[0103] Step 5.4: Input the multimodal fusion features into the second fully connected neural network to obtain the leaf area per plant of leafy vegetables.

[0104] Optionally, the second fully connected neural network has 3 layers, including 1 input layer, 1 hidden layer and 1 output layer, and each layer of neurons uses ReLU as the activation function.

[0105] Step 5.5: Output the plant height, diameter, canopy projection area and leaf area per plant of leafy vegetables.

[0106] In an optional embodiment, if Figure 2 As shown, before the RGB segmentation map, the depth segmentation map and the geometric parameters are input into the pre-trained leaf area regression model of leafy vegetables, the leaf area per plant of leafy vegetables is predicted, and the phenotypic parameters of the leafy vegetables are output, the method further includes:

[0107] According to steps 103 to 104, an RGB segmentation map, a depth segmentation map and geometric parameters of the leafy vegetable are obtained;

[0108] The leaf area regression model of leafy vegetables is trained based on RGB segmentation map, depth segmentation map and geometric parameters.

[0109] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application also provides a leafy vegetable phenotypic parameter measurement system based on binocular vision and deep learning, such as Figure 5 As shown, the system comprises:

[0110] An image acquisition module 501 is used to acquire an RGB left view and an RGB right view of the leafy vegetables using a binocular camera, and obtain a depth map of the leafy vegetables through stereo matching;

[0111] An image segmentation module 502 is used to perform semantic segmentation on the RGB left view using a pre-trained leafy vegetable canopy image segmentation model to obtain a segmentation mask;

[0112] The image segmentation module 502 is further used to segment the RGB left view and the depth map using the segmentation mask to obtain an RGB segmentation map and a depth segmentation map;

[0113] A geometric parameter acquisition module 503 is used to acquire geometric parameters of leafy vegetables based on the RGB segmentation map and the depth segmentation map;

[0114] The phenotypic parameter prediction module 504 is used to input the RGB segmentation map, the depth segmentation map and the geometric parameters into the pre-trained leaf area regression model of leafy vegetables, predict the leaf area per plant of leafy vegetables, and output the phenotypic parameters of the leafy vegetables; the phenotypic parameters include plant height, diameter, projected area and leaf area per plant.

[0115] In the embodiment of the present application, a binocular camera is used to collect leafy vegetable images in a factory environment with high-throughput data requirements, and combined with deep learning technology (leafy vegetable canopy image segmentation model and leafy vegetable leaf area regression model) for analysis, which can reduce monitoring costs and improve monitoring accuracy and efficiency. Among them, after non-destructively measuring the geometric parameters of leafy vegetables, the RGB information, depth information and geometric information of leafy vegetables are fused to predict the single plant leaf area of ​​leafy vegetables, and highly automated monitoring is achieved while avoiding damage to leafy vegetables. The embodiment of the present application can quickly monitor the production status of leafy vegetables, provide necessary leafy vegetable parameters for the control system and production personnel, facilitate plant factory production personnel to monitor leafy vegetable phenotypic characteristics in real time, more intuitively grasp the current growth state of leafy vegetables, and provide necessary data for the leafy vegetable production system, and finally realize low-cost and high-quality production of leafy vegetables in the factory.

[0116] The leafy vegetable phenotypic parameter measurement system based on binocular vision and deep learning provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0117] The leafy vegetable phenotypic parameter measurement system based on binocular vision and deep learning of the embodiment of the present application can execute the leafy vegetable phenotypic parameter measurement method based on binocular vision and deep learning provided in the embodiment of the present application, and its implementation principle is similar. The actions performed by each module and unit in the leafy vegetable phenotypic parameter measurement system based on binocular vision and deep learning in each embodiment of the present application correspond to the steps in the leafy vegetable phenotypic parameter measurement method based on binocular vision and deep learning in each embodiment of the present application. For the detailed functional description of each module of the leafy vegetable phenotypic parameter measurement system based on binocular vision and deep learning, please refer to the description of the corresponding leafy vegetable phenotypic parameter measurement method based on binocular vision and deep learning shown in the previous text, which will not be repeated here.

[0118] Based on the same principle as the method shown in the embodiment of the present application, the embodiment of the present application also provides an electronic device, which may include but is not limited to: a processor and a memory; a memory for storing a computer program; a processor for executing the leafy vegetable phenotypic parameter measurement method based on binocular vision and deep learning shown in any optional embodiment of the present application by calling a computer program. Compared with the prior art, the leafy vegetable phenotypic parameter measurement method based on binocular vision and deep learning provided in the present application uses a binocular camera to collect leafy vegetable images in a factory environment with high-throughput data requirements, and combines deep learning technology (leafy vegetable canopy image segmentation model and leafy vegetable leaf area regression model) for analysis, which can reduce monitoring costs and improve monitoring accuracy and efficiency. Among them, after the geometric parameters of the leafy vegetables are measured non-destructively, the RGB information, depth information and geometric information of the leafy vegetables are fused to predict the leaf area of ​​a single plant of the leafy vegetables, so that highly automated monitoring is achieved while avoiding damage to the leafy vegetables. The embodiments of the present application can quickly monitor the production status of leafy vegetables, provide the control system and production personnel with necessary leafy vegetable parameters, facilitate plant factory production personnel to monitor the phenotypic characteristics of leafy vegetables in real time, more intuitively grasp the current growth status of leafy vegetables, and provide necessary data for the leafy vegetable production system, ultimately achieving low-cost, high-quality production of leafy vegetables in the factory.

[0119] In an optional embodiment, an electronic device is also provided, such as Figure 6 As shown, Figure 6 The electronic device 600 shown may be a server, including: a processor 601 and a memory 603. The processor 601 and the memory 603 are connected, such as through a bus 602. Optionally, the electronic device 600 may further include a transceiver 604. It should be noted that in actual applications, the transceiver 604 is not limited to one, and the structure of the electronic device 600 does not constitute a limitation on the embodiments of the present application.

[0120] Processor 601 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 601 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0121] The bus 602 may include a path to transmit information between the above components. The bus 602 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 602 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0122] The memory 603 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0123] The memory 603 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 601. The processor 601 is used to execute the application code stored in the memory 603 to implement the contents shown in the above method embodiment.

[0124] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0125] The server provided in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, and this application does not limit this.

[0126] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0127] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0128] It should be noted that the computer-readable storage medium mentioned above in the present application can also be a computer-readable signal medium or a combination of a computer-readable storage medium and a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0129] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0130] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0131] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the leafy vegetable phenotypic parameter measurement method and system based on binocular vision and deep learning provided in the above-mentioned various optional implementations.

[0132] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0133] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0134] The modules described in the embodiments of the present application may be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the image acquisition module may also be described as "an image acquisition module for acquiring the RGB left view and RGB right view of leafy vegetables using a binocular camera and obtaining a depth map of the leafy vegetables through stereo matching".

[0135] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for measuring leafy vegetable phenotypic parameters based on binocular vision and deep learning, characterized in that: The method comprises: The RGB left view and RGB right view of the leafy vegetables are collected by using a binocular camera, and the depth map of the leafy vegetables is obtained by stereo matching; Using a pre-trained leafy vegetable canopy image segmentation model, semantically segmenting the RGB left view to obtain a segmentation mask; Segment the RGB left view and the depth map using the segmentation mask to obtain an RGB segmentation map and a depth segmentation map; Based on the RGB segmentation map and the depth segmentation map, obtaining geometric parameters of leafy vegetables; The RGB segmentation map, the depth segmentation map and the geometric parameters are input into a pre-trained leaf vegetable leaf area regression model to predict the leaf area per plant of the leaf vegetables, and output the phenotypic parameters of the leaf vegetables; the phenotypic parameters include plant height, diameter, projected area and leaf area per plant.

2. The method for measuring leafy vegetable phenotypic parameters based on binocular vision and deep learning according to claim 1, characterized in that: The method uses a binocular camera to collect an RGB left view and an RGB right view of the leafy vegetables, and obtains a depth map of the leafy vegetables through stereo matching, including: Calibrate the binocular camera to obtain its internal and external parameters and distortion coefficients; Use a binocular camera to shoot the canopy of leafy vegetables and obtain RGB left view and RGB right view of the leafy vegetables; Performing distortion correction on the RGB left view and the RGB right view using the internal and external parameters and the distortion coefficient; The Bouguet stereo correction algorithm is used to calibrate the distortion-corrected RGB left view and RGB right view to the same plane to obtain the corrected RGB left view and RGB right view; Taking the corrected RGB left view as a reference, stereo matching is performed on the corrected RGB left view and the RGB right view through a semi-global matching algorithm to obtain the disparity of the corrected RGB left view and the RGB right view; A depth map of the leafy vegetable is obtained according to the parallax calculation.

3. The method for measuring leafy vegetable phenotypic parameters based on binocular vision and deep learning according to claim 1, characterized in that: Before performing semantic segmentation on the RGB left view using the pre-trained leafy vegetable canopy image segmentation model to obtain a segmentation mask, the method further includes: Get the RGB left view and depth map of the leafy canopy; Measure the diameter, plant height and total leaf area of ​​each leafy vegetable plant in the leafy vegetable canopy; Labelme software was used to segment and label the RGB left view of the leafy vegetable canopy to obtain segmentation region labeling information; Creating labels based on the segmented area marking information and the diameter, plant height and total leaf area of ​​each leafy vegetable; Create a leafy vegetable dataset based on the RGB left view, depth map and labels of the leafy vegetable canopy; The leafy vegetable dataset is used to train a leafy vegetable canopy image segmentation model.

4. The method for measuring leafy vegetable phenotypic parameters based on binocular vision and deep learning according to claim 1, characterized in that: The leafy vegetable canopy image segmentation model uses the DeepLabV3+ network; The method of using a pre-trained leafy vegetable canopy image segmentation model to perform semantic segmentation on the RGB left view to obtain a segmentation mask includes: Input the RGB left view into the feature extraction module of the pre-trained DeepLabV3+ network for extraction to obtain high-level features and low-level features; The high-level features are upsampled and then fused with the low-level features to obtain RGB fusion features; The RGB fusion feature is convolved and up-sampled to obtain the segmentation mask.

5. The method for measuring leafy vegetable phenotypic parameters based on binocular vision and deep learning according to claim 1, characterized in that: The loss function of the leafy vegetable canopy image segmentation model is a cross entropy loss function.

6. The method for measuring leafy vegetable phenotypic parameters based on binocular vision and deep learning according to claim 1, characterized in that: The geometric parameters include plant height, diameter, projected area and volume; The step of obtaining geometric parameters of leafy vegetables based on the RGB segmentation map and the depth segmentation map includes: Determine the minimum depth value of the depth segmentation map, and obtain the plant height of the leafy vegetables by calculating the difference between the known distance from the binocular camera to the leafy vegetable cultivation platform and the minimum depth value; The RGB segmentation map and the depth segmentation map are combined to generate a 3D point cloud, the 3D point cloud is projected along the Z-axis direction, and the diameter of the leaf vegetable is obtained by computing the 2D point cloud; The 3D point cloud is voxelized, and the projection area of ​​the leafy vegetable is obtained by calculating the number of voxels and the area per unit voxel; A virtual point with a Z value of H1 is added to the 3D point cloud, and the 3D point cloud with the virtual point added is voxelized. The volume of the leafy vegetable is estimated by multiplying the distance from each voxel to the virtual point by the area of ​​the unit voxel; wherein the virtual point is located at the center of gravity of the point cloud in the XY plane.

7. The method for measuring leafy vegetable phenotypic parameters based on binocular vision and deep learning according to claim 6, characterized in that: The leafy vegetable leaf area regression model includes a first densely connected convolutional network, a second densely connected convolutional network, a first fully connected network, and a second fully connected network; The step of inputting the RGB segmentation map, the depth segmentation map and the geometric parameters into a pre-trained leaf vegetable leaf area regression model, predicting the leaf area per plant of the leaf vegetable, and outputting the phenotypic parameters of the leaf vegetable comprises: Inputting the RGB segmentation map and the depth segmentation map into the first densely connected convolutional network and the second densely connected convolutional network respectively to obtain corresponding RGB high-level features and depth high-level features; Combining the plant height, diameter, projected area and volume of the leafy vegetable into a four-dimensional vector, and inputting the four-dimensional vector into a first fully connected neural network to obtain geometric features of the leafy vegetable; The RGB high-level features, the depth high-level features and the geometric features are spliced ​​to obtain multimodal fusion features; Inputting the multimodal fusion features into a second fully connected neural network to obtain the leaf area per plant of leafy vegetables; Output plant height, diameter, canopy projection area and leaf area per plant of leafy vegetables.

8. A leafy vegetable phenotypic parameter measurement system based on binocular vision and deep learning, characterized in that: The system comprises: An image acquisition module is used to acquire an RGB left view and an RGB right view of the leafy vegetables using a binocular camera, and obtain a depth map of the leafy vegetables through stereo matching; An image segmentation module, used to perform semantic segmentation on the RGB left view using a pre-trained leafy vegetable canopy image segmentation model to obtain a segmentation mask; The image segmentation module is further used to segment the RGB left view and the depth map using the segmentation mask to obtain an RGB segmentation map and a depth segmentation map; A geometric parameter acquisition module, used to acquire geometric parameters of leafy vegetables based on the RGB segmentation map and the depth segmentation map; The phenotypic parameter prediction module is used to input the RGB segmentation map, the depth segmentation map and the geometric parameters into a pre-trained leaf vegetable leaf area regression model to predict the leaf area per plant of leaf vegetables and output the phenotypic parameters of the leaf vegetables; the phenotypic parameters include plant height, diameter, projected area and leaf area per plant.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images