Tomato growth environment cognition evaluation method

Abstract

The invention relates to a tomato growth environment cognitive evaluation method, which comprises the following steps: making a three-dimensional semantic map generation strategy based on deep learning and an octree map, including a semantic segmentation neural network model applied to a tomato growth environment and map construction nodes taking three-dimensional point cloud as input; objects in the tomato growth environment are classified into semantic categories; obtaining output information which is output by the last layer of the semantic segmentation model and contains semantic classification results of all pixel points; generating three-dimensional point cloud data carrying semantic information according to the semantic classification result information and the depth image information; performing outlier filtering on the three-dimensional point cloud data according to a statistical method; inputting the filtered three-dimensional semantic point cloud data into a map building node to generate a three-dimensional occupancy map; and processing the semantic information and the point cloud data according to agricultural task requirements to calculate the known degree of the tomato growth environment. The tomato growth environment three-dimensional semantic map construction and the known degree evaluation can be effectively realized.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision technology and agricultural planting environment map construction, and in particular to a tomato growth environment cognitive assessment method. Background Art

[0002] Tomato is one of the most popular greenhouse agricultural products today. It has the effects of promoting digestion and delaying aging. Its global production has tripled in the past four decades. In order to achieve efficient production and standardized management, open-air cultivation has gradually been replaced by greenhouse production, and the greenhouse tomato planting industry occupies an important share in my country's fruit and vegetable market. However, with the rise in labor costs in recent years, the greenhouse planting industry, where labor input accounts for more than 30%, is eager for further development of mechanization and automation of agricultural planting tasks. Automated and precise picking of greenhouse tomatoes is an important technology to promote the development of the greenhouse tomato planting industry. The picking work is not stable in an unstructured environment where it is difficult to obtain a full picture, and the understanding of the environment is low, and it is impossible to fully utilize the plant information for accurate collection. Therefore, correctly and comprehensively recognizing the environment is one of the key steps to achieve automated and precise collection of greenhouse tomatoes.

[0003] In the classic greenhouse picking environment, manual care greatly reduces the situation where the stems and leaves of the plants block the fruits to be picked. Most picking tasks do not need to specifically consider issues such as obstacle avoidance, but focus more on obtaining information such as the maturity, degree of occlusion, and spatial position of the tomato fruit. However, to increase the success rate of picking to a certain level, it is necessary to be able to handle a variety of picking situations. Therefore, how to pick tomatoes with a high degree of occlusion is a problem that must be solved. Summary of the invention

[0004] In order to meet the current information needs of agricultural robots for greenhouse tomato planting tasks, the present invention provides a tomato growth environment cognitive assessment method based on semantic segmentation and octree map for greenhouse tomato planting scenes, which can construct a three-dimensional semantic map in real time during greenhouse picking operations to guide the path planning of the picking robot. It can maintain good performance in real time, effectively deal with light changes, background noise, and slight shaking of plants in the greenhouse, and realize comprehensive real-time acquisition of three-dimensional semantic information of the tomato picking operation scene.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A method for cognitive assessment of a tomato growth environment comprises the following steps:

[0007] Step 1: Formulate a 3D semantic point cloud data generation strategy based on a deep learning semantic segmentation neural network, wherein the 3D semantic point cloud data generation strategy includes: a semantic segmentation model for a tomato growth environment and deep image point cloud processing;

[0008] Step 1.1, collecting multiple greenhouse tomato growth environment images, annotating the greenhouse tomato growth environment images with tomato fruits, tomato main stems, tomato peduncles and other objects as semantic categories, and obtaining the greenhouse tomato growth environment images containing annotation boxes, wherein the annotation boxes are pixel areas corresponding to the semantic categories; generating a tomato growth environment semantic segmentation dataset by using the greenhouse tomato growth environment images containing the annotation boxes according to a preset format;

[0009] Step 1.2, establishing a semantic segmentation neural network model, and training the semantic segmentation neural network model based on the tomato growth environment semantic segmentation dataset to obtain a semantic segmentation neural network model applied to a greenhouse tomato growth environment;

[0010] Step 1.3, input the two-dimensional RGB image to be detected that records the greenhouse tomato growth environment into the semantic segmentation neural network model applied to the greenhouse tomato growth environment for processing, and obtain the semantic category information S of all pixels in the two-dimensional RGB image to be detected:

[0011] S={S x，y |S x，y =k i ,x∈[0,w-1],y∈[0,h-1],i=1,2,...}

[0012] Among them, S x，y is the semantic category information of the pixel point with pixel coordinates (x, y) in the two-dimensional RGB image to be detected, k i is the i-th semantic category of the greenhouse tomato growth environment, w is the pixel width of the two-dimensional RGB image to be detected, and h is the pixel height of the image to be detected;

[0013] Step 1.4: Perform point cloud processing on the depth image corresponding to the two-dimensional RGB image to be detected to obtain three-dimensional point cloud data P of the greenhouse tomato growth environment corresponding to the two-dimensional RGB image to be detected:

[0014]

[0015] Among them, P j is the three-dimensional position information of the j-th pixel in the two-dimensional RGB image to be detected, d(x, y) is the depth information of the pixel in the depth image, (c x , c y ) is the principal point of the camera that shoots the two-dimensional RGB image to be detected, and f is the focal length of the depth camera that shoots the two-dimensional RGB image to be detected;

[0016] Step 2: Visualize the three-dimensional point cloud data P according to the semantic category information S;

[0017] Step 2.1, according to the semantic category information S, using a set of color schemes with significant visual contrast effect, adjusting the three-dimensional point cloud data P to three-dimensional point cloud data P′ corresponding to the semantic category information reflected by the color scheme;

[0018] Step 2.2: Based on a statistical method, the three-dimensional point cloud data P′ is subjected to an outlier filtering process to obtain three-dimensional point cloud data P″ with reduced noise;

[0019] Step 3: input the three-dimensional semantic point cloud data P″ into the map building node, and the map building node updates and maintains the octree-based grid map subnode according to the three-dimensional semantic point cloud data P″ and the spatial posture of its coordinate system in the world coordinate system at the current moment, and generates a three-dimensional occupancy map;

[0020] Step 4: Based on the semantic category information S, define and calculate the current observation known degree ρ:

[0021]

[0022] in, is the weight of the j-th pixel information of the two-dimensional RGB image to be detected, the pixel without semantic information is taken as 0, and the pixel with semantic information is taken as the inverse of the total number of pixels with semantic information, d(x, y) is the depth information of the j-th pixel of the two-dimensional RGB image to be detected in the depth image, d max is the maximum value of the depth information in the pixel having semantic information.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. The tomato growth environment cognitive assessment method proposed in the present invention processes the two-dimensional RGB-D image recording the greenhouse tomato growth environment to obtain three-dimensional point cloud data that reflects semantic categories with RGB colors. A semantic segmentation model is obtained by training a large number of tomato growth environment semantic segmentation data sets using deep learning. Only two-dimensional RGB images are used as input during training, and three-dimensional point cloud information is generated using semantic segmentation results and deep images, which reduces the amount of program calculation and has the characteristics of high accuracy, good real-time performance, and convenient post-processing. In the face of the actual environment of light changes, background noise, and slight shaking of plants in the greenhouse, it has good anti-interference ability, and can maintain a panoramic semantic map of the greenhouse tomato growth environment in real time, and can play a role when the RGB-D image is greatly interfered.

[0025] 2. The present invention solves the problem of assessing the degree of knowledge of the working environment based on the octree map and the imaging principle of a depth camera. The octree map of the present invention accumulates the results of multi-frame image processing to guide the picking robot to plan its task path in an incompletely observed greenhouse environment. The depth camera imaging principle is used to define and calculate the proportion of information obtained in a single frame image, making an attempt to solve the problem of assessing the degree of knowledge of the agricultural working environment.

[0026] 3. The present invention provides more comprehensive environmental information for realizing automated picking operations, which is conducive to guiding the picking robot to perform more reasonable path planning and provides the possibility for realizing precise picking. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0028] Figure 1 A schematic diagram of a flow chart of a tomato growth environment cognitive assessment method provided by an embodiment of the present invention;

[0029] Figure 2 The embodiment of the present invention provides Figure 1 A schematic diagram showing the time consumption of each link in the process shown;

[0030] Figure 3 An example of obtaining the known degree of greenhouse environment observation according to the method for calculating the known degree of environment observation provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] like Figure 1 As shown, a flow chart of a tomato growth environment cognitive assessment method provided by an embodiment of the present invention is provided to illustrate a tomato growth environment cognitive assessment method proposed by the present invention. Specifically, the method comprises the following steps:

[0033] S1. Initialize the trained semantic segmentation model.

[0034] Step 1: Formulate a three-dimensional semantic point cloud data generation strategy based on a deep learning semantic segmentation neural network, wherein the three-dimensional semantic point cloud data generation strategy includes: a semantic segmentation model for a tomato growth environment and deep image point cloud processing.

[0035] Step 1.1, collect multiple greenhouse tomato growth environment images and establish a tomato growth environment semantic segmentation dataset. Label the greenhouse tomato growth environment images with tomato fruits, tomato main stems, tomato pedicels and other objects as semantic categories to obtain images with labeled boxes, which are pixel areas of corresponding semantic categories; generate a tomato growth environment semantic segmentation dataset by using the greenhouse tomato growth environment images with labeled boxes in a preset format;

[0036] Step 1.2, establish a semantic segmentation neural network model, train the semantic segmentation neural network model based on the tomato growth environment semantic segmentation dataset, and obtain a semantic segmentation neural network model applied to the greenhouse tomato growth environment;

[0037] S2: Wait for image input.

[0038] S3. Determine whether there is real-time image input. If not, return to S2. If yes, execute S4.

[0039] S4. Segment the RGB image and obtain pixel semantic labels.

[0040] Step 1.3: Apply a two-dimensional image to be detected that records the growth environment of greenhouse tomatoes to the greenhouse tomato growth environment semantic segmentation neural network model for processing, and obtain the semantic category information S of all pixels in the two-dimensional RGB image to be detected:

[0041] S={S x，y |S x，y =k i ,x∈[0,w-1],y∈[0,h-1],i=1,2,...}

[0042] Among them, S x，y is the semantic category information of the pixel point with pixel coordinates (x, y) in the two-dimensional RGB image to be detected, k i is the i-th semantic category in the greenhouse tomato growth environment, w is the pixel width of the image to be detected, and h is the pixel height of the two-dimensional RGB image to be detected.

[0043] S5. Convert the depth image into point cloud data.

[0044] Step 1.4: Perform point cloud processing on the depth image corresponding to the two-dimensional image to be detected, and obtain the three-dimensional point cloud data P of the greenhouse tomato growth environment corresponding to the two-dimensional RGB image to be detected:

[0045]

[0046] Among them, P j is the 3D position information of the jth pixel in the 2D RGB image to be detected, d(x, y) is the depth information of the pixel in the depth image, (c x , c y ) is the principal point of the camera that shoots the two-dimensional RGB image to be detected, and f is the focal length of the depth camera that shoots the two-dimensional RGB image to be detected.

[0047] S6. Associate point cloud data with semantic labels.

[0048] Step 2: Visualize the three-dimensional point cloud data P according to the semantic category information S.

[0049] Step 2.1: According to the semantic category information S, a set of color schemes with significant visual contrast effects are used to adjust the three-dimensional point cloud data P to three-dimensional point cloud data P′ corresponding to the semantic category information reflected by the RGB colors in the color scheme.

[0050] In this embodiment, the tomato fruit, tomato main stem, tomato pedicel and other objects are respectively red, green, blue and purple, among which the tomato fruit, tomato main stem and tomato pedicel are the semantic category information corresponding to various colors.

[0051] S7. Point cloud outlier filtering.

[0052] Step 2.2: Based on statistical methods, outlier filtering is performed on the three-dimensional point cloud data P′ to obtain three-dimensional point cloud data P″ with reduced noise.

[0053] S8. Generate a three-dimensional occupancy map.

[0054] Step 3: Input the visualized three-dimensional semantic point cloud data P″ into the map building node. The map building node updates and maintains the octree-based grid map sub-node according to the three-dimensional semantic point cloud data P″ and the spatial posture of its coordinate system in the world coordinate system at that moment to generate a three-dimensional occupancy map.

[0055] S9. The degree to which the computing environment is known.

[0056] Step 4: Based on the semantic category information S, define and calculate the current observation known degree ρ:

[0057]

[0058] in, is the weight of the j-th pixel information of the two-dimensional RGB image to be detected. For pixels without semantic information, it is 0, and for pixels with semantic information, it is the inverse of the total number of pixels with semantic information. d(x, y) is the depth information of the j-th pixel of the two-dimensional RGB image to be detected in the depth image. max is the maximum value of depth information in pixels with semantic information.

[0059] Example:

[0060] (1) Establish a semantic segmentation model based on the deep learning MMSegmentation framework. Collect 173 greenhouse tomato growth environment images with a pixel size of 640×480 to establish a tomato growth environment semantic segmentation dataset. Annotate the greenhouse tomato growth environment images with tomato fruits, tomato main stems, tomato pedicels and other objects as semantic categories to obtain images with annotation boxes, which are pixel areas of corresponding semantic categories; generate a greenhouse tomato growth environment semantic segmentation dataset using the annotated greenhouse tomato growth environment images in the PASCAL-VOC2012 format.

[0061] Use data augmentation to expand the dataset, including random rotation, brightness adjustment, etc. Data augmentation makes the dataset larger and richer in content, which is conducive to enhancing the robustness of the model.

[0062] The greenhouse tomato growth environment semantic segmentation dataset is randomly divided into a training set and a test set in a ratio of 4:1. The training set is used to train the network model, and the test set is used to test the quality of the model. After multiple trainings, the K-Net network model with the best quality is selected.

[0063] Input a two-dimensional image to be detected that records the growth environment of greenhouse tomatoes into the deployed K-Net semantic segmentation neural network model to obtain the semantic category information S of all pixels in the RGB image:

[0064] S={S x，y |S x，y =k i ,x∈E[0,w-1],y∈[0,h-1],i=1,2,...}

[0065] Among them, S x，y is the semantic category information of the pixel point with pixel coordinates (x, y) in the image to be detected, k i is the i-th semantic category of the greenhouse tomato growth environment, w is the pixel width of the image to be detected, and h is the pixel height of the image to be detected.

[0066] The depth image corresponding to the two-dimensional image is processed into a point cloud to obtain the three-dimensional point cloud data P of the greenhouse tomato growth environment:

[0067]

[0068] Among them, P j is the three-dimensional position information of the j-th pixel, d(x, y) is the depth information of the pixel in the depth image, (c x , c y ) is the principal point of the camera that takes the image to be detected, and f is the focal length of the depth camera that takes the image to be detected.

[0069] (2) Visualize the three-dimensional point cloud data P according to the semantic category information S obtained in step 1. According to the semantic category information S obtained in step 1, use a set of color schemes with significant visual contrast effect to adjust the three-dimensional point cloud data P to three-dimensional point cloud data P′ that reflects the semantic category in RGB color. In this embodiment, the tomato fruit, tomato main stem, tomato pedicel and other objects are red, green, blue and purple, respectively. Use statistical methods to perform outlier filtering on the three-dimensional point cloud data P / to obtain three-dimensional point cloud data P″ with reduced noise.

[0070] (3) The visualized 3D semantic point cloud data P″ is input into the OctoMap node in the ROS (Robot Operating System) framework to generate a 3D occupancy map. The map creation node updates and maintains the OctoMap node based on the 3D semantic point cloud data P″ and the spatial posture of its coordinate system in the world coordinate system at that moment. The Line_profiler tool is used to monitor the program and obtain the following Figure 2 The time consumption of each link of the system is shown.

[0071] (4) Based on the depth image and the semantic category information S obtained in step 1, we can calculate Figure 3 The current observational knowledge level ρ is shown as:

[0072]

[0073] in, is the weight of the j-th pixel information. For pixels without semantic information, it is 0. For pixels with semantic information, it is the inverse of the total number of pixels with semantic information. d(x, y) is the depth information of the j-th pixel in the depth image. max is the maximum value of depth information in pixels with semantic information.

[0074] Figure 3 The tomato fruit, tomato stem, tomato pedicel and other objects are marked in red, green, blue and purple respectively. The observation degree ρ is Figure 2 The “Environment Known: 84.52%” indicator is shown in the upper left.

[0075] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0076] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0077] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for cognitive assessment of tomato growth environment, characterized in that: The following steps are involved: Step 1: Formulate a 3D semantic point cloud data generation strategy based on a deep learning semantic segmentation neural network, wherein the 3D semantic point cloud data generation strategy includes: a semantic segmentation model for a tomato growth environment and deep image point cloud processing; Step 1.1, collecting multiple greenhouse tomato growth environment images, annotating the greenhouse tomato growth environment images with tomato fruits, tomato main stems, tomato peduncles and other objects as semantic categories, and obtaining the greenhouse tomato growth environment images containing annotation boxes, wherein the annotation boxes are pixel areas corresponding to the semantic categories; generating a tomato growth environment semantic segmentation dataset by using the greenhouse tomato growth environment images containing the annotation boxes according to a preset format; Step 1.2, establishing a semantic segmentation neural network model, and training the semantic segmentation neural network model based on the tomato growth environment semantic segmentation dataset to obtain a semantic segmentation neural network model applied to a greenhouse tomato growth environment; Step 1.3, input the two-dimensional RGB image to be detected that records the greenhouse tomato growth environment into the semantic segmentation neural network model applied to the greenhouse tomato growth environment for processing, and obtain the semantic category information S of all pixels in the two-dimensional RGB image to be detected: S={S x，y |S x，y =k i ,x∈[0,w-1],y∈[0,h-1],i=1,2,...} Among them, S x，y is the semantic category information of the pixel point with pixel coordinates (x, y) in the two-dimensional RGB image to be detected, k i is the i-th semantic category in the greenhouse tomato growth environment, w is the pixel width of the two-dimensional RGB image to be detected, and h is the pixel height of the image to be detected; Step 1.4: Perform point cloud processing on the depth image corresponding to the two-dimensional RGB image to be detected to obtain three-dimensional point cloud data P of the greenhouse tomato growth environment corresponding to the two-dimensional RGB image to be detected: Among them, P j is the three-dimensional position information of the j-th pixel in the two-dimensional RGB image to be detected, d(x, y) is the depth information of the pixel in the depth image, (c x , c y ) is the principal point of the camera that shoots the two-dimensional RGB image to be detected, and f is the focal length of the depth camera that shoots the two-dimensional RGB image to be detected; Step 2: Visualize the three-dimensional point cloud data P according to the semantic category information S; Step 2.1, according to the semantic category information S, using a set of color schemes with significant visual contrast effect, adjusting the three-dimensional point cloud data P to three-dimensional point cloud data P′ corresponding to the semantic category information reflected by the color scheme; Step 2.2: Based on a statistical method, perform outlier filtering on the three-dimensional point cloud data P′ to obtain three-dimensional point cloud data P″ with reduced noise; Step 3: input the three-dimensional semantic point cloud data P″ into the map building node, and the map building node updates and maintains the octree-based grid map subnode according to the three-dimensional semantic point cloud data P″ and the spatial posture of its coordinate system in the world coordinate system at the current moment, and generates a three-dimensional occupancy map; Step 4: Based on the semantic category information S, define and calculate the current observation known degree ρ: in, is the weight of the j-th pixel information of the two-dimensional RGB image to be detected, the pixel without semantic information is taken as 0, and the pixel with semantic information is taken as the inverse of the total number of pixels with semantic information, d(x, y) is the depth information of the j-th pixel of the two-dimensional RGB image to be detected in the depth image, d max is the maximum value of the depth information in the pixel having semantic information.

2. The tomato growth environment cognitive assessment method according to claim 1, characterized in that: The step 2 comprises: The two-dimensional RGB-D image to be detected in the greenhouse tomato growth environment is input into the semantic segmentation neural network model applied to the greenhouse tomato growth environment for processing to obtain three-dimensional point cloud data reflecting semantic categories with the color scheme, and the semantic categories are semantic categories for dividing objects in the greenhouse tomato growth environment according to greenhouse agricultural task requirements.

3. The tomato growth environment cognitive assessment method according to claim 1, characterized in that: The step 3 also includes: fusing multiple frames of the three-dimensional semantic point cloud data P″ to construct and maintain a three-dimensional semantic map of the tomato growth environment.

4. The tomato growth environment cognitive assessment method according to claim 1, characterized in that: Step 4 includes: estimating the known degree of the current tomato growth environment based on the binocular ranging principle.

Images