A method for cognitive evaluation of a tomato growing environment

Abstract

The application relates to a tomato growth environment cognition evaluation method, which comprises the following steps: formulating a three-dimensional semantic map generation strategy based on deep learning and an octree map, including a semantic segmentation neural network model applied to a tomato growth environment and a map construction node taking a three-dimensional point cloud as input; dividing objects in the tomato growth environment into semantic categories; obtaining output information of a semantic classification result containing all pixel points output by a last layer of the semantic segmentation model; generating three-dimensional point cloud data carrying semantic information according to the semantic classification result information and depth image information; performing outlier filtering on the three-dimensional point cloud data according to a statistical method; inputting the filtered three-dimensional semantic point cloud data into the map construction node to generate a three-dimensional occupancy map; and processing the semantic information and the point cloud data according to agricultural task requirements to calculate the known degree of the tomato growth environment, so that the three-dimensional semantic map construction and the known degree evaluation of the tomato growth environment can be effectively realized.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision technology and agricultural planting environment map construction, and in particular to a tomato growth environment cognitive assessment method. Background Art

[0002] Tomatoes are one of today's most popular greenhouse agricultural products, known for their digestive benefits and anti-aging properties. Global production has tripled over the past four decades. To achieve efficient production and standardized management, open-air cultivation has gradually been replaced by greenhouse production, and the greenhouse tomato cultivation industry now holds a significant share of my country's fruit and vegetable market. However, with rising labor costs in recent years, the greenhouse cultivation industry, where manual labor accounts for over 30%, urgently needs further mechanization and automation of agricultural planting tasks. Automated and precise harvesting of greenhouse tomatoes is a key technology driving the development of the greenhouse tomato cultivation industry. However, harvesting in unstructured environments, where a comprehensive view is difficult to obtain, is unstable and requires limited understanding of the environment, making it difficult to fully utilize plant information for accurate harvesting. Therefore, a comprehensive and accurate understanding of the environment is a key step in achieving automated and precise harvesting of greenhouse tomatoes.

[0003] In a typical greenhouse harvesting environment, manual tending significantly reduces obstruction of the fruit by plant stems and leaves. Most harvesting tasks don't require specific considerations for obstacle avoidance, focusing instead on acquiring information about the tomato fruit's maturity, degree of obstruction, and spatial position. However, to achieve a high level of harvesting success, the ability to handle diverse harvesting scenarios is essential. Therefore, the challenge of harvesting tomatoes with high levels of obstruction remains a crucial issue. Summary of the Invention

[0004] To meet the current information needs of agricultural robots for greenhouse tomato cultivation tasks, this paper presents a tomato growing environment cognitive assessment method based on semantic segmentation and octree mapping for greenhouse tomato cultivation scenarios. This method constructs a 3D semantic map in real time during greenhouse harvesting operations to guide the path planning of harvesting robots. This method maintains excellent real-time performance, effectively addressing greenhouse lighting changes, background noise, and slight plant movement, enabling comprehensive, real-time acquisition of 3D semantic information for tomato harvesting scenarios.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A method for cognitive assessment of a tomato growing environment comprises the following steps:

[0007] Step 1: Develop a 3D semantic point cloud data generation strategy based on a deep learning semantic segmentation neural network. The 3D semantic point cloud data generation strategy includes: a semantic segmentation model for the tomato growth environment and deep image point cloud processing;

[0008] Step 1.1, collecting multiple greenhouse tomato growth environment images, annotating the greenhouse tomato growth environment images with tomato fruits, tomato main stems, tomato pedicels, and other objects as semantic categories, and obtaining the greenhouse tomato growth environment images containing annotated boxes, where the annotated boxes are pixel regions corresponding to the semantic categories; generating a tomato growth environment semantic segmentation dataset by formatting the greenhouse tomato growth environment images containing the annotated boxes;

[0009] Step 1.2: Establish a semantic segmentation neural network model, and train the semantic segmentation neural network model based on the tomato growth environment semantic segmentation dataset to obtain a semantic segmentation neural network model for greenhouse tomato growth environment;

[0010] Step 1.3: Input the two-dimensional RGB image to be detected that records the greenhouse tomato growth environment into the semantic segmentation neural network model for greenhouse tomato growth environment for processing, and obtain the semantic category information S of all pixels in the two-dimensional RGB image to be detected:

[0011] S={S x，y |S x，y =k i ,x∈[0,w-1],y∈[0,h-1],i=1,2,...}

[0012] Among them, S x，y is the semantic category information of the pixel point with pixel coordinates (x, y) in the two-dimensional RGB image to be detected, k i is the i-th semantic category of the greenhouse tomato growth environment, w is the pixel width of the two-dimensional RGB image to be detected, and h is the pixel height of the image to be detected;

[0013] Step 1.4: Perform point cloud processing on the depth image corresponding to the two-dimensional RGB image to be detected to obtain three-dimensional point cloud data P of the greenhouse tomato growth environment corresponding to the two-dimensional RGB image to be detected:

[0014]

[0015] Among them, P j is the three-dimensional position information of the j-th pixel in the two-dimensional RGB image to be detected, d(x, y) is the depth information of the pixel in the depth image, (c x , c y ) is the principal point of the camera that shoots the two-dimensional RGB image to be detected, and f is the focal length of the depth camera that shoots the two-dimensional RGB image to be detected;

[0016] Step 2: Visualize the three-dimensional point cloud data P according to the semantic category information S;

[0017] Step 2.1: Based on the semantic category information S, a set of color schemes with significant visual contrast is used to adjust the three-dimensional point cloud data P to three-dimensional point cloud data P′ corresponding to the semantic category information reflected by the color scheme;

[0018] Step 2.2: Based on a statistical method, perform outlier filtering on the three-dimensional point cloud data P′ to obtain three-dimensional point cloud data P″ with reduced noise;

[0019] Step 3: Input the 3D semantic point cloud data P″ into the map building node. The map building node updates and maintains the octree-based grid map child node based on the 3D semantic point cloud data P″ and the spatial posture of its coordinate system in the world coordinate system at the current moment to generate a 3D occupancy map.

[0020] Step 4: Based on the semantic category information S, define and calculate the current observation known degree ρ:

[0021]

[0022] in, is the weight of the j-th pixel information of the two-dimensional RGB image to be detected, and takes 0 for pixels without semantic information and takes the inverse of the total number of pixels with semantic information for pixels with semantic information. d(x, y) is the depth information of the j-th pixel of the two-dimensional RGB image to be detected in the depth image, and d max is the maximum value of the depth information in the pixel with semantic information.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1. The tomato growth environment cognitive assessment method proposed in the present invention processes two-dimensional RGB-D images recording the greenhouse tomato growth environment to obtain three-dimensional point cloud data that reflects semantic categories in RGB colors. Deep learning is used to train a large number of tomato growth environment semantic segmentation data sets to obtain a semantic segmentation model. During the training, only two-dimensional RGB images are used as input, and three-dimensional point cloud information is generated using the semantic segmentation results and depth images, which reduces the program's computational complexity and has the characteristics of high accuracy, good real-time performance, and convenient post-processing. It has good anti-interference ability in the face of the actual environment of light changes, background noise, and slight shaking of plants in the greenhouse, and can maintain a panoramic semantic map of the greenhouse tomato growth environment in real time, and can play a role when the RGB-D image is subject to significant interference.

[0025] 2. This invention addresses the problem of assessing the degree of knowledge of the working environment based on octree maps and depth camera imaging principles. The octree map of this invention accumulates the results of multi-frame image processing to guide a harvesting robot in task path planning in an incompletely observed greenhouse environment. The octree map utilizes the principles of depth camera imaging to define and calculate the proportion of information obtained in a single frame, attempting to address the problem of assessing the degree of knowledge of the agricultural working environment.

[0026] 3. The present invention provides more comprehensive environmental information for realizing automated picking operations, which is conducive to guiding the picking robot to perform more reasonable path planning and makes it possible to achieve precise picking. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0028] Figure 1 A schematic diagram of a flow chart of a method for cognitively evaluating a tomato growth environment provided by an embodiment of the present invention;

[0029] Figure 2 The embodiment of the present invention provides Figure 1 A schematic diagram showing the time consumption of each link in the process shown;

[0030] Figure 3 This is a case of obtaining the known degree of greenhouse environment observation according to the environmental observation known degree calculation method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0032] like Figure 1 FIG. 1 is a flow chart of a method for cognitively evaluating a tomato growing environment provided by an embodiment of the present invention. The method for cognitively evaluating a tomato growing environment provided by the present invention is described. Specifically, the method includes the following steps:

[0033] S1. Initialize the trained semantic segmentation model.

[0034] Step 1: Develop a 3D semantic point cloud data generation strategy based on a deep learning semantic segmentation neural network. The 3D semantic point cloud data generation strategy includes: a semantic segmentation model for the tomato growth environment and deep image point cloud processing.

[0035] Step 1.1: Collect multiple images of greenhouse tomato growth environments and create a tomato growth environment semantic segmentation dataset. Label the greenhouse tomato growth environment images using tomato fruits, tomato stems, tomato peduncles, and other objects as semantic categories to obtain images containing labeled boxes, where the labeled boxes are pixel regions of corresponding semantic categories. Generate a tomato growth environment semantic segmentation dataset by formatting the greenhouse tomato growth environment images containing the labeled boxes.

[0036] Step 1.2: Establish a semantic segmentation neural network model. Based on the tomato growth environment semantic segmentation dataset, train the semantic segmentation neural network model to obtain a semantic segmentation neural network model for greenhouse tomato growth environment.

[0037] S2. Wait for image input.

[0038] S3. Determine whether there is real-time image input. If not, return to S2. If yes, execute S4.

[0039] S4. Segment the RGB image and obtain pixel semantic labels.

[0040] Step 1.3: Apply a two-dimensional image to be detected that records the greenhouse tomato growth environment to the greenhouse tomato growth environment semantic segmentation neural network model for processing, and obtain the semantic category information S of all pixels in the two-dimensional RGB image to be detected:

[0041] S={S x，y |S x，y =k i ,x∈[0,w-1],y∈[0,h-1],i=1,2,...}

[0042] Among them, S x，y is the semantic category information of the pixel point with pixel coordinates (x, y) in the two-dimensional RGB image to be detected, k i is the i-th semantic category in the greenhouse tomato growth environment, w is the pixel width of the image to be detected, and h is the pixel height of the two-dimensional RGB image to be detected.

[0043] S5. Convert the depth image into point cloud data.

[0044] Step 1.4: Perform point cloud processing on the depth image corresponding to the two-dimensional image to be detected, and obtain the three-dimensional point cloud data P of the greenhouse tomato growth environment corresponding to the two-dimensional RGB image to be detected:

[0045]

[0046] Among them, P j is the three-dimensional position information of the j-th pixel in the two-dimensional RGB image to be detected, d(x, y) is the depth information of the pixel in the depth image, (c x , c y ) is the principal point of the camera that shoots the two-dimensional RGB image to be detected, and f is the focal length of the depth camera that shoots the two-dimensional RGB image to be detected.

[0047] S6. Associate point cloud data and semantic labels.

[0048] Step 2: Visualize the three-dimensional point cloud data P according to the semantic category information S.

[0049] Step 2.1: Based on the semantic category information S, a set of color schemes with significant visual contrast effects are used to adjust the three-dimensional point cloud data P to three-dimensional point cloud data P′ corresponding to the semantic category information reflected by the RGB colors in the color scheme.

[0050] In this embodiment, the tomato fruit, tomato main stem, tomato pedicel and other objects are respectively red, green, blue and purple, among which the tomato fruit, tomato main stem and tomato pedicel are the semantic category information corresponding to various colors.

[0051] S7. Point cloud outlier filtering.

[0052] Step 2.2: Based on statistical methods, perform outlier filtering on the three-dimensional point cloud data P′ to obtain three-dimensional point cloud data P″ with reduced noise.

[0053] S8. Generate a three-dimensional occupancy map.

[0054] Step 3: Input the visualized 3D semantic point cloud data P″ into the map building node. The map building node updates and maintains the octree-based grid map subnode based on the 3D semantic point cloud data P″ and the spatial posture of its coordinate system in the world coordinate system at that moment to generate a 3D occupancy map.

[0055] S9. The degree of knowledge of the computing environment.

[0056] Step 4: Based on the semantic category information S, define and calculate the current observation knowledge level ρ:

[0057]

[0058] in, is the weight of the j-th pixel information of the two-dimensional RGB image to be detected. For pixels without semantic information, it is 0, and for pixels with semantic information, it is the inverse of the total number of pixels with semantic information. d(x, y) is the depth information of the j-th pixel of the two-dimensional RGB image to be detected in the depth image. max is the maximum value of depth information among pixels with semantic information.

[0059] Example:

[0060] (1) Establish a semantic segmentation model based on the deep learning MMSegmentation framework. Collect 173 greenhouse tomato growth environment images with a pixel size of 640×480 to establish a tomato growth environment semantic segmentation dataset. Annotate the greenhouse tomato growth environment images with tomato fruits, tomato main stems, tomato pedicels, and other objects as semantic categories to obtain images with annotated boxes. The annotated boxes are the pixel areas of the corresponding semantic categories. The annotated greenhouse tomato growth environment images are converted into a greenhouse tomato growth environment semantic segmentation dataset in the PASCAL-VOC2012 format.

[0061] Use data augmentation to expand the dataset, including random rotation, brightness adjustment, etc. Data augmentation makes the dataset larger and richer in content, which helps to enhance the robustness of the model.

[0062] The greenhouse tomato growth environment semantic segmentation dataset was randomly split into a training set and a test set in a 4:1 ratio. The training set was used to train the network model, and the test set was used to verify the model's quality. After multiple training runs, the K-Net network model with the best quality was selected.

[0063] Input a two-dimensional image to be detected, which records the growth environment of greenhouse tomatoes, into the deployed K-Net semantic segmentation neural network model to obtain the semantic category information S of all pixels in the RGB image:

[0064] S={S x，y |S x，y =k i ,x∈E[0,w-1],y∈[0,h-1],i=1,2,...}

[0065] Among them, S x，y is the semantic category information of the pixel point with pixel coordinates (x, y) in the image to be detected, k i is the i-th semantic category of the greenhouse tomato growth environment, w is the pixel width of the image to be detected, and h is the pixel height of the image to be detected.

[0066] The depth image corresponding to the two-dimensional image is processed into a point cloud to obtain the three-dimensional point cloud data P of the greenhouse tomato growth environment:

[0067]

[0068] Among them, P j is the three-dimensional position information of the j-th pixel, d(x, y) is the depth information of the pixel in the depth image, (c x , c y ) is the principal point of the camera that takes the image to be detected, and f is the focal length of the depth camera that takes the image to be detected.

[0069] (2) Visualize the three-dimensional point cloud data P based on the semantic category information S obtained in step 1. Based on the semantic category information S obtained in step 1, adjust the three-dimensional point cloud data P to three-dimensional point cloud data P′ that reflects the semantic category using RGB colors using a set of color schemes with significant visual contrast. In this embodiment, the tomato fruit, tomato main stem, tomato pedicel, and other objects are colored red, green, blue, and purple, respectively. Statistical methods are used to perform outlier filtering on the three-dimensional point cloud data P′ to obtain three-dimensional point cloud data P″ with reduced noise.

[0070] (3) The visualized 3D semantic point cloud data P″ is input into the OctoMap node under the ROS (Robot Operating System) framework to generate a 3D occupancy map. The map creation node updates and maintains the OctoMap node based on the 3D semantic point cloud data P″ and the spatial posture of its coordinate system in the world coordinate system at that moment. The program is monitored using the Line_profiler tool and the following is obtained: Figure 2 The time consumption of each link of the system is shown.

[0071] (4) Based on the depth image and the semantic category information S obtained in step 1, the following is calculated: Figure 3 The degree of knowledge of the current observations is shown as:

[0072]

[0073] in, is the weight of the j-th pixel information. For pixels without semantic information, it is 0. For pixels with semantic information, it is the inverse of the total number of pixels with semantic information. d(x, y) is the depth information of the j-th pixel in the depth image. max is the maximum value of depth information among pixels with semantic information.

[0074] Figure 3 The tomato fruit, tomato stem, tomato pedicel and other objects are marked in red, green, blue and purple respectively. The observation degree ρ is Figure 2 The "Environment Known: 84.52%" indicator is shown in the upper left corner.

[0075] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0076] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0077] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for cognitive assessment of tomato growth environment, characterized in that: The following steps are involved: Step 1: Develop a 3D semantic point cloud data generation strategy based on a deep learning semantic segmentation neural network. The 3D semantic point cloud data generation strategy includes: a semantic segmentation model for the tomato growth environment and deep image point cloud processing; Step 1.1, collecting multiple greenhouse tomato growth environment images, annotating the greenhouse tomato growth environment images with tomato fruits, tomato main stems, tomato pedicels, and other objects as semantic categories, and obtaining the greenhouse tomato growth environment images containing annotated boxes, where the annotated boxes are pixel regions corresponding to the semantic categories; generating a tomato growth environment semantic segmentation dataset by formatting the greenhouse tomato growth environment images containing the annotated boxes; Step 1.2: Establish a semantic segmentation neural network model, and train the semantic segmentation neural network model based on the tomato growth environment semantic segmentation dataset to obtain a semantic segmentation neural network model for greenhouse tomato growth environment; Step 1.3: Input the two-dimensional RGB image to be detected that records the greenhouse tomato growth environment into the semantic segmentation neural network model for greenhouse tomato growth environment for processing, and obtain the semantic category information S of all pixels in the two-dimensional RGB image to be detected: S={S x，y |S x，y =k i ,x∈[0,w-1],y∈[0,h-1],i=1,2,...} Among them, S x，y is the semantic category information of the pixel point with pixel coordinates (x, y) in the two-dimensional RGB image to be detected, k i is the i-th semantic category in the greenhouse tomato growth environment, w is the pixel width of the two-dimensional RGB image to be detected, and h is the pixel height of the image to be detected; Step 1.4: Perform point cloud processing on the depth image corresponding to the two-dimensional RGB image to be detected to obtain three-dimensional point cloud data P of the greenhouse tomato growth environment corresponding to the two-dimensional RGB image to be detected: Among them, P j is the three-dimensional position information of the j-th pixel in the two-dimensional RGB image to be detected, d(x, y) is the depth information of the pixel in the depth image, (c x , c y ) is the principal point of the camera that shoots the two-dimensional RGB image to be detected, and f is the focal length of the depth camera that shoots the two-dimensional RGB image to be detected; Step 2: Visualize the three-dimensional point cloud data P according to the semantic category information S; Step 2.1: Based on the semantic category information S, a set of color schemes with significant visual contrast is used to adjust the three-dimensional point cloud data P to three-dimensional point cloud data P′ corresponding to the semantic category information reflected by the color scheme; Step 2.2: Based on a statistical method, perform outlier filtering on the three-dimensional point cloud data P′ to obtain three-dimensional point cloud data P″ with reduced noise; Step 3: Input the 3D semantic point cloud data P″ into the map building node. The map building node updates and maintains the octree-based grid map child node based on the 3D semantic point cloud data P″ and the spatial posture of its coordinate system in the world coordinate system at the current moment to generate a 3D occupancy map. Step 4: Based on the semantic category information S, define and calculate the current observation known degree ρ: in, is the weight of the j-th pixel information of the two-dimensional RGB image to be detected, and takes 0 for pixels without semantic information and takes the inverse of the total number of pixels with semantic information for pixels with semantic information. d(x, y) is the depth information of the j-th pixel of the two-dimensional RGB image to be detected in the depth image, and d max is the maximum value of the depth information in the pixel with semantic information.

2. The tomato growth environment cognitive assessment method according to claim 1, characterized in that: The step 2 includes: The two-dimensional RGB-D image to be detected of the greenhouse tomato growth environment is input into the semantic segmentation neural network model applied to the greenhouse tomato growth environment for processing to obtain three-dimensional point cloud data reflecting semantic categories using the color scheme. The semantic categories are semantic categories for dividing objects in the greenhouse tomato growth environment according to greenhouse agricultural task requirements.

3. The tomato growth environment cognitive assessment method according to claim 1, characterized in that: The step 3 further includes: fusing multiple frames of the three-dimensional semantic point cloud data P″ to construct and maintain a three-dimensional semantic map of the tomato growth environment.

4. The tomato growth environment cognitive assessment method according to claim 1, characterized in that: Step 4 includes: estimating the known degree of the current tomato growth environment based on the binocular ranging principle.

Images