Tomato three-dimensional posture judgment method based on deep learning and classification

Abstract

The application relates to a tomato three-dimensional posture judgment method based on deep learning and classification. The method comprises the following steps: establishing a two-stage cascaded network based on deep learning, including a first-stage target detection model and a second-stage target classification model; dividing a tomato fruit three-dimensional posture space range into a plurality of intervals, and establishing a tomato three-dimensional posture classification standard; obtaining first-stage output information of a last layer output of the first-stage target detection network, which contains all detection target occlusion categories and region information; obtaining second-stage output information of a last layer output of the second-stage target classification network, which is posture category confidence information of all tomato fruits; and selecting a category with the highest posture category confidence as the posture category of a target tomato fruit. The application only takes 2D pictures as input, does not need three-dimensional point clouds, has fast recognition and detection speed, a simple process, and can effectively realize tomato fruit three-dimensional posture detection.

Description

Technical Field

[0001] This invention relates to the fields of computer vision technology and 3D pose detection, specifically to a tomato-based 3D pose determination method based on deep learning and classification. Background Technology

[0002] Tomatoes are an important economic crop worldwide, prized for their rich nutrition, good taste, and ease of cultivation. Currently, tomato harvesting in my country is primarily done manually, resulting in high costs and accounting for approximately 30% of the total harvesting process in terms of time and expense. Therefore, automated harvesting is crucial for efficient tomato production and cost reduction. The more information a harvesting robot acquires during tomato harvesting, the higher its success rate, bringing it closer to achieving fully automated harvesting. Three-dimensional posture is one of the most important harvesting parameters. Obtaining the tomato's three-dimensional posture helps in selecting the appropriate harvesting posture, reducing interference between the end effector and the tomato stem, and improving the harvesting success rate.

[0003] Common harvesting information includes fruit location, ripeness, occlusion level, and pose. Currently, harvesting robots often only acquire one of these three categories of information, neglecting the fruit's 3D pose. Mainstream target 3D pose estimation algorithms primarily rely on regression calculations to determine the target's specific 3D pose, and utilize deep learning and point cloud processing in their computational methods. Summary of the Invention

[0004] To address the current need for precise tomato harvesting robots to determine the 3D posture of tomatoes, this invention aims to provide a method for judging the 3D posture of tomatoes based on deep learning and classification. This method can acquire the 3D posture of tomatoes in real time during greenhouse harvesting operations. It requires fast recognition and detection speed, effectively handling complex lighting changes, shading, and background interference in greenhouses, and achieving comprehensive real-time acquisition of tomato harvesting information.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A method for determining the 3D pose of a tomato plant based on deep learning and classification, comprising the following steps:

[0007] Step 1: Establish a two-level cascaded network model based on deep learning. The two-level cascaded network model includes a first-level object detection model and a second-level object classification model.

[0008] Step 1.1: Collect multiple greenhouse tomato images, establish a tomato target detection dataset, and perform first-level annotation on the greenhouse tomato images with tomato fruits as targets to obtain greenhouse tomato images containing annotation boxes. Each annotation box contains a single tomato fruit. The information of the first-level annotation includes the occlusion category and region information of the tomato fruit. The occlusion category indicates whether the tomato fruit within the annotation box is occluded. The region information includes the position and size of the tomato fruit. Generate the tomato target detection dataset from the annotated greenhouse tomato images according to a certain format.

[0009] Step 1.2: Establish a first-level object detection network model. Train the first-level object detection model using the tomato object detection dataset obtained in Step 1.1.

[0010] Step 1.3: Output the image within the labeled box according to the annotation information of the tomato target detection dataset described in Step 1.1. Use the screenshot of the tomato fruit as the tomato pose classification dataset of the second-level target classification network. Establish a three-dimensional pose classification standard for the tomato fruit as needed. Divide the three-dimensional pose space of the tomato fruit into multiple intervals to obtain multiple pose categories. Classify the three-dimensional pose of the tomato fruit according to the interval where the axis of the tomato fruit is located. Use the three-dimensional pose category information of the tomato fruit as the target to perform secondary annotation on each image in the tomato pose classification dataset to obtain the tomato pose classification dataset. The information of the secondary annotation is the pose category information of the tomato fruit, which represents the pose of the tomato fruit relative to the end effector.

[0011] Step 1.4: Establish a second-level target classification model. Train the second-level target classification model using the tomato pose classification dataset obtained in Step 1.3.

[0012] Step 2: Input an image containing one or more of the tomato fruits into the first-level object detection model obtained in Step 1, and obtain the first-level output information L1, which contains the occlusion category and region information of all tomato fruits, from the last layer of the first-level object detection network.

[0013] L1={li|li=(cls,x,y,width,height),i=1,2…}

[0014] Where li represents the i-th tomato fruit, cls represents the category of the area where the i-th tomato fruit is located, with a value of 1 indicating that the tomato fruit is not obscured, and a value of 0 indicating that the tomato fruit is obscured; x represents the x-coordinate of the center point of the area where the i-th tomato fruit is located, y represents the y-coordinate of the center point of the area where the i-th tomato fruit is located, width represents the width of the area where the i-th tomato fruit is located, and height represents the height of the area where the i-th tomato fruit is located.

[0015] Step 3: Crop the unoccluded tomato fruit area from Step 2 (cls=1), normalize it to a 128×128 tomato image, and input it into the second-level target classification model obtained in Step 1. Obtain the second-level output information L2 of the pose category confidence information of all tomato fruits from the last layer of the second-level target classification model.

[0016] L2={li|li=(cls1, cls2......, clsj), i=1,2......; j=1,2......}

[0017] Where li represents the i-th tomato fruit, j represents the number of categories, and cls1, cls2, ..., clsj represent the confidence scores of each pose category.

[0018] Step 4: Select the posture category with the highest posture category confidence from the posture category confidence information of the target tomato fruit as the posture category of the target tomato fruit.

[0019] Optionally, in step 1.3, a three-dimensional posture classification standard for tomato fruits is established, and the three-dimensional posture space of the tomato is divided into multiple intervals according to actual needs to obtain multiple posture categories. The posture category information of the tomato fruit is labeled according to the interval where the axis of the tomato fruit is located.

[0020] Optionally, in steps 3 and 4, the secondary output information of the second-level target classification model is the confidence information of each posture category of all tomato fruits. By selecting the posture category with the highest confidence from the posture category confidence information of the target tomato fruit as the posture category of the target tomato fruit, the interval in which the three-dimensional posture of the target tomato fruit is located can be determined.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0022] 1. The tomato 3D pose estimation method proposed in this invention uses deep learning to train a detection model on a large number of target samples. During training, only 2D images are used as input, without using 3D point cloud information. It features high accuracy, fast processing speed, and strong generalization ability. It also has strong anti-interference capabilities for complex lighting changes, shading, and background interference in greenhouse environments, and can function effectively even when point cloud information is difficult or inaccurate to obtain.

[0023] 2. This invention uses a classification method to solve the problem of tomato 3D pose estimation. Tomato 3D pose estimation is essentially a regression task. This algorithm, based on a classification approach, divides the continuous pose range of the tomato into several categories. A deep learning classification model is then used to predict the range in which the tomato's pose falls, thus estimating the approximate 3D pose. This effectively solves the problem of difficulty in accurately measuring the 3D pose of tomatoes in greenhouse environments, which makes it difficult to create 3D pose datasets.

[0024] 3. This invention provides more information for realizing automated harvesting operations, which is conducive to selecting a more suitable harvesting posture and makes it possible to achieve precise harvesting. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0026] Figure 1 A flowchart illustrating a tomato 3D pose determination method based on deep learning and classification provided by this invention;

[0027] Figure 2 This is a schematic diagram of a three-dimensional posture classification standard for tomato fruits provided by the present invention;

[0028] Figure 3 In accordance with Figure 2 Examples of tomatoes classified into different posture categories according to the Chinese classification standard. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] like Figure 1As shown, the present invention provides a tomato-based 3D pose determination method based on deep learning and classification, which includes the following steps:

[0031] Step 1: Establish a two-level cascaded network model based on deep learning. The two-level cascaded network model includes a first-level object detection model and a second-level object classification model.

[0032] Step 1.1: Collect multiple greenhouse tomato images and establish a tomato object detection dataset. Use tomato fruits as targets to perform first-level annotation on the greenhouse tomato images, resulting in greenhouse tomato images with bounding boxes containing individual tomato fruits. The first-level annotation information includes the occlusion category and region information of the tomato fruits. The occlusion category indicates whether the tomato fruits within the bounding box are occluded; the region information includes the location and size of the tomato fruits. Generate a tomato object detection dataset from the annotated greenhouse tomato images according to a certain format. The tomato object detection dataset is divided into a training set and a test set according to a certain ratio. The training set is used to train the first-level object detection model, and the test set is used to verify the performance of the model.

[0033] Step 1.2: Establish the first-level object detection network model. The first-level object detection model is trained using the tomato object detection dataset obtained in Step 1.1.

[0034] Step 1.3: Output the image within the labeled bounding box according to the annotation information of the tomato target detection dataset in Step 1.1. Use the screenshot of the tomato fruit as the tomato pose classification dataset for the second-level target classification network. Establish a three-dimensional pose classification standard for the tomato fruit as needed, divide the three-dimensional pose space of the tomato into multiple intervals, and obtain multiple pose categories. Classify the three-dimensional pose of the tomato fruit according to the interval where the axis of the tomato fruit is located, such as... Figure 2 As shown, secondary annotations are performed on each image in the tomato pose classification dataset using the 3D pose category information of the tomato fruit as the target, resulting in the tomato pose classification dataset. The secondary annotation information represents the pose category information of the tomato fruit relative to the end effector, such as... Figure 3 As shown.

[0035] Step 1.4: Establish a second-level target classification model. Train the second-level target classification model using the tomato pose classification dataset obtained in Step 1.3.

[0036] Step 2: Input an image containing one or more tomato fruits into the first-level object detection model obtained in Step 1, and obtain the first-level output information L1 of the last layer of the first-level object detection network, which contains the occlusion category and region information of all tomato fruits:

[0037] L1={li|li=(cls,x,y,width,height),i=1,2…}

[0038] Where li represents the i-th tomato fruit, cls represents the category of the region where the i-th tomato fruit is located, with a value of 1 indicating that the tomato fruit is not obscured and a value of 0 indicating that the tomato fruit is obscured; x represents the x-coordinate of the center point of the region where the i-th tomato fruit is located, y represents the y-coordinate of the center point of the region where the i-th tomato fruit is located, width represents the width of the region where the i-th tomato fruit is located, and height represents the height of the region where the i-th tomato fruit is located.

[0039] Step 3: Crop the unoccluded tomato fruit areas from Step 2 (cls=1), normalize them to a 128×128 tomato image, and input them into the second-level target classification model obtained in Step 1. Obtain the second-level output information L2 of the pose category confidence information of all tomato fruits from the last layer of the second-level target classification model.

[0040] L2={li|li=(cls1, cls2......, clsj), i=1,2......; j=1,2......}

[0041] Where li represents the i-th tomato fruit, j represents the number of categories, and cls1, cls2, ..., clsj represent the confidence scores of each pose category.

[0042] Step 4: Select the posture category with the highest posture category confidence from the posture category confidence information of the target tomato fruit as the posture category of the target tomato fruit.

[0043] Example

[0044] (1) Establish a two-level cascaded network model based on deep learning. This includes a first-level YOLOv5s object detection model and a second-level YOLOv5s-cls pose classification model. 2000 greenhouse tomato images were collected, and each image underwent first-level annotation, which included labeling the tomato's category information (whether it was occluded) and region information (center and size). The annotation results and tomato images were combined into a tomato object detection dataset in COCO format. The regions containing the tomato fruits in the greenhouse tomato images were cropped and normalized to 128×128 tomato fruit images. A three-dimensional pose classification standard for tomato fruits was established, such as... Figure 2 As shown, the three-dimensional pose space of the tomato is divided into four intervals, namely four pose categories (FL, FM, FR, B). The three-dimensional pose of the tomato fruit is classified according to the interval in which the fruit's axis is located. The classification results are as follows. Figure 3As shown, each tomato fruit image was annotated with secondary labeling, i.e., its pose category information was labeled, and a tomato fruit pose classification dataset was created according to the MNIST format.

[0045] Data augmentation is used to expand datasets, including random cropping, random rotation, and brightness adjustment. Data augmentation increases the size and richness of the dataset, which helps improve the robustness of the model.

[0046] The tomato object detection dataset and the tomato fruit pose classification dataset were randomly divided into training and test sets in a 4:1 ratio. The training set was used to train the network model, and the test set was used to evaluate the model's quality. After multiple training iterations, the best-performing YOLOv5s and YOLOv5s-cls network models were selected.

[0047] (2) Input multiple greenhouse tomato images into the first-level target detection model obtained in step 1, and obtain the first-level output information L1 of the last layer of the first-level target detection network model, which contains occlusion category and region information of all detected targets.

[0048] (3) Crop the image of the area where the tomato (cl s=1) was not occluded in step 2, normalize it to a 128×128 image, and input it into the second-level target classification model obtained in step 1 to obtain the second-level output information L2 of the pose category confidence information of all fruits output by the last layer of the second-level target classification network.

[0049] Step 4: Select the pose category with the highest confidence as the pose category of the tomato, and determine which interval the tomato's 3D pose belongs to.

[0050] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0051] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus systems, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0052] In a typical configuration, the device includes one or more processors (CPUs), memory, and a bus. The device may also include input / output interfaces, network interfaces, etc.

[0053] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM, and memory includes at least one memory chip. Memory is an example of computer-readable media.

[0054] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media do not include transient computer-readable media, such as modulated data signals and carrier waves.

[0055] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code, including but not limited to disk storage, CD-ROM, optical storage, etc.

[0056] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0057] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0058] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for determining the 3D pose of a tomato based on deep learning and classification, characterized in that: The tomato 3D pose determination method based on deep learning and classification includes the following steps: Step 1: Establish a two-level cascaded network model based on deep learning. The two-level cascaded network model includes a first-level object detection model and a second-level object classification model. Step 1.1: Collect multiple greenhouse tomato images, establish a tomato target detection dataset, and perform first-level annotation on the greenhouse tomato images with tomato fruits as targets to obtain greenhouse tomato images containing annotation boxes. Each annotation box contains a single tomato fruit. The information of the first-level annotation includes the occlusion category and region information of the tomato fruit. The occlusion category indicates whether the tomato fruit within the annotation box is occluded. The region information includes the position and size of the tomato fruit. Generate the tomato target detection dataset from the annotated greenhouse tomato images according to a certain format. Step 1.2: Establish a first-level object detection network model. Train the first-level object detection model using the tomato object detection dataset obtained in Step 1.

1. Step 1.3: Output the image within the labeled box according to the annotation information of the tomato target detection dataset described in Step 1.

1. Use the screenshot of the tomato fruit as the tomato pose classification dataset of the second-level target classification network. Establish a three-dimensional pose classification standard for the tomato fruit as needed. Divide the three-dimensional pose space of the tomato fruit into multiple intervals to obtain multiple pose categories. Classify the three-dimensional pose of the tomato fruit according to the interval where the axis of the tomato fruit is located. Use the three-dimensional pose category information of the tomato fruit as the target to perform secondary annotation on each image in the tomato pose classification dataset to obtain the tomato pose classification dataset. The information of the secondary annotation is the pose category information of the tomato fruit, which represents the pose of the tomato fruit relative to the end effector. Step 1.4: Establish a second-level target classification model. Train the second-level target classification model using the tomato pose classification dataset obtained in Step 1.

3. Step 2: Input an image containing one or more of the tomato fruits into the first-level object detection model obtained in Step 1, and obtain the first-level output information L1, which contains the occlusion category and region information of all tomato fruits, from the last layer of the first-level object detection network. L1={li|li=(cls,x,y,width,height),i=1,2…} Where li represents the i-th tomato fruit, cls represents the category of the area where the i-th tomato fruit is located, with a value of 1 indicating that the tomato fruit is not obscured, and a value of 0 indicating that the tomato fruit is obscured; x represents the x-coordinate of the center point of the area where the i-th tomato fruit is located, y represents the y-coordinate of the center point of the area where the i-th tomato fruit is located, width represents the width of the area where the i-th tomato fruit is located, and height represents the height of the area where the i-th tomato fruit is located. Step 3: Crop the unoccluded tomato fruit area from Step 2 (cls=1), normalize it to a 128×128 tomato image, and input it into the second-level target classification model obtained in Step 1. Obtain the second-level output information L2 of the pose category confidence information of all tomato fruits from the last layer of the second-level target classification model. L2={li|li=(cls1, cls2......, clsj), i=1,2......; j=1,2......} Where li represents the i-th tomato fruit, j represents the number of categories, and cls1, cls2, ..., clsj represent the confidence scores of each pose category. Step 4: Select the posture category with the highest posture category confidence from the posture category confidence information of the target tomato fruit as the posture category of the target tomato fruit.

2. The tomato 3D pose determination method based on deep learning and classification according to claim 1, characterized in that: In step 1.3, a three-dimensional posture classification standard for tomato fruits is established. The three-dimensional posture space of the tomato is divided into multiple intervals according to actual needs, resulting in multiple posture categories. The posture category information of the tomato fruit is labeled according to the interval where the axis of the tomato fruit is located.

3. The tomato 3D pose determination method based on deep learning and classification according to claim 1, characterized in that: In steps 3 and 4, the secondary output information of the second-level target classification model is the confidence information of each posture category of all tomato fruits. By selecting the posture category with the highest confidence from the posture category confidence information of the target tomato fruit as the posture category of the target tomato fruit, the interval in which the three-dimensional posture of the target tomato fruit is located can be determined.

Images