Tomato self-adaptive pollination method, device, equipment, storage medium and program product

Abstract

The invention relates to the field of tomato pollination, and provides a tomato adaptive pollination method, device and equipment, a storage medium and a program product, and the method comprises the following steps: obtaining a three-primary color image set and a three-dimensional point cloud picture of a tomato plant; training a target detection model through the three-primary color image set to obtain a flowering period segmented image output by the target detection model; training a deep learning network through the segmented image of the flowering period and a fused image of the three-dimensional point cloud image to obtain a neural network model; and controlling a target robot to pollinate the tomato plant based on the flower posture and the flower space coordinate output by the neural network model. According to the invention, the flowering stage flowers are identified through the target detection model, the flowering stage flowers are positioned through the neural network model, and the tomato flowers are adaptively pollinated through the target robot based on the postures and positions of the flowers, so that the pollination efficiency and accuracy are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of tomato pollination, and in particular to a tomato adaptive pollination method, device, equipment, storage medium and program product. BACKGROUND

[0002] At present, the pollination methods of tomato flowers mainly include artificial pollination, mechanical pollination and biological pollination. Although artificial pollination is more flexible than other pollination methods, it will increase the time cost and labor cost; mechanical pollination has high efficiency, but is only suitable for large-area pollination, requires a large amount of pollen, which will cause a certain amount of pollen waste, and has low pollination accuracy; the honeybee pollination method has high economic benefits and can save labor costs, but since the tomato flower does not have a honey gland and has an odor, the honeybee pollination method is not suitable for the planting environment of greenhouse tomatoes.

[0003] Therefore, in order to ensure the efficiency and accuracy of tomato flower pollination and save resources, a tomato adaptive pollination scheme is needed to improve the efficiency and accuracy of tomato pollination. SUMMARY

[0004] The present application provides a tomato adaptive pollination method, device, equipment, storage medium and program product to solve the low efficiency problem of existing tomato pollination methods and realize efficient adaptive pollination of tomatoes.

[0005] The present application provides a tomato adaptive pollination method, comprising the following steps: Obtain a set of three-primary-color images and a three-dimensional point cloud image of a tomato plant; Train a target detection model through the set of three-primary-color images to obtain a flowering period segmentation image output by the target detection model; Train a deep learning network through a fusion image of the flowering period segmentation image and the three-dimensional point cloud image to obtain a neural network model; Based on the flower pose and flower spatial coordinates output by the neural network model, control a target robot to pollinate the tomato plant.

[0006] According to the tomato adaptive pollination method provided by the present application, based on the flower pose and flower spatial coordinates output by the neural network model, the target robot is controlled to pollinate the tomato plant, and then comprises: Determine the accuracy feedback based on the pollination result of the target robot; Based on the accuracy feedback, adjust the parameters of the target detection model and the neural network model until the pollination accuracy of the target robot reaches the preset target.

[0007] According to the tomato adaptive pollination method provided by the application, the three-primary-color image set of the tomato plant comprises: Obtaining three-primary-color images of a tomato plant; Performing data enhancement on the denoised three-primary-color images to obtain image training samples; Labeling the image training samples based on the flowering period of the tomato to obtain a three-primary-color image set containing labeled image samples.

[0008] According to the tomato adaptive pollination method provided by the application, the target detection model comprises a spatial pyramid network and a multi-scale multi-input network; the flowering period segmentation image output by the target detection model is obtained by training the target detection model based on the three-primary-color image set, and comprises: Performing multi-scale prediction and instance segmentation on the target flowers in the labeled image samples through the spatial pyramid network to obtain segmented flower images; Performing sorting on the segmented flower images based on a greedy algorithm of a minimum spanning tree to obtain sorted flower images; Identifying the sorted flower images through the multi-scale multi-input network to obtain the flowering period segmentation image.

[0009] According to the tomato adaptive pollination method provided by the application, the neural network model is obtained by training a deep learning network based on the flowering period segmentation image and the fusion image of the three-dimensional point cloud image, and comprises: Mapping the flowering period segmentation image onto the three-dimensional point cloud image to obtain flower point cloud coordinates; Fusing the flower point cloud coordinates with the flowering period segmentation image to obtain a fusion image; Training a deep learning network based on three-dimensional point cloud data through the fusion image to obtain a neural network model for determining flower posture and flower spatial coordinates.

[0010] According to the tomato adaptive pollination method provided by the application, the target robot is controlled to pollinate the tomato plant based on the flower posture and flower spatial coordinates output by the neural network model, and comprises: Converting the flower spatial coordinates into mechanical arm coordinates of the target robot; Determining a pollination mode based on the flower posture and morphological features; the morphological features are determined based on the flowering period segmentation image; Controlling the target robot to pollinate the tomato plant based on the mechanical arm coordinates and the pollination mode.

[0011] The application further provides a tomato adaptive pollination device comprising the following modules: The acquisition module is configured to acquire a three-primary-color image set and a three-dimensional point cloud image of the tomato plant. The target detection model training module is configured to train a target detection model based on the three-primary-color image set to obtain a flowering period segmentation image output by the target detection model. The neural network model training module is configured to train a deep learning network based on a fusion image of the flowering period segmentation image and the three-dimensional point cloud image to obtain a neural network model. The tomato adaptive pollination module is configured to control a target robot to pollinate the tomato plant based on a flower posture and a flower spatial coordinate output by the neural network model.

[0012] The present application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the tomato adaptive pollination method according to any one of the above when executing the computer program.

[0013] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the tomato adaptive pollination method according to any one of the above.

[0014] The present application also provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the tomato adaptive pollination method according to any one of the above.

[0015] The tomato adaptive pollination method, device, equipment, storage medium, and program product provided by the present application acquire a three-primary-color image set and a three-dimensional point cloud image of a tomato plant, train a target detection model based on the three-primary-color image set to obtain a flowering period segmentation image output by the target detection model, fuse the flowering period segmentation image and the three-dimensional point cloud image, train a deep learning network based on the fusion image to obtain a neural network model, and control a target robot to pollinate the tomato plant based on a flower posture and a flower spatial coordinate output by the neural network model. The present application identifies flowering period flowers through a target detection model, positions the flowering period flowers through a neural network model, and controls a target robot to adaptively pollinate tomato flowers based on the flower posture and position, thereby improving pollination efficiency and accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0017] Figure 1 is one of the flowcharts of the tomato adaptive pollination method provided by the present application.

[0018] Figure 2 is the architecture schematic diagram of the target detection model provided by the present application.

[0019] Figure 3 is the pollination work flowchart of the target robot provided by the present application.

[0020] Figure 4 is the execution flowchart of the flower recognition and positioning provided by the present application.

[0021] Figure 5 is the second flowchart of the tomato adaptive pollination method provided by the present application.

[0022] Figure 6 is the execution flowchart of the multi-pollination mode provided by the present application.

[0023] Figure 7 is the structure schematic diagram of the tomato adaptive pollination device provided by the present application.

[0024] Figure 8 is the structure schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those of ordinary skill in the art without any creative work fall within the protection scope of the present application.

[0026] The tomato adaptive pollination method, device, equipment, storage medium and program product of the present application will be described below. Figures 1-8

[0027] Figure 1 is the flowchart of the tomato adaptive pollination method provided by the present application, as shown in Figure 1 , the method comprises the following steps: Step 100, acquiring a set of three-primary-color images and a three-dimensional point cloud map of a tomato plant; ​Specifically, the tomato adaptive pollination method provided by the present application is implemented by a target robot that can move autonomously. After the target robot enters a working state, an RGB image (three primary color image) and 3D point cloud information of a tomato plant are acquired by a 3D camera configured on the target robot. A series of pretreatments such as gray processing and denoising are performed on the RGB image, and then data enhancement is performed on the pretreated RGB image to obtain a three primary color image set.

[0028] Step 200, training a target detection model by using the three primary color image set to obtain a flowering period segmentation image output by the target detection model; Specifically, as shown in Figure 2 The present application proposes a method for accurately identifying the state of a tomato flower based on a cascaded convolutional neural network (i.e., a target detection model). The target detection model is composed of an image segmentation network and a multi-scale multi-input network. First, an end-to-end spatial pyramid network (Fused Pyramid Network, FPN) is used to perform multi-scale prediction and instance segmentation on the tomato flower to obtain a flower segmentation image. Then, the flower segmentation image is sorted, and a multi-scale multi-input network is used to accurately detect and identify each flower segmentation image according to the sorting. The identification result includes flower segmentation images of each flowering period, from which a flower segmentation image of the flowering period is selected, i.e., a flowering period segmentation image.

[0029] Step 300, training a deep learning network by using a fusion image of the flowering period segmentation image and the three-dimensional point cloud image to obtain a neural network model; Specifically, the flowering period segmentation image is fused with the spatial coordinates corresponding to each flowering period segmentation image in the 3D point cloud image to obtain a fusion image. The fusion image is trained by using a Point CNN network (a deep learning network for three-dimensional point cloud data) to obtain a neural network model that can output flower poses and flower spatial coordinates.

[0030] Step 400, based on the flower poses and flower spatial coordinates output by the neural network model, controlling the target robot to pollinate the tomato plant.

[0031] Specifically, the tomato adaptive pollination method provided by the present application adopts a target detection model to accurately identify the tomato flower, maps the obtained flowering period segmentation image to a three-dimensional point cloud image to obtain flower spatial coordinates, converts the flower spatial coordinates into target robot arm coordinates, and guides the execution end of the robot arm to pollinate the tomato plant according to the distribution coordinates of the flower spatial coordinates.

[0032] As Figure 3As shown, the target robot provided by the present application has the function of autonomous pollination. During the travel of the target robot before pollinating the tomato, the road conditions of the travel road are detected in real time by the configured laser sensor. If the laser sensor detects an obstacle in the forward direction of the road, obstacle avoidance processing is performed. If the road is actually impassable, the forward movement is stopped, and an alarm is triggered. When the 3D camera configured on the target robot detects a tomato flower, the target robot stops moving forward after approaching the tomato flower.

[0033] In this embodiment, the three-primary-color image set and the three-dimensional point cloud image of the tomato plant are obtained, the target detection model is trained based on the three-primary-color image set, and the flowering period segmentation image output by the target detection model is obtained. The flowering period segmentation image and the three-dimensional point cloud image are fused, and the deep learning network is trained based on the fused image to obtain a neural network model. The flower pose and the flower spatial coordinates output by the neural network model are used to control the target robot to pollinate the tomato plant. The target detection model is used to identify the flowers in the flowering period, and the neural network model is used to locate the flowers in the flowering period. Based on the flower pose and the position, the target robot is used to adaptively pollinate the tomato flowers, thereby improving the pollination efficiency and accuracy.

[0034] In one embodiment, the tomato adaptive pollination method provided by the present application embodiment can further include: Step 500, determining accuracy feedback based on the pollination result of the target robot; Step 600, adjusting the parameters of the target detection model and the neural network model based on the accuracy feedback until the pollination accuracy of the target robot reaches a preset target.

[0035] Specifically, the present application also provides a closed-loop feedback method for a tomato flower pollination related model (target detection model and neural network model).

[0036] After the target robot pollinates the tomato plant, the flower detection confidence, the offset between the predicted pose and the actual pose, the pollination success flag, the mechanical arm collision event, and the position of the failed positioning flower are counted (i.e., the accuracy feedback). According to these information, the key indicators such as the pollination success rate, the average pose error and the false alarm rate are calculated.

[0037] The adjustment parameters of the target detection model: for high false alarm rate, increase the positioning loss weight; for missed detection, expand the data enhancement strategy. The adjustment parameters of the neural network model: for the error of depth information, strengthen the depth channel feature extraction; for angle deviation, introduce a cyclic angle loss. Weight the failed image samples, and dynamically adjust the model learning rate.

[0038] The planting conditions (i.e., preset goals) for model training can be: the accuracy of the model output results for multiple consecutive iterations is higher than a threshold, for example, the accuracy of the model output results for 5 consecutive iterations is higher than 95%; or the total number of model iterations is higher than a certain value.

[0039] This embodiment transforms the failure feedback of tomato pollination into model training signals through reinforcement model learning to improve the pollination accuracy of the target robot.

[0040] In one embodiment, the tomato adaptive pollination method provided by the embodiment of the present invention may further include: Step 110: Acquire a three-primary-color image of a tomato plant; Step 120: performing data enhancement on the denoised three-primary color image to obtain an image training sample; Step 130: Label the image training samples based on the tomato flowering period to obtain a three-primary color image set containing the labeled image samples.

[0041] Specifically, the process of preprocessing RGB images includes: using the weighted average method to perform grayscale processing on the collected RGB images; smoothing the RGB images through Gaussian filtering to reduce the blurriness of the RGB images and denoising the RGB images; using the histogram equalization method on the denoised RGB images to map the pixel grayscale to obtain an image grayscale with uniform probability distribution, thereby improving the contrast of the RGB images.

[0042] The method of data enhancement for preprocessed RGB images includes flipping, rotating, scaling, and random cropping of RGB images to increase the diversity of image training samples, improve the robustness of related models, and avoid overfitting.

[0043] You can use the labelImg tool (a graphical image annotation tool primarily used to generate labeled data for computer vision tasks) to annotate image training samples, generating an RGB image dataset containing annotated image samples. The annotated image samples can be labeled with the bloom stage of the flower in the image, such as bud stage, fully open stage, flowering stage, and initial fruiting stage.

[0044] This embodiment preprocesses and enhances the data of the acquired RGB images to increase the diversity of image training samples, improve the robustness of the relevant models, and avoid overfitting of the relevant models.

[0045] In one embodiment, the tomato adaptive pollination method provided by the embodiment of the present invention may further include: Step 210: Perform multi-scale prediction and instance segmentation on the target flower in the labeled image sample using the spatial pyramid network to obtain a segmented flower image; Step 220, a greedy algorithm based on the minimum spanning tree is used to sort the segmented flower images to obtain sorted flower images; Step 230, the sorted flower images are identified by the multi-scale multi-input network to obtain the flowering period segmentation image.

[0046] Specifically, as shown in the drawings, Figure 2 The tomato flowers are identified by the target detection model, first, the labeled image samples in the three primary color image set are transmitted into the first level FS-FPN network (i.e. the spatial pyramid network), and the flowers existing in the labeled image samples are predicted. The flower region (Flower Segment, FS) image is obtained by multi-scale segmentation of the predicted flower features through up-sampling, including FSA (FSA1, FSA2 and FSA3) and FSB (FSB1, FSB2, FSB3), and the distant flower targets and single flower targets are removed.

[0047] The prim algorithm of the minimum spanning tree is used to sort the multiple segmented flower images to obtain the sorted flower images, the sorted flower images of each scale are put into the corresponding vector (storage container), and then the vector is transmitted into the second level Yolov3 network (a target detection model) according to the corresponding scale, the flowering period of the flower is accurately identified to obtain the segmentation image of each flowering period, and the target tomato flower in the flowering period is segmented, i.e. the flowering period segmentation image.

[0048] The above minimum spanning tree is a problem in graph theory, which refers to finding a tree containing all vertices in a weighted connected undirected graph, which is acyclic and connected, and the sum of the weights of all edges is minimized. Prim algorithm is one of the greedy algorithms for solving this problem. The core idea of Prim algorithm is: starting from an arbitrary vertex, gradually selecting the minimum weight edge connecting the selected vertex and the unselected vertex, and finally expanding into a minimum spanning tree covering all vertices.

[0049] The tomato flowers are detected by the target detection model, and the accurate identification and classification of the flowers in each flowering period are realized.

[0050] In one embodiment, the tomato adaptive pollination method provided by the embodiment of the present application can further include: Step 310, mapping the flowering period segmentation image to the three-dimensional point cloud image to obtain flower point cloud coordinates; Step 320, fusing the flower point cloud coordinates and the flowering period segmentation image to obtain a fusion image; Step 330, training a deep learning network based on three-dimensional point cloud data by using the fusion image to obtain a neural network model for determining flower posture and flower spatial coordinates.

[0051] Specifically, as shown in Figure 4 , the point cloud data of the flower in the flowering period is segmented by the Point CNN algorithm. The flower identified based on the image training sample is matched with the three-dimensional point cloud information to obtain a fusion image. That is, the two-dimensional flowering period segmentation image is mapped to the unstructured spatial point cloud data (three-dimensional point cloud map) to obtain the camera point cloud coordinates of the flower in the flowering period (flower point cloud coordinates); then, the flower point cloud coordinates are fused with the flowering period segmentation image to obtain a fusion image. The deep learning network (Point CNN) based on the three-dimensional point cloud data is trained through the fusion image to obtain a neural network model for outputting the three-dimensional imaging of the flower (flower posture) and the spatial position information of the flower (flower spatial coordinates), so as to spatially locate the flower in the flowering period.

[0052] The embodiment combines the identified flower in the flowering period with the 3D point cloud map to realize accurate positioning of the flower to be pollinated and provide a basis for accurate pollination.

[0053] Figure 5 is a flowchart of a tomato adaptive pollination method provided by the application, as shown in Figure 2 , the method can further include: Step 410, converting the flower spatial coordinates into the mechanical arm coordinates of the target robot; Step 420, determining the pollination mode based on the flower posture and the morphological features; the morphological features are determined based on the flowering period segmentation image; Step 430, controlling the target robot to pollinate the tomato plant based on the mechanical arm coordinates and the pollination mode.

[0054] Specifically, the execution end of the target robot includes a spray head (a pesticide spray head and an air spray head) and a vibrating member (a vibrating rod), and the pollination mode includes air spraying and shaking, and the mechanical arm of the target robot guides the execution end to adaptively select different pollination modes to pollinate the tomato flowers.

[0055] As shown in Figure 6 , since the growth time of each flower of the tomato is different, in the process of detecting the tomato flowers, the target robot, according to the flowering period of the flower identified by the target detection model, does not pollinate the flower if the flower is in the bud stage or the fruiting stage; if the flower is in the flowering period, different pollination modes or a combination of multiple pollination modes are selected according to the orientation of the flower (flower posture) and the shape and size (morphological features) to accurately pollinate the flower. For example, high-pressure air is used to simulate natural wind in all directions above and below the target flower to pollinate, or a vibrating rod is used to vibrate the tomato peduncle to make the pollen fall on the style, so as to achieve the goal of adaptive pollination.

[0056] The adaptive tomato pollination method implemented by the robot improves the tomato pollination efficiency.

[0057] The tomato adaptive pollination device provided by the present application is described below. The tomato adaptive pollination device described below can be referred to in correspondence with the tomato adaptive pollination method described above.

[0058] Please refer to Figure 7 The present application also provides a tomato adaptive pollination device, comprising: The acquisition module 701 is configured to acquire a three-primary-color image set and a three-dimensional point cloud image of a tomato plant. The target detection model training module 702 is configured to train a target detection model based on the three-primary-color image set to obtain a flowering period segmentation image output by the target detection model. The neural network model training module 703 is configured to train a deep learning network based on a fusion image of the flowering period segmentation image and the three-dimensional point cloud image to obtain a neural network model. The tomato adaptive pollination module 704 is configured to control a target robot to pollinate the tomato plant based on a flower pose and a flower spatial coordinate output by the neural network model.

[0059] Optionally, the tomato adaptive pollination device further comprises: The accuracy feedback determination module is configured to determine an accuracy feedback based on a pollination result of the target robot. The model parameter adjustment module is configured to adjust parameters of the target detection model and the neural network model based on the accuracy feedback until a pollination accuracy of the target robot reaches a preset target.

[0060] Optionally, the acquisition module comprises: The three-primary-color image acquisition unit is configured to acquire a three-primary-color image of a tomato plant. The image training sample determination unit is configured to perform data enhancement on the denoised three-primary-color image to obtain an image training sample. The image training sample labeling unit is configured to label the image training sample based on a tomato flowering period to obtain a three-primary-color image set containing labeled image samples.

[0061] Optionally, the target detection model comprises a spatial pyramid network and a multi-scale multi-input network; and the target detection model training module comprises: The segmented flower image determination unit is configured to perform multi-scale prediction and instance segmentation on a target flower in the labeled image sample based on the spatial pyramid network to obtain a segmented flower image. a segmented flower image sorting unit, configured to sort the segmented flower images based on a greedy algorithm of a minimum spanning tree to obtain sorted flower images; The sorted flower image recognition unit is used to recognize the sorted flower images through the multi-scale multi-input network to obtain a flowering period segmentation image.

[0062] Optionally, the neural network model training module includes: a flower point cloud coordinate determination unit, configured to map the flowering period segmented image onto the three-dimensional point cloud image to obtain flower point cloud coordinates; a fused image determining unit, configured to fuse the flower point cloud coordinates with the flowering period segmented image to obtain a fused image; The neural network model determination unit is used to train a deep learning network based on three-dimensional point cloud data through the fused image to obtain a neural network model for determining the flower posture and flower spatial coordinates.

[0063] Optionally, the tomato adaptive pollination module includes: A coordinate conversion unit, used to convert the flower space coordinates into the target robot's robotic arm coordinates; a pollination mode determination unit, configured to determine the pollination mode based on flower posture and morphological features; the morphological features being determined based on the flowering stage segmented image; A pollination control unit is used to control the target robot to pollinate the tomato plant based on the coordinates of the robotic arm and the pollination method.

[0064] Figure 8 An example of a physical structure diagram of an electronic device is shown below. Figure 8 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communications bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communications bus 840. The processor 810 may invoke logic instructions in the memory 830 to execute a tomato adaptive pollination method, which includes: obtaining a three-primary color image set and a three-dimensional point cloud image of a tomato plant; training a target detection model using the three-primary color image set to obtain a flowering period segmentation image output by the target detection model; training a deep learning network using a fusion image of the flowering period segmentation image and the three-dimensional point cloud image to obtain a neural network model; and controlling a target robot to pollinate the tomato plant based on the flower posture and flower spatial coordinates output by the neural network model.

[0065] Moreover, the logic instructions in the memory 830 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0066] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the tomato adaptive pollination method provided by the above-mentioned methods, the method comprising: obtaining a three-primary-color image set and a three-dimensional point cloud image of a tomato plant; training a target detection model through the three-primary-color image set to obtain a flowering period segmentation image output by the target detection model; training a deep learning network through a fusion image of the flowering period segmentation image and the three-dimensional point cloud image to obtain a neural network model; and controlling a target robot to pollinate the tomato plant based on a flower pose and a flower spatial coordinate output by the neural network model.

[0067] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the tomato adaptive pollination method provided by the above-mentioned methods, the method comprising: obtaining a three-primary-color image set and a three-dimensional point cloud image of a tomato plant; training a target detection model through the three-primary-color image set to obtain a flowering period segmentation image output by the target detection model; training a deep learning network through a fusion image of the flowering period segmentation image and the three-dimensional point cloud image to obtain a neural network model; and controlling a target robot to pollinate the tomato plant based on a flower pose and a flower spatial coordinate output by the neural network model.

[0068] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0069] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0070] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A tomato adaptive pollination method, characterized in that: include: Obtain a three-color image set and a three-dimensional point cloud image of a tomato plant; The target detection model is trained using the three primary color image set to obtain a flowering period segmentation image output by the target detection model; Training a deep learning network using the flowering period segmentation image and the fused image of the three-dimensional point cloud image to obtain a neural network model; Based on the flower posture and flower space coordinates output by the neural network model, the target robot is controlled to pollinate the tomato plant.

2. The tomato adaptive pollination method according to claim 1, characterized in that: The method further comprises: controlling the target robot to pollinate the tomato plant based on the flower posture and flower space coordinates output by the neural network model; and then comprising: Determining accuracy feedback based on the pollination results of the target robot; Based on the accuracy feedback, the parameters of the target detection model and the neural network model are adjusted until the pollination accuracy of the target robot reaches a preset target.

3. The tomato adaptive pollination method according to claim 1, characterized in that: The obtaining of the three-primary-color image set of the tomato plant comprises: Get three primary color images of tomato plants; Perform data enhancement on the denoised three-primary color image to obtain image training samples; The image training samples are labeled based on the tomato flowering period to obtain a three-primary color image set containing the labeled image samples.

4. The tomato adaptive pollination method according to claim 3, characterized in that: The target detection model includes a spatial pyramid network and a multi-scale multi-input network; the target detection model is trained using the three-primary color image set, and the flowering period segmentation image output by the target detection model includes: Performing multi-scale prediction and instance segmentation on the target flower in the labeled image sample by using the spatial pyramid network to obtain a segmented flower image; sorting the segmented flower images based on a greedy algorithm of a minimum spanning tree to obtain sorted flower images; The sorted flower images are recognized by the multi-scale multi-input network to obtain a flowering period segmentation image.

5. The tomato adaptive pollination method according to claim 1, characterized in that: The deep learning network is trained by using the flowering period segmentation image and the fusion image of the three-dimensional point cloud image to obtain a neural network model, which includes: Mapping the flowering period segmented image onto the three-dimensional point cloud image to obtain flower point cloud coordinates; fusing the flower point cloud coordinates with the flowering period segmentation image to obtain a fused image; A deep learning network based on three-dimensional point cloud data is trained using the fused image to obtain a neural network model for determining the flower posture and spatial coordinates of the flower.

6. The tomato adaptive pollination method according to claim 1, characterized in that: The controlling the target robot to pollinate the tomato plant based on the flower posture and flower spatial coordinates output by the neural network model includes: Convert the flower space coordinates into the target robot's robotic arm coordinates; Determining the pollination mode based on flower posture and morphological characteristics; the morphological characteristics are determined based on the flowering period segmentation image; Based on the robotic arm coordinates and the pollination method, the target robot is controlled to pollinate the tomato plants.

7. A tomato adaptive pollination device, characterized in that: include: an acquisition module, for acquiring a three-primary-color image set and a three-dimensional point cloud image of a tomato plant; A target detection model training module is used to train the target detection model using the three-primary color image set to obtain a flowering period segmentation image output by the target detection model; A neural network model training module is used to train a deep learning network using the flowering period segmentation image and the fused image of the three-dimensional point cloud image to obtain a neural network model; The tomato adaptive pollination module is used to control the target robot to pollinate the tomato plants based on the flower posture and flower space coordinates output by the neural network model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the tomato adaptive pollination method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the tomato adaptive pollination method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the tomato adaptive pollination method according to any one of claims 1 to 6 is implemented.

Images