Method and device for positioning picking points of multi-grade hongyou No.9 tea leaves, equipment and medium

Abstract

The application relates to a multi-level English red No. 9 tea leaf picking point positioning method, device, equipment and medium. The method comprises the following steps: obtaining detection box information and mask information of a target area corresponding to tea leaves to be picked, inputting the detection box information of the target area corresponding to the tea leaves to be picked into a pre-trained English red No. 9 tea leaf picking point positioning model, performing feature extraction on an improved RegNet backbone network, generating a heat map in an up-sampling module to indicate the picking point coordinates of single bud, one bud and one leaf and one bud and two leaves in an English red No. 9 tea tree image; obtaining the depth information of each mask pixel point based on the mask information, obtaining the average depth information of the tea leaves by taking the average value, and guiding a picking robot to pick the tea leaves by grade based on the picking point coordinates and the average depth information. The application can realize the grading picking of English red No. 9 tea leaves.

Description

Technical Field

[0001] This application relates to the field of agricultural production, and in particular to a method for locating the picking point of multi-grade Yinghong No. 9 tea leaves, a corresponding device, electronic equipment, and a computer-readable storage medium. Background Technology

[0002] Yinghong No. 9 tea is a famous tea from Guangdong Province. It is an asexual variety developed by the Guangdong Tea Research Institute from the Yunnan large-leaf tea population through systematic selection. The plant is tall with a semi-open growth habit, dense branching, and leaves that grow slightly upwards. The leaves are exceptionally large, oval-shaped, light green, and glossy. The leaf surface is raised, the leaf body slightly folds inwards, the leaf margin is wavy, and the leaf tip gradually sharpens. Yinghong No. 9 tea has significant economic value, and every stage of its production process has a significant impact on the quality of the final product. Especially during the tea leaf picking stage, the quality of the tea leaves directly affects the quality of the finished tea. Therefore, how to efficiently and accurately pick high-quality Yinghong No. 9 tea leaves has become a key issue in tea production. Rapid and accurate identification and location of Yinghong No. 9 tea leaves is of great significance for realizing the intelligent harvesting of Yinghong No. 9 tea.

[0003] In recent years, vision-based automated harvesting robots have been used for harvesting premium teas. However, the tea leaves of Yinghong No. 9 are easily obscured, and their color is similar to that of older leaves. These factors make it extremely difficult to identify and locate the harvesting points for Yinghong No. 9 tea leaves, severely limiting the automated harvesting of this premium tea. Current methods for locating the harvesting points of Yinghong No. 9 tea leaves suffer from low accuracy and efficiency, and cannot achieve graded harvesting.

[0004] In summary, existing technologies have limitations on the automated harvesting of Yinghong No. 9 premium tea, such as the ease with which tea leaves are obscured and the similar color of tender buds and old leaves. Furthermore, the current methods for locating Yinghong No. 9 tea leaf picking points suffer from low accuracy and efficiency, and cannot achieve graded harvesting of Yinghong No. 9 tea leaves. Therefore, the applicant has made corresponding explorations to address these issues. Summary of the Invention

[0005] The purpose of this application is to solve the above-mentioned problems by providing a method, device, electronic equipment and computer-readable storage medium for locating the picking point of multi-grade Yinghong No. 9 tea leaves.

[0006] To achieve the various objectives of this application, the following technical solution is adopted:

[0007] A method for locating the picking points of multi-grade Yinghong No. 9 tea leaves, proposed to meet one of the purposes of this application, includes:

[0008] Responding to the picking point location command of Yinghong No. 9 tea leaves at multiple levels, obtain the image of the Yinghong No. 9 tea tree to be detected, which contains the image of Yinghong No. 9 tea leaves;

[0009] The Yinghong No. 9 tea leaf instance segmentation model, which has been trained to convergence, is used to segment the Yinghong No. 9 tea tree image to be detected, and the detection box information and mask information of the target area corresponding to the Yinghong No. 9 tea leaves to be picked are obtained. The Yinghong No. 9 tea leaves to be picked include one or any combination of single bud, one bud and one leaf, and one bud and two leaves.

[0010] The detection box information of the target area corresponding to the Yinghong No. 9 tea leaves to be picked is input into the pre-trained Yinghong No. 9 tea leaf picking point localization model. Feature extraction is performed in the improved RegNet backbone network, and a heat map is generated in the upsampling module to indicate the picking point coordinates of the single bud, one bud and one leaf and one bud and two leaves in the Yinghong No. 9 tea tree image.

[0011] The depth information of each mask pixel is obtained based on the mask information, and the average depth information of the tea leaves is obtained by taking the average value. Based on the picking point coordinates and the average depth information, the picking robot is guided to pick the tea leaves in different grades, so as to complete the picking point positioning of Yinghong No. 9 tea leaves in multiple grades.

[0012] Optionally, the basic network architecture of the Yinghong No. 9 tea leaf instance segmentation model is an improved Mask-RCNN model, wherein the backbone network of the improved Mask-RCNN model is a CSPNEXt network, the neck network is a PAFPN network, the loss function is a Smooth L1Loss function, and it introduces a Coordinating Attention mechanism module and a PointRend module.

[0013] Optionally, the step of using a converged Yinghong No. 9 tea leaf instance segmentation model to perform instance segmentation on the Yinghong No. 9 tea tree image to be detected, and obtaining the detection box information and mask information of the target region corresponding to the Yinghong No. 9 tea leaves to be picked, includes:

[0014] The image of the Yinghong No. 9 tea tree to be detected is input into the Yinghong No. 9 tea leaf instance segmentation model that has been trained to convergence, and the CSPNEXt network in the Yinghong No. 9 tea leaf instance segmentation model is used to extract features from the image of the Yinghong No. 9 tea tree to be detected.

[0015] The PAFPN network receives the features extracted by the CSPNEXt network and constructs a multi-scale feature pyramid;

[0016] A region generation network is used to generate candidate regions for the target Yinghong No. 9 tea leaves to be picked from the feature pyramid;

[0017] For each candidate region, RoI pooling or RoI alignment is performed to map candidate regions of different sizes and shapes to feature maps of fixed sizes.

[0018] The PointRend module is used to perform instance segmentation on each candidate region to obtain the detection box information and mask information of the target region corresponding to the Yinghong No. 9 tea leaves to be picked.

[0019] Optionally, the steps of training the Yinghong No. 9 tea leaf instance segmentation model include:

[0020] The improved Mask-RCNN model was trained using the SGD optimizer, with the initial learning rate set to 0.001, the weight decay set to 0.001, the epochs set to 50, and the batch size set to 8.

[0021] The training set of Yinghong No. 9 tea leaf dataset is input into the improved Mask-RCNN model for training, and the network parameters are updated once per epoch.

[0022] The validation set data is input into the improved Mask-RCNN model for prediction, and the model accuracy is evaluated. The model file parameters are saved after every 10 eopchs are trained. The model parameters with the highest validation accuracy are selected as the optimal solution to obtain the Yinghong No. 9 tea leaf instance segmentation model.

[0023] Optionally, the basic network architecture of the Yinghong No. 9 tea leaf picking point localization model is a PLNet network. The PLNet network includes an improved RegNet backbone network and an upsampling module. The improved RegNet backbone network includes four stages, each stage containing multiple convolutional layers and bottleneck blocks. The convolutional layers in the latter three stages are deformable convolutions.

[0024] Optionally, the steps of inputting the detection box information of the target area corresponding to the Yinghong No. 9 tea leaves to be picked into the pre-trained Yinghong No. 9 tea leaf picking point localization model, performing feature extraction in the improved RegNet backbone network, and generating a heatmap in the upsampling module to indicate the picking point coordinates of the single bud, one bud and one leaf, and one bud and two leaves in the Yinghong No. 9 tea tree image include:

[0025] In the improved RegNet backbone network of the Yinghong No. 9 tea leaf picking point localization model, representative feature maps are extracted from the detection box information of the target area corresponding to the Yinghong No. 9 tea leaves to be picked.

[0026] In the upsampling module, interpolation or deconvolution techniques are used to increase the spatial resolution of the feature map;

[0027] The feature map is converted into a heat map to determine the confidence or probability distribution of the picking point coordinates of each level of Yinghong No. 9 tea leaves in the Yinghong No. 9 tea tree image.

[0028] Based on the confidence or probability distribution of the picking point coordinates of Yinghong No. 9 tea leaves at each level, the picking point coordinates of single bud, one bud and one leaf, and one bud and two leaves in the Yinghong No. 9 tea tree image are determined.

[0029] Optionally, after the step of obtaining the image of the Yinghong No. 9 tea tree to be detected, which includes the image of Yinghong No. 9 tea leaves, the process includes:

[0030] In response to the image preprocessing instruction, the image of the Yinghong No. 9 tea tree to be detected is subjected to data augmentation, which includes one or any combination of random illumination enhancement, random scaling, random rotation, random cropping, and horizontal flipping.

[0031] A picking point positioning device for multi-grade Yinghong No. 9 tea leaves, provided for another purpose of this application, includes:

[0032] The image acquisition module is configured to respond to the picking point positioning command of multi-level Yinghong No. 9 tea leaves and acquire the image of the Yinghong No. 9 tea tree to be detected, which contains the image of Yinghong No. 9 tea leaves.

[0033] The module for determining tea leaves to be picked is configured to use a converged Yinghong No. 9 tea leaf instance segmentation model to perform instance segmentation on the Yinghong No. 9 tea tree image to be detected, and obtain the detection box information and mask information of the target area corresponding to the Yinghong No. 9 tea leaves to be picked. The Yinghong No. 9 tea leaves to be picked include one or any combination of single bud, one bud and one leaf, and one bud and two leaves.

[0034] The picking point coordinate determination module is set to input the detection box information of the target area corresponding to the Yinghong No. 9 tea leaves to be picked into the pre-trained Yinghong No. 9 tea leaf picking point localization model, perform feature extraction in the improved RegNet backbone network, and generate a heat map in the upsampling module to indicate the picking point coordinates of the single bud, one bud and one leaf and one bud and two leaves in the Yinghong No. 9 tea tree image.

[0035] The picking point positioning module is configured to obtain the depth information of each mask pixel based on the mask information, and obtain the average depth information of the tea leaves by taking the average value. Based on the picking point coordinates and the average depth information, the picking robot is guided to pick the tea leaves in different grades to complete the picking point positioning of Yinghong No. 9 tea leaves of multiple grades.

[0036] An electronic device provided for another purpose of this application includes a central processing unit and a memory, the central processing unit being used to invoke and run a computer program stored in the memory to perform the steps of the multi-level Yinghong No. 9 tea leaf picking location method described in this application.

[0037] A computer-readable storage medium is provided for another purpose of this application, which stores, in the form of computer-readable instructions, a computer program implemented according to the multi-level Yinghong No. 9 tea leaf picking point positioning method, which, when called by a computer, executes the steps included in the corresponding method.

[0038] Compared to existing technologies, this application addresses the problems in existing technologies, such as excessively strong or dim lighting, low resolution between tender buds and old leaves, which severely limit the automated harvesting of premium teas. It also addresses the issues of low positioning accuracy and efficiency in the current positioning method for Yinghong No. 9 tea leaf picking points, which cannot achieve graded harvesting of tea leaves. This application provides, but is not limited to, the following beneficial effects:

[0039] Firstly, replacing the backbone network of the Mask-RCNN model with CSPNEXt improves the network's ability to extract features of Yinghong No. 9 tea leaves. Replacing the FPN module with the PAFPN module improves the network's ability to detect Yinghong No. 9 tea leaves of different sizes. Adding a coordinate attention module between the backbone and neck network enhances the network's ability to distinguish Yinghong No. 9 tea leaves, suppressing the background and highlighting the foreground target. Adding a PointRend module to the mask output branch improves the accuracy of the segmentation mask.

[0040] Secondly, the classification loss of the Mask-RCNN model is replaced with the Focal Loss function. Focal Loss can alleviate the class imbalance problem, making the model pay more attention to those samples that are difficult to classify, thereby improving the accuracy and generalization ability of instance segmentation. The bounding box loss of the Mask-RCNN model is replaced with the Smooth L1 Loss function. Since the range of bounding box coordinates is large, the Smooth L1 Loss function can effectively handle the prediction of bounding boxes, improving the accuracy and stability of the detector.

[0041] Third, the joint instance segmentation and key point detection are performed. First, the improved Mask-RCNN model is used to detect Yinghong No. 9 tea leaves at various levels, and then the bounding boxes and masks are output. Then, the PLNet model is used to predict the picking point coordinates of the detected Yinghong No. 9 tea leaves.

[0042] Fourth, the PLNet model can be used to obtain the single bud picking point, one bud and one leaf picking point, and one bud and two leaves picking point of Yinghong No. 9 tea leaves, so as to realize the graded picking of Yinghong No. 9 tea leaves.

[0043] Furthermore, by using the Yinghong No. 9 tea leaf picking point positioning model, the picking points for single buds, one bud and one leaf, and one bud and two leaves of Yinghong No. 9 tea leaves can be obtained, realizing graded picking of Yinghong No. 9 tea leaves and greatly improving the picking efficiency and quality of Yinghong No. 9 tea leaves. Attached Figure Description

[0044] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0045] Figure 1 An exemplary network architecture used in the method for locating picking points of multi-level Yinghong No. 9 tea leaves in this application;

[0046] Figure 2 This is a flowchart illustrating the method for locating the picking points of multi-level Yinghong No. 9 tea leaves in the embodiments of this application.

[0047] Figure 3 This is a schematic diagram of the PointRend module in an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of the Coordinating Attention (COA) mechanism module in an embodiment of this application.

[0049] Figure 5 This is a schematic diagram of the PAFPN network in an embodiment of this application;

[0050] Figure 6 This is a schematic diagram of the picking point labels for multi-level Yinghong No. 9 tea leaves in the embodiments of this application;

[0051] Figure 7 This is a flowchart illustrating the overall process of determining the picking points for Yinghong No. 9 tea leaves at multiple levels, as described in this application embodiment.

[0052] Figure 8 This is a schematic diagram of the PLNet network model in an embodiment of this application;

[0053] Figure 9 This is a schematic diagram of the upsampling module in an embodiment of this application;

[0054] Figure 10 This is a schematic diagram of the deformable convolution module in an embodiment of this application;

[0055] Figure 11 This is a schematic diagram of the principle of the picking point positioning device for multi-level Yinghong No. 9 tea leaves in the embodiments of this application;

[0056] Figure 12 This is a schematic diagram of the structure of the computer device in the embodiments of this application. Detailed Implementation

[0057] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0058] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0059] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0060] Those skilled in the art will understand that the terms "client," "terminal," and "terminal device" as used herein include both devices that receive wireless signals, devices that only possess wireless signal receiver capabilities without transmission capabilities, and devices with receiving and transmitting hardware, devices that have receiving and transmitting hardware capable of bidirectional communication over a bidirectional communication link. Such devices may include: cellular or other communication devices such as personal computers or tablets, having single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service) that can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant) that may include radio frequency receivers, pagers, internet / intranet access, web browsers, notepads, calendars, and / or GPS (Global Positioning System) receivers; and conventional laptops and / or handheld computers or other devices that have and / or include radio frequency receivers. As used herein, "client," "terminal," and "terminal device" can be portable, transportable, installed in a means of transportation (air, sea, and / or land), or suitable and / or configured to operate locally and / or in a distributed manner, operating in any other location on Earth and / or in space. "Client," "terminal," and "terminal device" as used herein can also be a communication terminal, an internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, set-top box, etc.

[0061] The hardware referred to by the names "server," "client," and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann architecture, such as a central processing unit (including an arithmetic logic unit and a control unit), memory, input devices, and output devices. The computer program is stored in its memory, and the central processing unit loads the program stored in the secondary storage into the main memory to run it, execute the instructions in the program, and interact with the input and output devices to complete specific functions.

[0062] It should be noted that the concept of "server" used in this application can also be extended to the case of server clusters. Based on the network deployment principles understood by those skilled in the art, the servers should be logically divided. Physically, these servers can be independent of each other but accessible through interfaces, or they can be integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method in this application.

[0063] One or more of the technical features of this application, unless explicitly specified herein, can be deployed on a server and accessed by a client remotely calling the online service interface provided by the server, or can be directly deployed and run on a client for access.

[0064] Unless otherwise specified, the neural network models referenced or potentially referenced in this application may be deployed on a remote server and invoked remotely on the client, or deployed on a client with the capability to invoke directly. In some embodiments, when running on the client, the corresponding intelligence may be acquired through transfer learning in order to reduce the requirements on the client's hardware resources and avoid excessive consumption of the client's hardware resources.

[0065] Unless otherwise specified, all data involved in this application may be stored remotely on a server or on a local terminal device, as long as it is suitable for use by the technical solution of this application.

[0066] Those skilled in the art will understand that although the various methods in this application are described based on the same concept and thus present commonality among them, they can be performed independently unless otherwise specified. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept; therefore, concepts expressed in the same way, as well as concepts that are appropriately changed for convenience but are expressed differently, should be understood equivalently.

[0067] Unless otherwise expressly stated, the various embodiments disclosed in this application can be combined in a cross-cutting manner to flexibly construct new embodiments, as long as such combination does not depart from the inventive spirit of this application and can meet the needs of the prior art or solve a certain deficiency in the prior art. Those skilled in the art should be aware of such modifications.

[0068] Please see Figure 1 In one embodiment of the method for locating the picking points of multi-grade Yinghong No. 9 tea leaves of this application, the method includes:

[0069] Step S10: Respond to the picking point positioning command of multi-level Yinghong No. 9 tea leaves and obtain the image of the Yinghong No. 9 tea tree to be detected containing the image of Yinghong No. 9 tea leaves;

[0070] The Yinghong No. 9 tea leaf picking robot can respond to the picking point positioning command of multi-level Yinghong No. 9 tea leaves and acquire the image of the Yinghong No. 9 tea tree to be detected, which contains the image of Yinghong No. 9 tea leaves. The image of the Yinghong No. 9 tea tree to be detected can be acquired based on the binocular camera of the Yinghong No. 9 tea leaf picking robot. The multi-level Yinghong No. 9 tea leaves include one or any combination of single bud, one bud and one leaf, and one bud and two leaves.

[0071] In some embodiments, to obtain high-quality images of Yinghong No. 9 tea leaves, this study used professional image acquisition equipment to take oblique photos at angles ranging from 15° to 75° to the ground. This range of angles allows for better capture of the details and growth status of the Yinghong No. 9 tea leaves. The data collection was conducted at a local tea research institute, between 9:00 AM and 5:00 PM in summer, which facilitated obtaining images of Yinghong No. 9 tea trees containing Yinghong No. 9 tea leaves under different lighting conditions. Polygonal bounding boxes were annotated on the acquired images using LabelMe annotation software to construct a Yinghong No. 9 tea leaf instance segmentation dataset. After annotation, the dataset was divided into training and validation sets in a 9:1 ratio.

[0072] In some embodiments, after the step of obtaining an image of the Yinghong No. 9 tea tree to be detected, which includes an image of Yinghong No. 9 tea leaves, the method includes:

[0073] In response to the image preprocessing instruction, the image of the Yinghong No. 9 tea tree to be detected is subjected to data augmentation, which includes one or any combination of random illumination enhancement, random scaling, random rotation, random cropping, and horizontal flipping.

[0074] Step S20: Use the Yinghong No. 9 tea leaf instance segmentation model that has been trained to convergence to perform instance segmentation on the Yinghong No. 9 tea tree image to be detected, and obtain the detection box information and mask information of the target area corresponding to the Yinghong No. 9 tea leaves to be picked. The Yinghong No. 9 tea leaves to be picked include one or any combination of single bud, one bud and one leaf, and one bud and two leaves.

[0075] After obtaining the image of the Yinghong No. 9 tea tree to be detected, which contains the image of Yinghong No. 9 tea leaves, the image of the Yinghong No. 9 tea tree to be detected is segmented by the Yinghong No. 9 tea leaf instance segmentation model that has been trained to convergence, so as to obtain the detection box information and mask information of the target area corresponding to the Yinghong No. 9 tea leaves to be picked. The Yinghong No. 9 tea leaves to be picked include one or more of the following: single bud, one bud and one leaf, and one bud and two leaves.

[0076] In some embodiments, please refer to Figure 2The basic network architecture of the Yinghong No. 9 tea leaf instance segmentation model is an improved Mask-RCNN model. The backbone network of the improved Mask-RCNN model is a CSPNEXt network, the neck network is a PAFPN network, and the loss function is a Smooth L1Loss function. It introduces a Coordinating Attention (CAT) mechanism module and a PointRend module.

[0077] Furthermore, the step of using a converged Yinghong No. 9 tea leaf instance segmentation model to perform instance segmentation on the Yinghong No. 9 tea tree image to be detected, and obtaining the detection box information and mask information of the target region corresponding to the Yinghong No. 9 tea leaves to be picked, includes:

[0078] The image of the Yinghong No. 9 tea tree to be detected is input into the Yinghong No. 9 tea leaf instance segmentation model that has been trained to convergence, and the CSPNEXt network in the Yinghong No. 9 tea leaf instance segmentation model is used to extract features from the image of the Yinghong No. 9 tea tree to be detected.

[0079] Step S201: The PAFPN network receives the

[0080] Features extracted by the CSPNeXt network are used to construct a multi-scale feature pyramid.

[0081] Step S203: Use a region generation network to generate candidate regions for the target Yinghong No. 9 tea leaves to be picked from the feature pyramid;

[0082] Step S205: Perform Ro I pooling or Ro I alignment on each candidate region to map candidate regions of different sizes and shapes into feature maps of fixed sizes.

[0083] Step S207: Use the PointRend module to perform instance segmentation on each candidate region to obtain the detection box information and mask information of the target region corresponding to the Yinghong No. 9 tea leaves to be picked.

[0084] Specifically, the backbone network is responsible for extracting feature representations from the input image. In the improved Mask-RCNN model, the CSPNeXt network replaces the original ResNet50 as the backbone network. The CSPNeXt network is the backbone network of RTMDet. Compared with ResNet50, the CSPNeXt network is relatively lightweight and can also improve the network's ability to extract the green features of Yinghong No. 9 tea. The CSPNeXt network extracts three layers of feature maps, which are processed by the Coordinate Attention mechanism and then input into the PAFPN network for feature fusion to generate a feature map.

[0085] Next, a Region Proposal Network (RPN) is used to generate potential target regions, which are predicted rectangular bounding boxes that may contain Yinghong No. 9 tea leaves. Each region has a probability score representing the likelihood that it contains the object;

[0086] For each region, an ROI pooling layer is used. This step resizes regions of different sizes into feature maps of the same size so that they can be input into subsequent network layers.

[0087] The adjusted features are fed into the target classifier and bounding box regressor to classify each region (identify the target category) and accurately locate it (adjust the bounding box to better surround the target);

[0088] For the target regions that pass the classification, the PointRend module in the improved Mask-RCNN model further performs instance segmentation. The PointRend module performs instance segmentation on each candidate region, and the final output includes the detected location of Yinghong No. 9 tea leaves and a pixel-level segmentation mask for the Yinghong No. 9 tea leaves, thereby obtaining the detection box information and mask information of the target region corresponding to the Yinghong No. 9 tea leaves to be picked.

[0089] In some embodiments, the step of training the Yinghong No. 9 tea leaf instance segmentation model includes: training the improved Mask-RCNN model using the SGD optimizer, and using learning rate decay to improve the convergence speed and performance of the model, wherein the initial learning rate is set to 0.001, the weight decay is set to 0.001, the epochs are set to 50, and the batch size is set to 8; inputting the training set of the Yinghong No. 9 tea leaf dataset into the improved Mask-RCNN model for training, updating the network parameters once per epoch; inputting the validation set data into the improved Mask-RCNN model for prediction, evaluating the model accuracy, saving the model file parameters once after every 10 epochs of training, comparing and selecting the model parameters with the highest validation accuracy as the optimal solution, and obtaining the Yinghong No. 9 tea leaf instance segmentation model.

[0090] In some embodiments, the PyTorch framework is used to load pre-trained model files and their parameters, and to construct the corresponding network model. PyTorch provides flexible deep learning tools that can easily load and build complex network structures. Images of Yinghong No. 9 tea leaves to be identified are input into the pre-trained network model. This model, trained on a large amount of labeled data, can accurately identify and segment Yinghong No. 9 tea leaf instances. After processing the input image, the model outputs an image containing the segmentation results of the Yinghong No. 9 tea leaf instances. The output instance segmentation image clearly identifies the boundaries and locations of the Yinghong No. 9 tea leaves, providing a reliable foundation for subsequent automated harvesting.

[0091] Step S30: Input the detection box information of the target area corresponding to the Yinghong No. 9 tea leaves to be picked into the pre-trained Yinghong No. 9 tea leaf picking point localization model, perform feature extraction in the improved RegNet backbone network, and generate a heat map in the upsampling module to indicate the picking point coordinates of the single bud, one bud and one leaf and one bud and two leaves in the Yinghong No. 9 tea tree image.

[0092] After obtaining the detection box information and mask information of the target area corresponding to the Yinghong No. 9 tea leaves to be picked, the detection box information of the target area corresponding to the Yinghong No. 9 tea leaves to be picked is input into the pre-trained Yinghong No. 9 tea leaf picking point localization model. Feature extraction is performed in the improved RegNet backbone network, and a heat map is generated in the upsampling module to indicate the picking point coordinates of the single bud, one bud and one leaf, and one bud and two leaves in the Yinghong No. 9 tea tree image.

[0093] In some embodiments, the basic network architecture of the Yinghong No. 9 tea leaf picking point localization model is a PLNet network. The PLNet network includes an improved RegNet backbone network and an upsampling module. The improved RegNet backbone network includes four stages, each stage containing multiple convolutional layers and bottleneck blocks. The convolutional layers in the latter three stages are deformable convolutions.

[0094] Specifically, the PLNet network includes an improved RegNet backbone network and an upsampling module. The upsampling module is a SimpleBaseline-based upsampling module used for heatmap generation. The improved RegNet backbone network includes four stages, each containing multiple convolutional layers and a bottleneck block. The convolutional layers in the latter three stages are deformable convolutions, which enhance the network's feature extraction capabilities.

[0095] Further, the steps of inputting the detection box information of the target area corresponding to the Yinghong No. 9 tea leaves to be picked into the pre-trained Yinghong No. 9 tea leaf picking point localization model, performing feature extraction in the improved RegNet backbone network, and generating a heatmap in the upsampling module to indicate the picking point coordinates of the single bud, one bud and one leaf, and one bud and two leaves in the Yinghong No. 9 tea tree image include:

[0096] Step S301: In the improved RegNet backbone network of the Yinghong No. 9 tea leaf picking point localization model, extract representative feature maps from the detection box information of the target area corresponding to the Yinghong No. 9 tea leaves to be picked.

[0097] Step S303: In the upsampling module, interpolation or deconvolution techniques are used to increase the spatial resolution of the feature map;

[0098] Step S305: Convert the feature map into a heat map and determine the confidence or probability distribution of the picking point coordinates of each level of Yinghong No. 9 tea leaves in the Yinghong No. 9 tea tree image.

[0099] Step S307: Based on the confidence or probability distribution of the picking point coordinates of Yinghong No. 9 tea leaves at each level, determine the picking point coordinates of single bud, one bud and one leaf, and one bud and two leaves in the Yinghong No. 9 tea tree image.

[0100] In some embodiments, after the step of obtaining an image of the Yinghong No. 9 tea tree to be detected, which includes an image of Yinghong No. 9 tea leaves, the method includes:

[0101] In response to the image preprocessing instruction, the image of the Yinghong No. 9 tea tree to be detected is subjected to data augmentation, which includes one or any combination of random illumination enhancement, random scaling, random rotation, random cropping, and horizontal flipping.

[0102] In some embodiments, training the PLNet network and setting training parameters to train the Yinghong No. 9 tea leaf key point model specifically includes:

[0103] During training, the SGD optimizer was used with an initial learning rate of 0.001, and learning rate decay was used to improve the model's convergence speed and performance. Weight decay was set to 0.001, epochs to 300, and batch size to 16. Data augmentation was performed on the input images during training to increase data diversity and robustness. Data augmentation methods included random scaling, random rotation, random cropping, and horizontal flipping. The network parameters were updated once per epoch. Validation set data was input into the model for prediction, and the model accuracy was evaluated. The model file parameters were saved after every 10 epochs of training, and the model parameters with the highest validation accuracy were selected as the optimal solution. The model file parameters were loaded using PyTorch, and the network model was constructed. First, the image of the Yinghong No. 9 tea leaves to be identified was input into the trained instance segmentation model to obtain the mask and instance segmentation box of the Yinghong No. 9 tea leaves. Then, the image was cropped according to the instance segmentation box corresponding to the Yinghong No. 9 tea leaves to be picked and fed into the keypoint detection model to obtain the coordinates of the picking point.

[0104] Step S40: Obtain the depth information of each mask pixel based on the mask information, and obtain the average depth information of the tea leaves by taking the average value. Guide the picking robot to pick the tea leaves according to grade based on the picking point coordinates and the average depth information, so as to complete the picking point positioning of multi-grade Yinghong No. 9 tea leaves.

[0105] After generating a heatmap in the upsampling module to indicate the picking point coordinates of the single bud, one bud and one leaf, and one bud and two leaves in the Yinghong No. 9 tea tree image, the depth information of each mask pixel is obtained based on the mask information, and the average depth information of the tea leaves is obtained by taking the average value. Based on the picking point coordinates and the average depth information, the picking robot is guided to pick the tea leaves in different grades to complete the picking point positioning of multi-grade Yinghong No. 9 tea leaves.

[0106] In some embodiments, for each specific Yinghong No. 9 tea leaf area, its center point or other specific feature points can be extracted as the coordinates of its picking point. These coordinates can be pixel-level coordinates or physical coordinates in the actual scene (if there is geographical information or camera calibration).

[0107] In some embodiments, please refer to Figures 3 to 10 ,in, Figure 3 This is a schematic diagram of the PointRend module. Figure 4 This is a schematic diagram of the Coordinating Attention (COA) mechanism module. Figure 5 This is a schematic diagram of a PAFPN network. Figure 6 This is a diagram illustrating the picking point labels for Yinghong No. 9 tea leaves of various grades. Figure 7 To determine the overall process flow diagram for the multi-level Yinghong No. 9 tea leaf picking sites, Figure 8 This is a schematic diagram of the PLNet network model. Figure 9 This is a schematic diagram of the upsampling module. Figure 10 This is a schematic diagram of a deformable convolution module.

[0108] As can be seen from the above embodiments, compared with the prior art, this application addresses the problems in the prior art, such as excessively strong or dim lighting, low resolution between tender buds and old leaves, which severely limit the automated harvesting of famous and high-quality teas, as well as the low positioning accuracy and efficiency of the current positioning method for Yinghong No. 9 tea leaf picking points, and its inability to achieve graded harvesting of tea leaves. This application has, but is not limited to, the following beneficial effects:

[0109] Firstly, replacing the backbone network of the Mask-RCNN model with CSPNEXt improves the network's ability to extract features of Yinghong No. 9 tea leaves. Replacing the FPN module with the PAFPN module improves the network's ability to detect Yinghong No. 9 tea leaves of different sizes. Adding a coordinate attention module between the backbone and neck network enhances the network's ability to distinguish Yinghong No. 9 tea leaves, suppressing the background and highlighting the foreground target. Adding a PointRend module to the mask output branch improves the accuracy of the segmentation mask.

[0110] Secondly, the classification loss of the Mask-RCNN model is replaced with the Focal Loss function. Focal Loss can alleviate the class imbalance problem, making the model pay more attention to those samples that are difficult to classify, thereby improving the accuracy and generalization ability of instance segmentation. The bounding box loss of the Mask-RCNN model is replaced with the Smooth L1 Loss function. Since the range of bounding box coordinates is large, the Smooth L1 Loss function can effectively handle the prediction of bounding boxes, improving the accuracy and stability of the detector.

[0111] Third, the joint instance segmentation and key point detection are performed. First, the improved Mask-RCNN model is used to detect Yinghong No. 9 tea leaves at various levels, and then the bounding boxes and masks are output. Then, the PLNet model is used to predict the picking point coordinates of the detected Yinghong No. 9 tea leaves.

[0112] Fourth, the PLNet model can be used to obtain the single bud picking point, one bud and one leaf picking point, and one bud and two leaves picking point of Yinghong No. 9 tea leaves, so as to realize the graded picking of Yinghong No. 9 tea leaves.

[0113] Furthermore, by using the Yinghong No. 9 tea leaf picking point positioning model, the picking points for single buds, one bud and one leaf, and one bud and two leaves of Yinghong No. 9 tea leaves can be obtained, realizing graded picking of Yinghong No. 9 tea leaves and greatly improving the picking efficiency and quality of Yinghong No. 9 tea leaves.

[0114] Please see Figure 11 A multi-level Yinghong No. 9 tea leaf picking point positioning device is provided to meet one of the purposes of this application, including an image acquisition module 1100, a Yinghong No. 9 tea leaf picking determination module 1200, a picking point coordinate determination module 1300, and a picking point positioning module 1400. The image acquisition module is configured to acquire an image of the Yinghong No. 9 tea tree to be detected containing an image of the Yinghong No. 9 tea leaf in response to a multi-level Yinghong No. 9 tea leaf picking point positioning command.

[0115] The module for determining tea leaves to be picked is configured to use a converged Yinghong No. 9 tea leaf instance segmentation model to perform instance segmentation on the Yinghong No. 9 tea tree image to be detected, and obtain the detection box information and mask information of the target area corresponding to the Yinghong No. 9 tea leaves to be picked. The Yinghong No. 9 tea leaves to be picked include one or any combination of single bud, one bud and one leaf, and one bud and two leaves.

[0116] The picking point coordinate determination module is set to input the detection box information of the target area corresponding to the Yinghong No. 9 tea leaves to be picked into the pre-trained Yinghong No. 9 tea leaf picking point localization model, perform feature extraction in the improved RegNet backbone network, and generate a heat map in the upsampling module to indicate the picking point coordinates of the single bud, one bud and one leaf and one bud and two leaves in the Yinghong No. 9 tea tree image.

[0117] The picking point positioning module is configured to obtain the depth information of each mask pixel based on the mask information, and obtain the average depth information of the tea leaves by taking the average value. Based on the picking point coordinates and the average depth information, the picking robot is guided to pick the tea leaves in different grades to complete the picking point positioning of Yinghong No. 9 tea leaves of multiple grades.

[0118] Based on any embodiment of this application, please refer to Figure 12 Another embodiment of this application also provides an electronic device, which can be implemented by a computer device, such as... Figure 12The diagram shows the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium stores an operating system, a database, and computer-readable instructions. The database stores control information sequences. When executed by the processor, the computer-readable instructions enable the processor to implement a multi-level method for locating the picking points of Yinghong No. 9 tea leaves. The processor provides computing and control capabilities to support the operation of the entire computer device. The memory stores computer-readable instructions, which, when executed by the processor, enable the processor to execute the multi-level method for locating the picking points of Yinghong No. 9 tea leaves of this application. The network interface of the computer device is used for communication with a terminal. Those skilled in the art will understand that… Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0119] In this embodiment, the processor is used to execute... Figure 11 The system contains the specific functions of each module and its sub-modules. The memory stores the program code and various data required to execute these modules or sub-modules. The network interface is used for data transmission between the user terminal and the server. In this embodiment, the memory stores the program code and data required to execute all modules / sub-modules in the multi-level Yinghong No. 9 tea leaf picking point positioning device of this application. The server can call the server's program code and data to execute the functions of all sub-modules.

[0120] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the multi-level Yinghong No. 9 tea leaf picking location method described in any embodiment of this application.

[0121] This application also provides a computer program product, including a computer program / instruction that, when executed by one or more processors, implements the steps of the multi-level Yinghong No. 9 tea leaf picking point positioning method described in any embodiment of this application.

[0122] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM).

[0123] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

[0124] In summary, the model for locating picking points of Yinghong No. 9 tea leaves can identify the picking points for single buds, one bud and one leaf, and one bud and two leaves, enabling graded picking of Yinghong No. 9 tea leaves and significantly improving the picking efficiency and quality.

Claims

1. A method for locating the picking points of multi-grade Yinghong No. 9 tea leaves, characterized in that, include: Responding to the picking point location command of Yinghong No. 9 tea leaves at multiple levels, obtain the image of the Yinghong No. 9 tea tree to be detected, which contains the image of Yinghong No. 9 tea leaves; The Yinghong No. 9 tea leaf instance segmentation model, trained to convergence, is used to segment the Yinghong No. 9 tea tree image to be detected, obtaining the detection box information and mask information of the target region corresponding to the Yinghong No. 9 tea leaves to be picked. The Yinghong No. 9 tea leaves to be picked include one or more of the following: single bud, one bud and one leaf, and one bud and two leaves. The basic network architecture of the Yinghong No. 9 tea leaf instance segmentation model is an improved Mask-RCNN model. The backbone network of the improved Mask-RCNN model is a CSPNEXt network, the neck network is a PAFPN network, and the loss function is a Smooth L1 Loss function. It introduces a Coordinate Attention module and a PointRend module. The detection bounding box information of the target area corresponding to the unharvested Yinghong No. 9 tea leaves is input into a pre-trained Yinghong No. 9 tea leaf picking point localization model. Feature extraction is performed in the improved RegNet backbone network. A heatmap is generated in the upsampling module to indicate the picking point coordinates of the single bud, one bud and one leaf, and one bud and two leaves in the Yinghong No. 9 tea tree image. This includes: extracting representative feature maps from the detection bounding box information of the target area corresponding to the unharvested Yinghong No. 9 tea leaves in the improved RegNet backbone network of the Yinghong No. 9 tea leaf picking point localization model; in the upsampling module, interpolation or deconvolution techniques are used to increase the spatial resolution of the feature maps; and the feature maps are then processed. The image is converted into a heatmap to determine the confidence or probability distribution of the picking point coordinates of Yinghong No. 9 tea leaves at each level in the Yinghong No. 9 tea tree image; based on the confidence or probability distribution of the picking point coordinates of Yinghong No. 9 tea leaves at each level, the picking point coordinates of single bud, one bud and one leaf, and one bud and two leaves in the Yinghong No. 9 tea tree image are determined; wherein, the basic network architecture of the Yinghong No. 9 tea leaf picking point localization model is a PLNet network, which includes an improved RegNet backbone network and an upsampling module, wherein the improved RegNet backbone network includes four stages, each stage containing multiple convolutional layers and bottleneck blocks, and the convolutional layers of the last three stages are deformable convolutions; The depth information of each mask pixel is obtained based on the mask information, and the average depth information of the tea leaves is obtained by taking the average value. Based on the picking point coordinates and the average depth information, the picking robot is guided to pick the tea leaves in different grades, so as to complete the picking point positioning of Yinghong No. 9 tea leaves in multiple grades.

2. The method for locating the picking point of multi-grade Yinghong No. 9 tea leaves according to claim 1, characterized in that, The steps of using a converged Yinghong No. 9 tea leaf instance segmentation model to perform instance segmentation on the image of the Yinghong No. 9 tea tree to be detected, and obtaining the detection box information and mask information of the target region corresponding to the Yinghong No. 9 tea leaves to be picked, include: The image of the Yinghong No. 9 tea tree to be detected is input into the Yinghong No. 9 tea leaf instance segmentation model that has been trained to convergence, and the feature extraction of the image of the Yinghong No. 9 tea tree to be detected is performed based on the backbone network in the Yinghong No. 9 tea leaf instance segmentation model. The PAFPN network receives features extracted by the backbone network and constructs a multi-scale feature pyramid. A region generation network is used to generate candidate regions for the target Yinghong No. 9 tea leaves to be picked from the feature pyramid; For each candidate region, RoI pooling or RoI alignment is performed to map candidate regions of different sizes and shapes to feature maps of fixed sizes. The PointRend module is used to segment each candidate region to obtain the detection box information and mask information of the target region corresponding to the Yinghong No. 9 tea leaves to be picked.

3. The method for locating the picking point of multi-grade Yinghong No. 9 tea leaves according to claim 1, characterized in that, The steps for training the Yinghong No. 9 tea leaf instance segmentation model include: The improved Mask-RCNN model was trained using the SGD optimizer, with the initial learning rate set to 0.001, weight decay set to 0.001, epochs set to 50, and batch size set to 8. The training set of Yinghong No. 9 tea leaf dataset is input into the improved Mask-RCNN model for training, and the network parameters are updated once per epoch. The validation set data is input into the improved Mask-RCNN model for prediction, and the model accuracy is evaluated. The model file parameters are saved after every 10 eopchs are trained. The model parameters with the highest validation accuracy are selected as the optimal solution to obtain the Yinghong No. 9 tea leaf instance segmentation model.

4. The method for locating the picking point of multi-grade Yinghong No. 9 tea leaves according to any one of claims 1 to 3, characterized in that, After obtaining the image of the Yinghong No. 9 tea tree to be detected, which includes an image of the Yinghong No. 9 tea leaves, the process includes: In response to the image preprocessing instruction, the image of the Yinghong No. 9 tea tree to be detected is subjected to data augmentation, which includes one or any combination of random illumination enhancement, random scaling, random rotation, random cropping, and horizontal flipping.

5. A device for locating the picking point of multi-grade Yinghong No. 9 tea leaves, characterized in that, include: The image acquisition module is configured to respond to the picking point positioning command of multi-level Yinghong No. 9 tea leaves and acquire the image of the Yinghong No. 9 tea tree to be detected, which contains the image of Yinghong No. 9 tea leaves. The module for determining tea leaves to be picked is configured to use a converged Yinghong No. 9 tea leaf instance segmentation model to segment the image of the Yinghong No. 9 tea tree to be detected, and obtain the detection box information and mask information of the target region corresponding to the Yinghong No. 9 tea leaves to be picked. The Yinghong No. 9 tea leaves to be picked include one or more of the following: single bud, one bud and one leaf, and one bud and two leaves. The basic network architecture of the Yinghong No. 9 tea leaf instance segmentation model is an improved Mask-RCNN model. The backbone network of the improved Mask-RCNN model is a CSPNEXt network, the neck network is a PAFPN network, and the loss function is a Smooth L1 Loss function. It introduces a Coordinate Attention module and a PointRend module. The picking point coordinate determination module is configured to input the detection box information of the target area corresponding to the Yinghong No. 9 tea leaves to be picked into a pre-trained Yinghong No. 9 tea leaf picking point localization model, perform feature extraction in an improved RegNet backbone network, and generate a heatmap in the upsampling module to indicate the picking point coordinates of the single bud, one bud and one leaf, and one bud and two leaves in the Yinghong No. 9 tea tree image. This includes: extracting representative feature maps from the detection box information of the target area corresponding to the Yinghong No. 9 tea leaves in the improved RegNet backbone network of the Yinghong No. 9 tea leaf picking point localization model; and using interpolation or deconvolution techniques in the upsampling module to increase the spatial resolution of the feature maps. The feature map is converted into a heatmap to determine the confidence or probability distribution of the picking point coordinates of Yinghong No. 9 tea leaves at each level in the Yinghong No. 9 tea tree image. Based on the confidence or probability distribution of the picking point coordinates of Yinghong No. 9 tea leaves at each level, the picking point coordinates of single bud, one bud and one leaf, and one bud and two leaves in the Yinghong No. 9 tea tree image are determined. The basic network architecture of the Yinghong No. 9 tea leaf picking point localization model is a PLNet network, which includes an improved RegNet backbone network and an upsampling module. The improved RegNet backbone network includes four stages, each containing multiple convolutional layers and bottleneck blocks. The convolutional layers in the last three stages are deformable convolutions. The picking point positioning module is configured to obtain the depth information of each mask pixel based on the mask information, and obtain the average depth information of the tea leaves by taking the average value. Based on the picking point coordinates and the average depth information, the picking robot is guided to pick the tea leaves in different grades to complete the picking point positioning of Yinghong No. 9 tea leaves of multiple grades.

6. An electronic device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, It stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 4, which, when invoked by a computer, executes the steps included in the corresponding method.

Images