Tea bud picking point positioning method based on fusion of thermal images and RGB images

Abstract

The application discloses a tea bud picking point positioning method based on fusion of thermal images and RGB images, and comprises the following steps: firstly, a plurality of tea bud image pairs of tea buds of tea trees are acquired by using an image acquisition device, each of the tea bud image pairs is labeled, and a tea bud target detection database and a tea bud key point detection database are obtained; secondly, the tea bud target detection database and the tea bud key point detection database are respectively input into a target detection model and a key point model based on a deep convolutional neural network for training, and a target detection model and a key point detection model are respectively obtained; finally, the tea bud image pairs are processed in sequence by using the trained target detection model and key point detection model to obtain the positions of key points of the tea buds, the tea buds of tea trees are positioned at picking points in combination with the growth characteristics of the tea buds, and the positions of the picking points of the tea buds are obtained. The application improves the precision and efficiency of the tea bud picking point positioning.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence recognition, machine vision and image processing technology, and in particular to a method for locating tea bud picking points based on the fusion of thermal images and RGB images. Background Art

[0002] Tea is primarily harvested by hand and by machine. Manual plucking is selective and produces high-quality tea. Humans can determine the suitability and grade of leaf buds by observing their shape, color, and other characteristics. Tea buds are then broken off and plucked at designated locations. This method ensures the integrity of the buds and results in higher quality, but it is also costly. Due to industrial restructuring and labor shifts, a labor shortage often occurs during the tea-picking season.

[0003] In recent years, vision-based automated picking robots have been used to pick premium teas, and the automatic identification and positioning of picking points has become a key and difficult point restricting their development. Premium teas are lightweight, and wind or the movement of the picking machine can cause the tea leaves to sway; the tea garden environment is complex, and the tea leaves block each other; the light is too strong or too dark, resulting in low resolution between young buds and old leaves. These factors make the identification and positioning of the tender bud picking points very difficult, severely limiting the automated picking of premium teas. The current method for locating tea bud picking points has low positioning accuracy and efficiency. In order to achieve rapid identification and positioning of picking points and ensure the efficiency and high quality of premium tea machine picking, it is necessary to develop a method for obtaining the location information of the picking points of premium teas. Summary of the Invention

[0004] The main purpose of the present invention is to propose a tea bud picking point positioning method based on the fusion of thermal images and RGB images, aiming to improve the accuracy and efficiency of tea bud picking point positioning.

[0005] To achieve the above object, the present invention provides a method for locating tea bud picking points based on the fusion of thermal images and RGB images. The method for locating tea bud picking points based on the fusion of thermal images and RGB images comprises the following steps:

[0006] Acquire a plurality of tea bud image pairs of tea tree buds, wherein the tea bud image pairs include a tea bud thermal image and a tea bud RGB image;

[0007] Labeling each pair of tea bud images to obtain a tea bud target detection database and a tea bud key point detection database;

[0008] Inputting the sprout target detection database into a target detection model based on a deep convolutional neural network for training to obtain a trained target detection model;

[0009] Inputting the sprout key point detection database into the key point detection model for training to obtain a trained key point detection model;

[0010] Using an image acquisition device to capture an image pair of young tea buds to be picked;

[0011] According to the trained target detection model, key point detection model and image pairs, the key point positions of the sprouts are obtained;

[0012] The tea tree buds are picked at locations according to the bud growth characteristics and the locations of the key points of the buds to obtain the locations of the bud picking points.

[0013] Optionally, the step of labeling each pair of tea bud images to obtain a tea bud target detection database and a tea bud key point detection database includes:

[0014] Each pair of tea bud images is labeled according to the formats of input data of two types of tasks, namely, target detection and key point detection, to obtain target labeling data and key point labeling data corresponding to the tea bud image pair;

[0015] Augmentation operations are performed on the target annotation data and the key point annotation data respectively to obtain a tea bud target detection database and a tea bud key point detection database.

[0016] Optionally, the step of inputting the sprout target detection database into a target detection model based on a deep convolutional neural network for training to obtain the target detection model includes:

[0017] Dividing the sprout target detection database into a target detection training set and a target detection verification set according to a first preset ratio;

[0018] Inputting the target detection training set into a target detection network model based on a deep convolutional neural network for training to obtain a target detection model weight file;

[0019] Load the trained key point detection model weight file into the key point detection model to obtain the trained object detection model.

[0020] Optionally, the step of inputting the sprout key point detection database into the key point detection model for training to obtain the key point detection model includes:

[0021] Dividing the sprout key point detection database into a key point training set and a key point verification set according to a second preset ratio;

[0022] Inputting the key point training set into the key point detection model for training to obtain a key point detection model weight file;

[0023] The trained key point detection model weight file is loaded into the key point detection model to obtain the trained key point detection model.

[0024] Optionally, the step of inputting the key point training set into a key point detection model for training to obtain a tea key point detection model weight file includes:

[0025] An improved key point detection model is established based on the HRNet network using the pytorch program. The improved key point detection model is to improve the input layer of the key point detection model to 4 channels for inputting the image obtained by fusion of the thermal image and the RGB image;

[0026] The key point training set is input into the key point detection model in batches and continuously and iteratively. When the model converges, the training of the key point detection model is completed;

[0027] According to the trained key point detection model, the key point detection model weight file is obtained.

[0028] Optionally, the loss function of the key point detection model is:

[0029] min L=λ1*L1+λ2*L2+λ3*L3;

[0030]

[0031] L2=d(P1,C);

[0032] L3=d(P2,C);

[0033] Among them, λ1, λ2, λ3 are the weight coefficients of L1, L2, L3; Pk represents the kth key point of the sample, Represents the key point heat map predicted by the network, y(P k ) represents the heat map obtained from the true value; d(P1,C) represents the Euclidean distance from the key point of the tea bud predicted by the network to the center C of the rectangular area where it is located.

[0034] Optionally, the step of obtaining the positions of key points of the sprouts based on the trained target detection model, key point detection model and image pairs includes:

[0035] The trained object detection model is used to perform object detection on the image pairs to obtain the tea bud detection results;

[0036] The trained key point detection model is used to perform key point detection on the tea bud detection results to obtain the key point positions.

[0037] Optionally, the step of locating the tea tree bud picking point based on the bud growth characteristics to obtain the bud picking point position includes:

[0038] When there are two key points of the bud, d is the Euclidean distance between the two key points. Use P1 and P2 to establish a straight line equation, and take a point on the line segment P1P2 that is 0.4d away from point P1 as the tea bud picking point;

[0039] When the key point of the tender bud is a point, the image segmentation is performed using the position and region growing algorithm based on P1 or P2 to obtain a binary image of the tea bud branches. The noise therein is filtered out through expansion and corrosion manipulation, and the binary image is fitted using a straight line fitting method. The key point is offset downward on the straight line by a preset distance to determine the position of the tea bud picking point.

[0040] This invention provides a method for locating tea bud picking points based on the fusion of thermal and RGB images. This method can determine the picking points based on the growth posture of the tea buds, ensuring that the coordinates of the located picking points all fall on the petioles of the tea buds. This improves the integrity of the picked tea buds, reduces the impact of the surrounding environment on the positioning, and improves the accuracy and efficiency of tea bud picking point positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of a first embodiment of a method for locating tea bud picking points based on the fusion of thermal images and RGB images according to the present invention;

[0042] Figure 2 The figure is a flow chart of the tea bud picking point positioning method based on the fusion of thermal images and RGB images of the present invention.

[0043] Figure 3 Schematic diagram of key point positions of the tea bud picking point positioning method based on the fusion of thermal image and RGB image of the present invention;

[0044] Figure 4 This is a thermal image of tea buds taken by the tea bud picking point positioning method based on the fusion of thermal images and RGB images of the present invention;

[0045] Figure 5 This is an RGB image of tea buds taken by the tea bud picking point positioning method based on the fusion of thermal images and RGB images of the present invention;

[0046] Figure 6 This is the effect diagram after the target detection model is executed on the thermal image and RGB image of tea buds in the present invention;

[0047] Figure 7 This is the effect diagram after executing the key point detection model on the thermal image and RGB image of tea buds;

[0048] Figure 8 This is a flow chart of step S71 of the method for locating tea bud picking points based on the fusion of thermal images and RGB images of the present invention;

[0049] Figure 9 This is a flowchart of step S72 of the method for locating tea bud picking points based on the fusion of thermal images and RGB images of the present invention.

[0050] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0051] Reference Figure 1 , Figure 1 This is a flow chart of the first embodiment of the method for locating tea bud picking points based on the fusion of thermal images and RGB images of the present invention.

[0052] In an embodiment of the present invention, the tea bud picking point positioning method based on the fusion of thermal images and RGB images is applied to a bud picking point positioning device, and the method includes:

[0053] Step S10, obtaining a plurality of tea bud image pairs of tea tree buds, wherein the tea bud image pairs include a tea bud thermal image and a tea bud RGB image;

[0054] In this embodiment, in order to improve the accuracy and efficiency of tea bud picking point positioning, the bud picking point positioning device uses image acquisition equipment to shoot several pairs of tea bud images from the side of the buds perpendicular to the height close to the tea buds, so as to realize tea bud target detection data collection and tea bud key point detection data collection; the tea bud image pair includes a tea bud thermal image and a tea bud RGB image, and the tea bud thermal image and the tea bud RGB image are side views of the tea buds, wherein the tea bud RGB image is a color picture of the tea buds, which is an image composed of the three primary colors of red, green and blue.

[0055] Step S20, labeling each pair of tea bud images to obtain a tea bud target detection database and a tea bud key point detection database;

[0056] In this embodiment, after obtaining a plurality of tea bud image pairs of tea tree buds, the bud picking point positioning device marks each pair of tea bud image pairs to obtain a tea bud target detection database and a tea bud key point detection database.

[0057] Step S20 labels each pair of tea bud images to obtain a tea bud target detection database and a tea bud key point detection database, which may include:

[0058] Step S21, annotating each pair of tea bud images according to the formats of target detection and key point detection task input data, respectively, to obtain target annotation data and key point annotation data corresponding to the tea bud image pair;

[0059] In this embodiment, after obtaining a number of tea bud image pairs of tea tree buds, the bud picking point positioning device labels each pair of tea bud image pairs according to the formats of two types of task input data: target detection and key point detection, and obtains target labeling data and key point labeling data corresponding to the tea bud image pairs. Among them, the labeling method for target detection refers to labeling the collected data according to the target detection task; the labeling method for key point detection refers to labeling the collected data according to the key point detection task; for target detection, the rectangular area where the tea buds are located on the image is labeled, and the labeling result of each tea bud is the coordinates of the upper left corner and the lower right corner of the rectangular area {(x1, y1), (x2, y2)}; for key point detection, two key points on the tea buds are labeled, the tea buds have one bud and one leaf, and the bud and leaf connection point P1, and the connection point P2 between the bud and the branch corresponding to one bud and two leaves. The labeling result of each group of buds is the information of the two key points P1 and P2 {(Px1, Py1, V1), (Px2, Py2, V)}, where the first two numbers are the position coordinate information, and the third number represents the visibility flag, which is 0 for unlabeled, 1 for labeled but invisible, and 2 for labeled and visible.

[0060] Step S22 , performing augmentation operations on the target annotation data and the key point annotation data respectively to obtain a tea bud target detection database and a tea bud key point detection database.

[0061] In this embodiment, after obtaining the target annotation data and key point annotation data, the bud picking point positioning device performs augmentation operations on the target annotation data and the key point annotation data to expand the dataset samples and construct a tea bud target detection database Do and a tea bud key point detection database Dk, respectively. The augmentation operations include image translation, rotation, sharpening, flipping, scaling, and other operations.

[0062] Step S30, inputting the sprout target detection database into a target detection model based on a deep convolutional neural network for training to obtain a target detection model;

[0063] In this embodiment, after obtaining the tea bud target detection database and the tea bud key point detection database, the bud picking point positioning device inputs the bud target detection database into a target detection model based on a deep convolutional neural network for training to obtain a trained target detection model.

[0064] Step S30 inputs the sprout target detection database into a deep convolutional neural network model for training to obtain target detection model weights, which may include:

[0065] Step S31, dividing the sprout target detection database into a target detection training set and a target detection verification set according to a first preset ratio;

[0066] In this embodiment, after obtaining the tea bud target detection database and the tea bud key point detection database, the bud picking point positioning device divides the bud target detection database into a target detection training set and a target detection verification set according to a first preset ratio; wherein the first preset ratio is 8:2.

[0067] Step S32, training the target detection training set through a target detection model based on a deep convolutional neural network to obtain a target detection model weight file;

[0068] In this embodiment, after obtaining a target detection training set, the bud picking point positioning device trains the target training set through a deep convolutional neural network model to obtain a target detection model weight file. The trained deep convolutional neural network model is then verified using a target validation set. The deep convolutional neural network model is a YOLOV5m model.

[0069] In this embodiment, after obtaining the target detection model weight file and the target detection model, the bud picking point positioning device obtains the target detection model based on the target detection model and the target detection model weight file. The target detection model and the target detection model weight file corresponding to the target detection model together constitute the target detection model Fo, which can obtain the coordinates of n tea bud rectangular areas and corresponding confidence scores on the image:

[0070]

[0071] Where ((xn,yn),(xn,yn),sn) represents the coordinates of one of the rectangular regions and the corresponding confidence score.

[0072] Step S33: Load the trained target detection model weight file into the target detection model to obtain the trained target detection model.

[0073] In this embodiment, after obtaining the target detection model weight file, the tender shoot picking point positioning device loads the trained target detection model weight file into the target detection model to obtain the trained target detection model.

[0074] Step S40, inputting the sprout key point detection database into the key point detection model for training to obtain a key point detection model;

[0075] In this embodiment, after obtaining the tea bud target detection database and the tea bud key point detection database, the bud picking point positioning device inputs the bud key point detection database into the key point detection model for training to obtain the key point detection model.

[0076] Step S40 inputs the sprout key point detection database into the key point detection model for training to obtain the key point detection model, which may include:

[0077] Step S41, dividing the sprout key point detection database into a key point training set and a key point verification set according to a second preset ratio;

[0078] In this embodiment, after obtaining the tea bud target detection database and the tea bud key point detection database, the bud picking point positioning device divides the bud key point detection database into a key point training set and a key point verification set according to a second preset ratio; wherein the second preset ratio is 8:2.

[0079] Step S42: inputting the key point training set into a key point detection model for training to obtain a key point detection model weight file;

[0080] In this embodiment, after obtaining the key point training set, the tender sprout picking point positioning device trains the key point training set through a key point detection model to obtain a key point detection model weight file.

[0081] Step S42 inputs the key point training set into the key point detection model for training to obtain a tea key point detection model weight file, which may include:

[0082] Step S421: Using the pytorch program to establish a key point detection model based on the HRNet network, the improved key point detection model is to improve the input layer of the key point detection model to 4 channels for inputting the image after the thermal image and the RGB image are fused;

[0083] In this embodiment, after obtaining the key point training set, the tender bud picking point positioning device establishes a key point detection model based on the HRNet network using the pytorch program. The improved key point detection model is to improve the input layer of the key point detection model to 4 channels for inputting the image after the fusion of the thermal image and the RGB image. Among them, the key point detection model is based on the HRNet network, and the model output is 2 feature maps, which represent the prediction maps of the key points P1 and P2 respectively, and the loss function of the key point detection model redesigned using the key point features of the tea tender buds is: min L = λ1*L1+λ2*L2+λ3*L3;

[0084]

[0085] L2=d(P1,C);

[0086] L3=d(P2,C);

[0087] Among them, λ1, λ2, λ3 are the weight coefficients of L1, L2, L3; Pk represents the kth key point of the sample, Represents the key point heat map predicted by the network, y(P k ) represents the heat map obtained from the true value; d(P1, C) represents the Euclidean distance between the tea bud key point predicted by the network and the center C of the rectangular region where it is located. The loss function of the key point detection model is used for training the key point detection model.

[0088] Step S422, inputting the key point training set into the key point detection model in batches and continuously iteratively, and when the model converges, the training of the key point detection model is completed, and a key point detection model weight file is obtained;

[0089] In this embodiment, after establishing the key point detection model, the bud picking point positioning device inputs the key point training set into the key point detection model in batches and continuously iteratively. As the iteration proceeds, the model gradually converges, and the training of the key point detection model is eventually completed, and the key point detection model weight file is obtained. During the training process, when the loss function of the training set tends to be stable and the decline is small, the model is considered to have converged, that is, the training is completed. In this model, when the difference between the loss functions of three consecutive epochs is less than 0.1 during the training process, the training is stopped.

[0090] An iterative process in model training involves forward propagation and backpropagation. During forward propagation, a batch of image data samples from the training set is input into the keypoint detection model. This data is then propagated backward through the model's convolutional layers, ultimately yielding the model's output. This output is then compared against the label file to calculate the model's loss function. During backpropagation, this loss function is used to calculate the partial derivatives of the parameters in the convolutional layers, and gradients are then backpropagated to update the weight parameters of each convolutional layer, thereby adjusting the model's training.

[0091] The key point detection model and the corresponding weight file together constitute the function Fk, which can obtain the position of the key points P1 and P2;

[0092]

[0093] Step S43: Load the trained key point detection model weight file into the key point detection model to obtain the trained key point detection model.

[0094] In this embodiment, after obtaining the key point detection model weight file, the tender shoot picking point positioning device loads the trained key point detection model weight file into the key point detection model to obtain the trained key point detection model; and verifies the key point detection model through the key point verification set.

[0095] Step S50, using an image acquisition device to capture an image pair of young tea buds to be picked;

[0096] In this embodiment, after obtaining the trained target detection model and key point detection model, the bud picking point positioning device uses an image acquisition device to capture an image pair I of the tea buds to be picked. The image pair is a thermal image and an RGB image of the tea buds to be picked.

[0097] Step S60, obtaining the key point positions of the sprouts based on the trained target detection model, key point detection model and image pairs;

[0098] In this embodiment, after obtaining the image pair I, the tender shoot picking point positioning device obtains the tender shoot key point position based on the target detection model, the key point detection model and the image pair.

[0099] Step S60 obtains the key point positions of the sprouts based on the trained target detection model, key point detection model, and image pairs, which may include:

[0100] Step S61, performing target detection on the image pair using the trained target detection model to obtain a tea bud detection result;

[0101] In this embodiment, after obtaining the image pair I, the bud picking point positioning device performs target detection on the image pair using the trained target detection model to obtain the tea bud detection result. Figure 6 As shown in the figure, the target detection model and the corresponding target detection model weight file are used to form the target detection model Fo to perform target detection on the image pair, obtain the corresponding tea bud detection frame, and crop the image in the detection frame and adjust it to a fixed resolution (128*224) to obtain the tea bud detection result; wherein, Figure 6 This is the effect diagram after the present invention executes the target detection model on the thermal image and RGB image of tea buds. For the convenience of visualization, only the detected bud part is displayed on the RGB image of tea buds.

[0102] Step S62: Perform key point detection on the tea bud detection results using the trained key point detection model to obtain key point positions.

[0103] In this embodiment, after obtaining the tea bud detection results, the bud picking point positioning device performs key point detection on the tea bud detection image pair using the trained key point detection model to obtain the key point positions. Figure 7 As shown, the key point detection model Fk is constructed by using the key point detection model and the corresponding key point detection model weight file to detect the key points of the tea bud detection results obtained in the tea bud detection frame; Figure 7 This figure shows the effect of applying the keypoint detection model to the thermal image and RGB image of tea buds. For ease of visualization, only the detected tea buds are shown on the RGB image. The keypoint detection model and the corresponding keypoint detection weight file together form the function Fk, which can be used to obtain the positions of keypoints P1 and P2.

[0104]

[0105] Step S70, positioning the tea tree bud picking point based on the bud growth characteristics and the bud key point positions to obtain the bud picking point positions.

[0106] In this embodiment, after obtaining the position of the key point of the tender bud, the tender bud picking point positioning device locates the tea tree tender bud picking point according to the tender bud growth characteristics and the position of the key point of the tender bud to obtain the position of the tender bud picking point.

[0107] Step S70 locates the tea tree bud picking point based on the bud growth characteristics and the bud key point positions to obtain the bud picking point positions, which may include:

[0108] Step S71: When the bud key points are two points, Figure 8 As shown, that is, P1 and P2 exist at the same time, at this time d is the Euclidean distance between the two key points, and the straight line equation is established using P1 and P2. According to the characteristics that the tender bud picking point is always below the key point and close to the key point, combined with the key point position and the fitting straight line equation, the point on the line segment P1P2 with a distance of 0.4d from point P1 is taken as the tender bud picking point position of the tea leaves. Figure 8 This is a flowchart of step S71 of the method for locating tea bud picking points based on the fusion of thermal images and RGB images of the present invention. For the sake of visualization, the figure is only displayed on the tea bud RGB image. The square points in the figure represent the key points of the tea buds detected by the model, and the triangular points represent the tea bud picking points located by the present invention.

[0109] Step S72, when the bud key point is one point, Figure 9As shown in the figure, that is, if there is only one P1 or P2, the position and region growing algorithm based on P1 or P2 is used to perform image segmentation to obtain a binary image of the tea bud branches. The noise is filtered out through expansion and corrosion manipulation, and the binary image is fitted by the straight line fitting method. Finally, according to the characteristics that the bud picking point is always below the key point and close to the key point, combined with the key point position and the fitting straight line equation, the key point is offset downward by a certain distance on the straight line to determine the position of the tea bud picking point, where the offset value is 5 pixels. Figure 9 This is a flow chart of step S72 of the method for locating tea bud picking points based on the fusion of thermal images and RGB images of the present invention. For ease of visualization, the figure is only displayed on the tea bud RGB image. The uppermost and lowermost square points in the figure represent the key tea bud picking points detected by the present invention; the triangular points represent the tea bud picking points located by the present invention.

[0110] Step S73: When the number of key points of the young sprout obtained is zero, that is, no key points are detected, no subsequent operation is performed at this time, and the next set of image pairs is calculated.

[0111] Through the above scheme, this embodiment determines the picking points according to the growth posture of the tea buds, ensuring that the coordinates of the located picking points all fall on the petioles of the tea buds, thereby improving the integrity of the picked tea buds, while reducing the impact of the surrounding environment on its positioning, and improving the accuracy and efficiency of the positioning of the tea bud picking points.

[0112] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0113] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0114] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0115] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for locating tea bud picking points based on the fusion of thermal images and RGB images, characterized by: The method comprises the following steps: Acquire a plurality of tea bud image pairs of tea tree buds, wherein the tea bud image pairs include a tea bud thermal image and a tea bud RGB image; Labeling each pair of tea bud images to obtain a tea bud target detection database and a tea bud key point detection database; Inputting the sprout target detection database into a target detection model based on a deep convolutional neural network for training, to obtain a trained target detection model, comprising: Dividing the sprout target detection database into a target detection training set and a target detection verification set according to a first preset ratio; Inputting the target detection training set into a target detection model based on a deep convolutional neural network for training to obtain a target detection model weight file; Load the trained target detection model weight file into the target detection model to obtain the trained target detection model; The sprout key point detection database is input into the key point detection model for training, and the trained key point detection model is obtained, including: Dividing the sprout key point detection database into a key point training set and a key point verification set according to a second preset ratio; Inputting the key point training set into the key point detection model for training to obtain a key point detection model weight file; An improved key point detection model is established based on the HRNet network using the pytorch program. The improved key point detection model is to improve the input layer of the key point detection model to 4 channels for inputting the image after the thermal image and the RGB image are fused; The key point training set is input into the key point detection model in batches and continuously and iteratively. When the model converges, the training of the key point detection model is completed, and a key point detection model weight file is obtained; Load the trained key point detection model weight file into the key point detection model to obtain the trained key point detection model; The loss function of the key point detection model is: ; ; ; ; in, 1.

2. 3 is the weight coefficient of L1, L2, L3; P k represents the kth key point of the sample, ŷ(P k ) represents the key point heat map predicted by the network, y(P k ) represents the heat map obtained by the true value; d(P1, C) represents the Euclidean distance from the key point of the tea bud predicted by the network to the center C of the rectangular area where it is located; Using an image acquisition device to capture an image pair of young tea buds to be picked; According to the trained target detection model, key point detection model and image pairs, the key point positions of the sprouts are obtained; Positioning the tea tree bud picking point based on the bud growth characteristics and the bud key point positions to obtain the bud picking point positions; The step of locating the tea tree bud picking point based on the bud growth characteristics and the bud key point positions to obtain the bud picking point positions includes: When there are two key points for the buds, d is the Euclidean distance between the two key points. Use P1 and P2 to establish a straight line equation, and take the point on the line segment P1P2 that is 0.4 d away from point P1 as the tea bud picking point; When the key point of the tea bud is a single point, image segmentation is performed using a position and region growing algorithm based on P1 or P2 to obtain a binary image of the tea bud branches. The noise is filtered out through dilation and corrosion manipulation, and the binary image is fitted using a straight-line fitting method. The key point is offset downward by a preset distance on the straight line to determine the position of the tea bud picking point.

2. The method for locating tea bud picking points based on the fusion of thermal images and RGB images according to claim 1 is characterized in that: The step of labeling each pair of tea bud images to obtain a tea bud target detection database and a tea bud key point detection database includes: Each pair of tea bud images is labeled according to the formats of input data of two types of tasks, namely, target detection and key point detection, to obtain target labeling data and key point labeling data corresponding to the tea bud image pair; Augmentation operations are performed on the target annotation data and the key point annotation data respectively to obtain a tea bud target detection database and a tea bud key point detection database.

3. The method for locating tea bud picking points based on the fusion of thermal images and RGB images according to claim 1 is characterized in that: The step of obtaining the key point positions of the sprouts based on the trained target detection model, the key point detection model and the image pair comprises: The trained object detection model is used to perform object detection on the image pairs to obtain the tea bud detection results; The trained key point detection model is used to perform key point detection on the tea bud detection results to obtain the key point positions.

Images