A method and system for extracting three-dimensional coordinates of cotton top buds based on key point detection

Abstract

The application provides a cotton top bud three-dimensional coordinate extraction method and system based on key point detection. The method comprises the following steps: collecting a cotton plant image sample, and pre-processing the cotton plant image; labeling the position of the cotton bud on the pre-processed image sample, and dividing the labeled image sample into a training set and a verification set; training a deep learning network by using the training set; processing the labeled image sample in the verification set by using the trained deep learning network, and obtaining the position of the two-dimensional key point of the cotton top bud; and extracting the three-dimensional coordinate of the cotton top bud by using a ROS-based three-dimensional coordinate extraction method according to the position of the two-dimensional key point of the cotton top bud. The scheme provided by the application significantly improves the problems of low recognition accuracy of top end recognition and incoherence of the target detection model which can only output two-dimensional coordinates and is not conducive to mechanical operation. Therefore, the accuracy and efficiency of cotton top bud recognition are effectively improved, and accurate positioning is provided for subsequent mechanical topping.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of deep learning target detection, and particularly relates to a cotton top bud three-dimensional coordinate extraction method and system based on key point detection. BACKGROUND

[0002] Cotton has the characteristics of unlimited growth, and cotton topping technology can inhibit the growth of the top bud, thereby preventing the overdevelopment of the top of the cotton, allowing more nutrients to be supplied to the growth of the reproductive organs, thereby promoting the cotton to form more bolls and fruits, and thus improving the quality and yield of the cotton. The topping method also has an important influence on the yield and quality of cotton. It is also crucial to choose the right topping method.

[0003] The current main topping methods include manual topping, mechanical topping and chemical topping. Topping is usually carried out from late July to early August, and is achieved by removing a leaf and a top bud at the top of the cotton plant. Manual topping is a method in which a cotton farmer with rich cotton planting experience manually pinches off the top bud. This method has high accuracy, but requires a large amount of labor and high labor costs. The low efficiency is gradually being replaced by mechanical and chemical topping. Chemical topping is the use of plant top growth inhibitors, such as fluridone and mepiquat chloride, to inhibit the growth of the top bud of the cotton. Chemical inhibitors have been proven to have good topping effect, but chemical topping is not yet mature in domestic use, and improper use poses the risk of reduced cotton yield and large-scale use can easily cause environmental pollution. Mechanical topping is the use of agricultural machinery equipped with a cutting knife to cut the top of the cotton, which is much more efficient than manual topping, but the accuracy is not high, and it is easy to cause missed topping, incorrect topping, peach damage, and damage to side branches and leaves. Therefore, mechanical topping is still only used for testing in some areas and has not been widely used nationwide.

[0004] The key technology in cotton mechanical topping is to accurately identify the cotton top bud. In recent years, deep learning target detection methods have been widely used in agriculture. Existing cotton top bud intelligent recognition technology uses deep learning target detection technology to accurately identify the cotton top bud. In the early preparation stage, the collected cotton top bud images are trained using a target detection model to form a unique image database. Then, the top bud images captured by the binocular camera are compared with the images in the database, and the deep learning target detection model is used to accurately identify the top bud.

[0005] Defects of the prior art

[0006] The existing deep learning target detection model mainly relies on a data set made of high-quality images taken directly above the cotton top bud. In addition, the deep learning target detection intelligence identifies the cotton top bud on a two-dimensional image and cannot convert it into a three-dimensional coordinate to provide the required information for mechanical topping. SUMMARY

[0007] To solve the above technical problems, the application provides a key point detection-based cotton top bud three-dimensional coordinate extraction method.

[0008] The first aspect of the application discloses a key point detection-based cotton top bud three-dimensional coordinate extraction method, which comprises the following steps:

[0009] Step S1, collect cotton plant image samples, and pre-process the cotton plant images to obtain pre-processed image samples;

[0010] Step S2, label the cotton bud positions of the pre-processed image samples to obtain labeled image samples, and divide the labeled image samples into a training set and a verification set;

[0011] Step S3, train a top-down heatmap network using the training set, process the labeled image samples in the verification set using the trained top-down heatmap network to obtain the positions of the two-dimensional key points of the cotton top buds, and verify the trained top-down heatmap network;

[0012] Step S4, based on the positions of the two-dimensional key points of the cotton top buds, a ROS-based three-dimensional coordinate extraction method is applied to extract the three-dimensional coordinates of the cotton top buds.

[0013] According to the method of the first aspect of the application, in the step S1, the cotton plant image samples are collected, and the cotton plant images are pre-processed to obtain pre-processed image samples, which comprises the following steps:

[0014] In the cotton plant image samples, select the image samples whose clarity of cotton top buds reaches a predefined index and which contain only one cotton plant in a frame.

[0015] According to the method of the first aspect of the application, in the step S2, the pre-processed image samples are labeled for cotton bud positions, which comprises the following steps:

[0016] Dot labeling is performed on the positions of the cotton top buds in the pre-processed image, and an auxiliary algorithm is used for identification, and 1-2 nodes at the lower end of the top bud are selected for labeling.

[0017] According to the method of the first aspect of the application, in the step S4, based on the positions of the two-dimensional key points of the cotton top buds, a ROS-based three-dimensional coordinate extraction method is applied to extract the three-dimensional coordinates of the cotton top buds, which comprises the following steps:

[0018] Under the condition that the depth of the depth camera is known:

[0019]

[0020] wherein, [X C Y C Z C ] represents the three-dimensional coordinates of the cotton top bud; represents the camera intrinsic parameters; [xy] T represents the position of the two-dimensional key point of the cotton top bud.

[0021] The second aspect of the present application discloses a cotton top bud three-dimensional coordinate extraction system based on key point detection, which comprises:

[0022] A first processing module configured to collect cotton plant image samples and pre-process the cotton plant images to obtain pre-processed image samples;

[0023] A second processing module configured to label the positions of cotton buds in the pre-processed image samples to obtain labeled image samples, and divide the labeled image samples into a training set and a validation set;

[0024] A third processing module configured to train a top-down heatmap network using the training set, process the labeled image samples in the validation set using the trained top-down heatmap network to obtain the positions of the two-dimensional key points of the cotton top buds, and verify the trained top-down heatmap network;

[0025] A fourth processing module configured to extract the three-dimensional coordinates of the cotton top buds using a ROS-based three-dimensional coordinate extraction method according to the positions of the two-dimensional key points of the cotton top buds.

[0026] According to the system of the second aspect of the present application, the first processing module is specifically configured to collect cotton plant image samples and pre-process the cotton plant images to obtain pre-processed image samples, which comprises:

[0027] In the cotton plant image samples, select image samples whose clarity of cotton top buds reaches a predefined index and which contain only one cotton plant in a frame.

[0028] According to the system of the second aspect of the present application, the second processing module is specifically configured to label the positions of cotton buds in the pre-processed image samples, which comprises:

[0029] Dot labeling is performed on the positions of cotton top buds in the pre-processed images, and an auxiliary algorithm is used for recognition, and 1-2 nodes at the lower end of the top bud are selected for labeling.

[0030] According to the system of the second aspect of the present application, the fourth processing module is specifically configured to apply a ROS-based three-dimensional coordinate extraction method to extract the three-dimensional coordinates of the cotton top bud according to the positions of the two-dimensional key points of the cotton top bud, including:

[0031] In the condition that the depth is known through the depth camera:

[0032]

[0033] wherein, [X C Y C Z C ] represents the three-dimensional coordinates of the cotton top bud; represents the camera intrinsic parameter; [xy] T represents the positions of the two-dimensional key points of the cotton top bud.

[0034] The third aspect of the present application discloses an electronic device. The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of any one of the key point detection-based cotton top bud three-dimensional coordinate extraction method of the first aspect of the present application when executing the computer program.

[0035] The fourth aspect of the present application discloses a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of any one of the key point detection-based cotton top bud three-dimensional coordinate extraction method of the first aspect of the present application.

[0036] In summary, the scheme of the present application designs a complete cotton top bud three-dimensional coordinate recognition technology from sample-training-deployment-recognition. The key point recognition technology is used to recognize the cotton top bud from the side and output the three-dimensional coordinates, which significantly improves the low recognition accuracy of the top recognition and the incoherent and unfavorable mechanical operation problem of the target detection model recognition only outputting two-dimensional coordinates. Thus, the accuracy and efficiency of cotton top bud recognition are effectively improved, and accurate positioning is provided for subsequent mechanical topping. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the drawings needed in the specific embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0038] Figure 1 The flowchart of the key point detection-based cotton top bud three-dimensional coordinate extraction method according to the embodiment of the present application;

[0039] Figure 2 A sample labeling tool interface according to an embodiment of the present application;

[0040] Figure 3 A camera coordinate system and image coordinate system relationship diagram according to an embodiment of the present application;

[0041] Figure 4 A structure diagram of a cotton top bud three-dimensional coordinate extraction system based on key point detection according to an embodiment of the present application;

[0042] Figure 5 A structure diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0043] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0044] The first aspect of the present application discloses a cotton top bud three-dimensional coordinate extraction method based on key point detection. Figure 1 A flowchart of a cotton top bud three-dimensional coordinate extraction method based on key point detection according to an embodiment of the present application is shown in Figure 1 The method comprises the following steps:

[0045] Step S1, collecting a cotton plant image sample and pre-processing the cotton plant image to obtain a pre-processed image sample;

[0046] Step S2, labeling the position of cotton bud on the pre-processed image sample to obtain a labeled image sample, and dividing the labeled image sample into a training set and a verification set;

[0047] Step S3, training a top-down heatmap network by using the training set, processing the labeled image sample in the verification set by using the trained top-down heatmap network to obtain the position of the two-dimensional key point of the cotton top bud, and verifying the trained top-down heatmap network;

[0048] Top-down heatmap is a method based on human pose estimation, which predicts human pose by generating a human key point heatmap.

[0049] Step S4, according to the position of the two-dimensional key points of the cotton terminal bud, a ROS-based three-dimensional coordinate extraction method is applied to extract the three-dimensional coordinates of the cotton terminal bud.

[0050] ROS (Robot Operating System) provides a series of libraries and tools to help software developers create robot application software.

[0051] In step S1, cotton plant image samples are collected, and the cotton plant images are preprocessed to obtain preprocessed image samples.

[0052] In some embodiments, in the step S1, the cotton plant image samples are collected, and the cotton plant images are preprocessed to obtain preprocessed image samples, which include:

[0053] In the cotton plant image samples, the image samples with the clarity of the cotton terminal bud reaching the predefined index and containing only one cotton plant in a frame are selected.

[0054] Specifically, the obtained sample data can be videos / images taken by different devices (including mobile phones, cameras, cameras, computers, etc.), which need to be preprocessed such as video frame conversion and image adjustment to a uniform size. The specific process includes:

[0055] (1) In order to obtain a large amount of sample data in a short time, reduce the working time of sample collection personnel, and improve the working efficiency, the sample collection can be carried out by shooting video during the sample collection process. In the preprocessing, the video frame rate is judged by the program, and the video is converted according to the corresponding frame rate.

[0056] (2) If the sample data taken by different devices may have inconsistent resolution sizes, in order to facilitate subsequent network training, the sample size is adjusted in the preprocessing process. Of course, it can also be adjusted during the training process.

[0057] (3) The images converted by video frame conversion may have the problems of no target image and high image repetition rate caused by high frame rate. In the preprocessing process, the images also need to be selected to some extent, and part of the no-target images and multiple frame repeated images are selectively retained.

[0058] (4) In the cotton plant image samples, the image samples with the clarity of the cotton terminal bud reaching the predefined index and containing only one cotton plant in a frame are selected.

[0059] In step S2, the preprocessed image samples are labeled for cotton bud position to obtain labeled image samples, and the labeled image samples are divided into training set and validation set.

[0060] In some embodiments, in the step S2, the cotton bud position labeling on the pre-processed image sample comprises:

[0061] The position of the cotton top bud in the pre-processed image is marked by dotting, and an algorithm is used to assist in identifying and selecting 1-2 nodes at the lower end of the top bud for labeling.

[0062] Specifically, (1) for the identification of the cotton top bud, a key point detection technology based on deep learning is used. Therefore, the labeling content should be the node position of the cotton top bud and some key nodes used to assist in positioning the cotton top bud.

[0063] (2) The labeling work is carried out on a professional image labeling software or image labeling website, such as Figure 2 As shown, a labeling platform is used to label the sample images collected in the field. The position of the cotton top bud in the image is marked by dotting, and an algorithm is used to assist in identifying and selecting 1-2 nodes at the lower end of the top bud for labeling. The purpose of such labeling is twofold. First, the node position is convenient for the algorithm to locate the top bud. Second, in actual engineering practice, if the top bud cannot be identified, the identified nodes can be used as an alternative solution for topping work to prevent the phenomenon of missing the cotton plant topping. Each sample image is labeled separately, and a single image is exported as a single label json file.

[0064] After all the sample images are labeled, they are randomly divided into a training set and a validation set according to a certain training and validation set ratio (usually 8:2 or 9:1, etc.). The training set is used for network training, and the validation set is used for model accuracy evaluation.

[0065] In step S3, the training set is applied to train the top-down heatmap network, and the trained top-down heatmap network is applied to process the labeled image samples in the validation set to obtain the position of the two-dimensional key points of the cotton top bud, and the trained top-down heatmap network is verified.

[0066] Specifically, the network model is a top-down heatmap structure, which divides the key point detection task into two steps: first, detecting the target position, then generating a heat map of key points in each detected target area, and finally decoding the heat map to obtain the position of each key point.

[0067] (1) In the feature extraction stage, HRNet is used to extract the deep features of the input image, and then a target detector (HRNet) is used to detect the position of each target cotton plant in the image. The detector will give the bounding box of each detected cotton plant.

[0068] (2) Keypoint Heatmap Generation: For each detected object instance, a keypoint heatmap is generated. This heatmap is typically a 2D image, where each pixel represents the confidence of a corresponding keypoint. In terms of heatmap encoding, the MSRAHeatmap encoding method is used, which uses a Gaussian distribution to represent the location of keypoints. Specifically, for each keypoint, MSRAHeatmap generates a Gaussian distribution at its location, representing the probability of the existence of this keypoint, usually represented as a pixel-level heatmap.

[0069] (3) Heatmap Decoding: The generated heatmaps are decoded to obtain the precise location of each keypoint. The goal of decoding is to find the most likely location of each keypoint from the heatmap. This method uses Max-pooling for heatmap decoding. Max-pooling: For each keypoint's heatmap, max-pooling is performed to find the maximum value in the heatmap as the location of the keypoint.

[0070] In network training, various data augmentation strategies are used, including random horizontal flipping, random cropping, etc. The loss function used is KeypointMSELoss as the loss function for keypoint regression. The Adam optimizer is used with an initial learning rate of 5e-4, and the learning rate strategy uses LinearLR and MultiStepLR combination for adaptive adjustment, and the Dropout (random dropout probability) is 0.1 during training. By adjusting the learning rate to reduce the learning rate, the model converges. Setting the random dropout probability can speed up the training and improve the generalization ability of the network to some extent.

[0071] The evaluation indicators of the coco evaluation system are used for verification, including AP (Average Precision) average precision, AR (Average Recall) average recall, PR curve, and mAP (multi-class average precision) as precision evaluation indicators.

[0072] By comparing with the sample label, the correct extraction total number of true positives TP, the total number of false extraction false positives FP, and the total number of missed false negatives FN can be obtained, so that the accuracy and recall can be calculated as:

[0073]

[0074] The AP is the average precision, which is used to measure the detection accuracy of each key point of the model. The AR is the average recall rate, which is used to measure the proportion of real key points that can be detected by the model. The PR curve is the precision-recall curve, which shows the relationship between the precision and the recall rate under different thresholds. The mAP is the multi-class average precision, which is used to measure the average detection accuracy of the model for multiple key points.

[0075] In step S4, according to the position of the two-dimensional key points of the cotton top bud, a ROS-based three-dimensional coordinate extraction method is applied to extract the three-dimensional coordinates of the cotton top bud.

[0076] In some embodiments, in the step S4, the ROS-based three-dimensional coordinate extraction method is applied to extract the three-dimensional coordinates of the cotton top bud according to the position of the two-dimensional key points of the cotton top bud, including:

[0077] Under the condition that the depth is known by the depth camera:

[0078]

[0079] where [X C Y C Z C ] represents the three-dimensional coordinates of the cotton top bud; represents the camera intrinsic parameter; [xy] T represents the position of the two-dimensional key points of the cotton top bud.

[0080] Specifically, the extraction of the three-dimensional coordinates of the top bud in the camera coordinate system: the extracted two-dimensional coordinates are extended to the three-dimensional coordinates in the camera coordinate system.

[0081] First, the imaging principle of the depth camera is as shown in Figure 3 , the object is imaged through a small hole and the final image is recorded by a digital unit. Therefore, the working principle of the depth camera is essentially the mutual conversion between the camera coordinate system, the image coordinate system (and the pixel coordinate system). The specific relationship between them is:

[0082]

[0083] where f represents the focal length of the camera;

[0084] Therefore, according to the above formula, under the condition that the depth is known by the depth camera, the three-dimensional coordinates can be calculated inversely. Given the depth Z C , the image coordinates (pixel coordinates), the camera intrinsic parameter, the coordinates in the camera coordinate system can be obtained:

[0085] Specific embodiments

[0087] (1) Depth camera connects computer, starts depth camera through ROS system, and broadcasts camera signal in the system.

[0088] (2) The key point detection model based on deep learning acquires the camera signal, and performs inference prediction on the image / video to acquire the two-dimensional key point information of the cotton plant on the image / video.

[0089] (3) The two-dimensional key point information of cotton and the depth map of the image / video are jointly used as input to the three-dimensional coordinate extraction module of the application to perform three-dimensional coordinate extraction. And real-time output of labeled image / video information and three-dimensional coordinates of the apical bud of the cotton plant in the camera coordinate system on the image / video are completed to extract the three-dimensional coordinates of the apical bud of the cotton based on key point detection.

[0090] In summary, the scheme of the application designs a complete cotton apical bud three-dimensional coordinate recognition technology from sample-training-deployment-recognition. Through the key point recognition technology, the cotton apical bud is recognized from the side and the three-dimensional coordinates are output, which significantly improves the low recognition accuracy of the top recognition and the problem that the target detection model recognition can only output two-dimensional coordinates, which is not conducive to mechanical operation. Therefore, the accuracy and efficiency of cotton apical bud recognition are effectively improved, and accurate positioning is provided for subsequent mechanical topping.

[0091] The second aspect of the application discloses a cotton apical bud three-dimensional coordinate extraction system based on key point detection. Figure 4 As shown in the structure diagram of the cotton apical bud three-dimensional coordinate extraction system based on key point detection according to an embodiment of the application, the system 100 comprises: Figure 4

[0092] The first processing module 101 is configured to collect cotton plant image samples, and pre-process the cotton plant image to obtain pre-processed image samples;

[0093] The second processing module 102 is configured to mark the position of the cotton bud on the pre-processed image samples to obtain marked image samples, and divide the marked image samples into a training set and a verification set;

[0094] The third processing module 103 is configured to train the top-down heatmap network by using the training set, process the marked image samples in the verification set by using the trained top-down heatmap network to obtain the position of the two-dimensional key point of the cotton apical bud, and verify the trained top-down heatmap network;

[0095] The fourth processing module 104 is configured to extract the three-dimensional coordinates of the cotton apical bud by using the ROS-based three-dimensional coordinate extraction method according to the position of the two-dimensional key point of the cotton apical bud.​

[0096] According to the system of the second aspect of the application, the first processing module 101 is specifically configured to collect cotton plant image samples and pre-process the cotton plant images to obtain pre-processed image samples.

[0097] In the cotton plant image samples, image samples in which the definition of the cotton top bud reaches a pre-defined index and only contain one cotton plant in a frame are selected.

[0098] Specifically, the obtained sample data can be videos / images taken by different devices (including mobile phones, cameras, cameras, computers, etc.), and video frame conversion and image adjustment to a uniform size and other preprocessing operations are required. The specific process includes:

[0099] (1) In order to obtain a large amount of sample data in a short time, reduce the working time of sample collection personnel, and improve the working efficiency, the sample collection can be carried out by shooting video during the sample collection process. In the preprocessing, the video frame rate is judged by the program, and the video is converted according to the corresponding frame rate.

[0100] (2) If the sample data taken by different devices may have inconsistent resolution sizes, in order to facilitate subsequent network training, the sample size is adjusted in the preprocessing process. Of course, the adjustment can also be carried out during the training process.

[0101] (3) The images converted by the video frame conversion may have the problems of no target image and high image repetition rate caused by high frame rate. In the preprocessing process, the images also need to be selected to some extent, and part of the no-target images and multiple frame repeated images are selectively retained.

[0102] (4) In the cotton plant image samples, image samples in which the definition of the cotton top bud reaches a pre-defined index and only contain one cotton plant in a frame are selected.

[0103] According to the system of the second aspect of the application, the second processing module 102 is specifically configured to perform cotton bud position labeling on the pre-processed image samples.

[0104] The position of the cotton top bud in the pre-processed image is marked by dotting, and the auxiliary algorithm is identified, and 1-2 nodes at the lower end of the top bud are selected for labeling.

[0105] Specifically, (1) for the identification of the cotton top bud, the key point detection technology based on deep learning is adopted. Therefore, the labeling content should be the node position of the cotton top bud and part of the key nodes used to assist in positioning the cotton top bud.

[0106] (2) The labeling work is carried out on professional image labeling software or image labeling websites, such asFigure 2 As shown, the annotation platform is used to annotate the sample images collected in the field. The cotton top bud position in the image is marked with dots, and the algorithm can also identify and select 1-2 nodes at the lower end of the top bud for annotation. The purpose of such annotation is as follows: first, the annotated node position facilitates the algorithm to locate the top bud; second, in actual engineering practice, if the top bud cannot be identified, the identified node can be used as an alternative solution for topping work to prevent the phenomenon of missing cotton plant topping. Each sample image is annotated separately, and a single image is exported as a separate label json file.

[0107] After all the sample images are annotated, the samples are randomly divided into a training set and a validation set according to a certain training and validation set ratio (usually 8:2 or 9:1 or the like), wherein the training set is used for network training, and the validation set is used for model precision evaluation.

[0108] According to the system of the second aspect of the application, the third processing module 103 is specifically configured, and the network model is a top-down heatmap structure. This structure decomposes the key point detection task into two steps: first, detecting the target position, then generating a heatmap of key points in each detected target region, and finally decoding the heatmap to obtain the position of each key point.

[0109] (1) In the feature extraction stage, HRNet is used to extract the deep features of the input image, and then a target detector (HRNet) is used to detect the position of each target cotton plant in the image. The detector will give the boundary box of each detected cotton plant.

[0110] (2) Key point heatmap generation: For each detected target, a region is cropped out, and the size is usually adjusted to a preset fixed size for subsequent processing. This can reduce background interference and concentrate resources on processing the details of a single target. For each detected cotton plant region, a key point heatmap is generated. This heatmap is usually a two-dimensional image, and each pixel represents the confidence of the corresponding key point. In heatmap encoding, the MSRAHeatmap heatmap encoding method is used, which uses a Gaussian distribution to represent the position of the key point. Specifically, for each key point, MSRAHeatmap will generate a Gaussian distribution at its position to represent the existence probability of the key point, usually represented as a pixel-level heatmap.

[0111] (3) Heatmap decoding: The generated heatmap is decoded to obtain the precise position of each key point. The goal of decoding is to find the most likely position of each key point from the heatmap. This method uses maximum pooling for heatmap decoding. Maximum pooling (Max-pooling): Maximum pooling is performed on the heatmap of each key point to find the maximum value in the heatmap as the position of the key point.

[0112] In network training, various data enhancement strategies are adopted, including random horizontal flip, random cropping, etc. The loss function used is KeypointMSELoss as the loss function for key point regression. The Adam optimizer is used, the initial learning rate is 5e-4, the learning rate strategy uses LinearLR and MultiStepLR combination for adaptive adjustment, and the Dropout (random dropout probability) is 0.1 in training. The adaptive learning rate adjustment is used to reduce the learning rate to make the model converge. Setting the random dropout probability can speed up the training and improve the generalization ability of the network to a certain extent.

[0113] The evaluation indicators of the coco evaluation system are verified, including AP (Average Precision) average precision, AR (Average Recall) average recall, PR curve, and mAP (multi-class average precision) as precision evaluation indicators.

[0114] By comparing with the sample label, the correct extraction total number true positive TP, the error extraction total number false positive FP and the missed total number false negative FN of the detection result can be obtained, so that the accuracy and the recall rate can be calculated respectively as:

[0115]

[0116] AP is the average precision, which is used to measure the detection accuracy of the model for each key point. AR is the average recall, which is used to measure the proportion of real key points that the model can detect. The PR curve is the precision-recall curve, which shows the relationship between precision and recall at different thresholds. mAP is the multi-class average precision, which is used to measure the average detection accuracy of the model for multiple key points.

[0117] According to the system of the second aspect of the application, the fourth processing module 104 is specifically configured to apply a ROS-based three-dimensional coordinate extraction method to extract the three-dimensional coordinates of the cotton top bud according to the positions of the two-dimensional key points of the cotton top bud, including:

[0118] Under the condition that the depth is known through the depth camera:

[0119]

[0120] where [X C Y C Z C ] represents the three-dimensional coordinates of the cotton top bud; represents the camera intrinsic parameter; [xy] T represents the positions of the two-dimensional key points of the cotton top bud.

[0121] Specifically, the top bud three-dimensional coordinate extraction in the camera coordinate system: the extracted two-dimensional coordinate is extended to the three-dimensional coordinate in the camera coordinate system.

[0122] Firstly, the imaging principle of the depth camera is as shown in the figure Figure 3 The object is imaged through a small hole and finally forms an image by recording with a digital unit. Therefore, the working principle of the depth camera is essentially the mutual conversion between the camera coordinate system, the image coordinate system (and the pixel coordinate system). The specific relationship between them is:

[0123]

[0124] Wherein, f represents the focal length of the camera;

[0125] Therefore, according to the above formula, the three-dimensional coordinate can be calculated inversely under the condition that the depth is known by the depth camera. In the case of known depth Z C , image coordinate (pixel coordinate), camera intrinsic parameter, the coordinate in the camera coordinate system can be obtained:

[0126]

[0127] The third aspect of the present application discloses an electronic device. The electronic device comprises a memory and a processor, and the memory stores a computer program. When the processor executes the computer program, the steps of any one of the key point detection based cotton top bud three-dimensional coordinate extraction methods disclosed in the first aspect of the present application are realized.

[0128] Figure 5 The structure diagram of the electronic device according to the embodiment of the present application is as shown in the figure Figure 5 The electronic device comprises a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control ability. The memory of the electronic device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used for wired or wireless communication with external terminals. Wireless mode can be realized through WIFI, operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen. The input device of the electronic device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0129] Those skilled in the art can understand, Figure 5The structure shown in the figures is only a structural diagram of part of the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the technical solution of the present application is applied. The specific electronic device can include more or fewer components than those shown in the figures, or combine certain components, or have a different arrangement of components.

[0130] The fourth aspect of the present application discloses a computer readable storage medium. The computer readable storage medium stores a computer program. When the computer program is executed by a processor, the steps in the computer program are implemented.

[0131] Please note that the technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application. The above examples only express several embodiments of the present application, and the description is more specific and detailed, but it should not be construed as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for extracting three-dimensional coordinates of cotton terminal buds based on key point detection, characterized in that: The method comprises: Step S1, collecting cotton plant image samples, and preprocessing the cotton plant images to obtain preprocessed image samples; Step S2: marking the cotton bud positions on the preprocessed image samples to obtain marked image samples, and dividing the marked image samples into a training set and a validation set; Step S3, using the training set to train the top-down heatmap network; using the trained top-down heatmap network to process the labeled image samples in the verification set to obtain the positions of the two-dimensional key points of the cotton terminal buds, and verifying the trained top-down heatmap network; Step S4, applying a three-dimensional coordinate extraction method based on ROS according to the positions of the two-dimensional key points of the cotton terminal bud to extract the three-dimensional coordinates of the cotton terminal bud; In step S2, marking the cotton bud positions on the pre-processed image sample includes: Mark the position of the cotton terminal bud in the pre-processed image, and assist the algorithm in identification by selecting 1-2 nodes at the bottom of the terminal bud for marking; In step S4, according to the position of the two-dimensional key point of the cotton terminal bud, applying the ROS-based three-dimensional coordinate extraction method to extract the three-dimensional coordinates of the cotton terminal bud includes: Under the condition that the depth is known through the depth camera: Among them, [X C Y C Z C ] represents the three-dimensional coordinates of the cotton terminal bud; Indicates the camera internal parameters; [xy] T Indicates the positions of the two-dimensional key points of the cotton terminal bud.

2. A cotton terminal bud three-dimensional coordinate extraction method based on key point detection according to claim 1, characterized in that: In step S1, collecting cotton plant image samples and preprocessing the cotton plant images to obtain preprocessed image samples includes: Among the cotton plant image samples, image samples whose clarity of cotton terminal buds reaches a predefined index and whose frames contain only one cotton plant are selected.

3. A cotton terminal bud three-dimensional coordinate extraction system based on key point detection, characterized in that: The system comprises: A first processing module is configured to collect cotton plant image samples and preprocess the cotton plant images to obtain preprocessed image samples; A second processing module is configured to mark the cotton bud positions on the preprocessed image samples to obtain marked image samples, and divide the marked image samples into a training set and a validation set; The second processing module is specifically configured to mark the cotton bud position on the preprocessed image sample, including: Mark the position of the cotton terminal bud in the pre-processed image, and assist the algorithm in identification, selecting 1-2 nodes at the bottom of the terminal bud for marking; A third processing module is configured to train a top-down heatmap network using the training set; process the labeled image samples in the validation set using the trained top-down heatmap network to obtain the positions of the two-dimensional key points of the cotton terminal buds, and verify the trained top-down heatmap network; a fourth processing module configured to extract the three-dimensional coordinates of the cotton terminal bud by applying a three-dimensional coordinate extraction method based on ROS according to the positions of the two-dimensional key points of the cotton terminal bud; The fourth processing module is specifically configured to apply a ROS-based three-dimensional coordinate extraction method according to the position of the two-dimensional key point of the cotton terminal bud, and extract the three-dimensional coordinates of the cotton terminal bud, including: Under the condition that the depth is known through the depth camera: Among them, [X C Y C Z C ] represents the three-dimensional coordinates of the cotton terminal bud; Indicates the camera internal parameters; [xy] T Indicates the positions of the two-dimensional key points of the cotton terminal bud.

4. The cotton terminal bud three-dimensional coordinate extraction system based on key point detection according to claim 3, characterized in that: The first processing module is specifically configured to collect cotton plant image samples and preprocess the cotton plant images to obtain preprocessed image samples, including: Among the cotton plant image samples, image samples whose clarity of cotton terminal buds reaches a predefined index and whose frames contain only one cotton plant are selected.

5. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the method for extracting three-dimensional coordinates of cotton terminal buds based on key point detection according to any one of claims 1 to 2.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the method for extracting three-dimensional coordinates of cotton terminal buds based on key point detection according to any one of claims 1 to 2 are implemented.

Images