Cotton terminal bud three-dimensional coordinate extraction method and system based on key point detection

Abstract

The invention provides a cotton terminal bud three-dimensional coordinate extraction method and system based on key point detection. The method comprises the following steps: collecting a cotton plant image sample, and preprocessing a cotton plant image; performing cotton bud position labeling on the preprocessed image samples, and dividing the labeled image samples into a training set and a verification set; using the training set to train the deep learning network; processing the marked image sample in the verification set by using the trained deep learning network to obtain the position of the two-dimensional key point of the cotton terminal bud; and according to the position of the two-dimensional key point of the cotton terminal bud, extracting a three-dimensional coordinate of the cotton terminal bud by applying an ROS-based three-dimensional coordinate extraction method. According to the scheme provided by the invention, the problems that the recognition precision of top end recognition is low and the incoherence that only two-dimensional coordinates can be output during target detection model recognition is not beneficial to mechanical operation are remarkably solved. Therefore, the precision and efficiency of cotton terminal bud identification are effectively improved, and accurate positioning is provided for subsequent mechanical topping.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning target detection, and in particular relates to a method and system for extracting three-dimensional coordinates of cotton terminal buds based on key point detection. Background Art

[0002] Cotton has the characteristic of unlimited growth, and cotton topping technology can inhibit the growth of terminal buds, thereby preventing excessive development of the cotton top, so that more nutrients can be supplied to the growth of reproductive organs, thereby promoting more bolls and berries in cotton, thereby improving cotton quality and yield. The topping method also has an important impact on the yield and quality of cotton. Choosing the right topping method is also crucial.

[0003] At present, the main topping methods include manual topping, mechanical topping and chemical topping. Topping is usually carried out from late July to early August, and is achieved by removing a leaf and a terminal bud from the top of the cotton plant. Manual topping is a method in which cotton farmers with rich experience in cotton planting manually pinch off the terminal buds. This method has a high accuracy rate, but requires a large amount of labor and high labor costs. Due to its low efficiency, it has gradually been replaced by mechanical and chemical topping. Chemical topping is the use of plant apical inhibitors, such as flumetralin, chlortetracycline and other chemical agents to inhibit the growth of cotton terminal buds. Chemical inhibitors have been proven to have a good topping effect, but chemical topping is not mature in domestic use technology, and improper use poses a risk of reducing cotton production, and large-scale use is prone to environmental pollution. Mechanical topping is the use of agricultural machinery with a cutter to cut the top of cotton. It is much more efficient than manual topping, but the accuracy is not high, and it is easy to cause omissions, wrong topping, collisions, and damage to lateral branches and leaf buds. Therefore, mechanical topping is still only tested in some areas and has not yet been promoted nationwide.

[0004] The key technology in mechanical cotton topping is to accurately identify the cotton terminal buds. In recent years, deep learning target detection methods have been widely used in agriculture. The existing cotton terminal bud intelligent recognition technology uses deep learning target monitoring technology to achieve accurate recognition of cotton terminal buds. In the early preparation stage, the collected cotton terminal bud images are trained using the target detection model to form a unique image database. Subsequently, the terminal bud image captured by the binocular camera is compared with the image in the database, and the terminal bud image is accurately identified through the deep learning target detection model.

[0005] Disadvantages of the prior art

[0006] Existing deep learning object detection models mainly rely on high-quality datasets made from images taken directly above the cotton buds. In addition, deep learning object detection can only identify cotton buds on two-dimensional images, but cannot be converted into three-dimensional coordinates for mechanical topping. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes a technical solution of a method for extracting three-dimensional coordinates of cotton terminal buds based on key point detection to solve the above technical problems.

[0008] The first aspect of the present invention discloses a method for extracting three-dimensional coordinates of cotton terminal buds based on key point detection, the method comprising:

[0009] Step S1, collecting cotton plant image samples, and preprocessing the cotton plant images to obtain preprocessed image samples;

[0010] Step S2, marking the cotton bud positions on the preprocessed image samples to obtain marked image samples, and dividing the marked image samples into a training set and a verification set;

[0011] Step S3, using the training set to train the top-down heatmap network; using the trained top-down heatmap network to process the labeled image samples in the verification set to obtain the positions of the two-dimensional key points of the cotton terminal buds, and verifying the trained top-down heatmap network;

[0012] Step S4: according to the positions of the two-dimensional key points of the cotton terminal bud, a three-dimensional coordinate extraction method based on ROS is applied to extract the three-dimensional coordinates of the cotton terminal bud.

[0013] According to the method of the first aspect of the present invention, in step S1, collecting cotton plant image samples and preprocessing the cotton plant images to obtain preprocessed image samples includes:

[0014] Among the cotton plant image samples, image samples whose clarity of cotton terminal buds reaches a predefined index and whose image samples contain only one cotton plant in one frame are selected.

[0015] According to the method of the first aspect of the present invention, in step S2, marking the cotton bud position of the preprocessed image sample includes:

[0016] The position of the cotton apical bud in the preprocessed image is marked with dots, and the auxiliary algorithm is used to identify and select 1-2 nodes at the lower end of the apical bud for marking.

[0017] According to the method of the first aspect of the present invention, in step S4, applying a three-dimensional coordinate extraction method based on ROS according to the position of the two-dimensional key point of the cotton terminal bud to extract the three-dimensional coordinates of the cotton terminal bud comprises:

[0018] Under the condition that the depth is known through the depth camera:

[0019]

[0020] Among them, [X C Y C Z C ] represents the three-dimensional coordinates of cotton terminal bud; Indicates the camera internal parameters; [xy] T Represents the positions of the two-dimensional key points of the cotton terminal bud.

[0021] The second aspect of the present invention discloses a cotton terminal bud three-dimensional coordinate extraction system based on key point detection, the system comprising:

[0022] A first processing module is configured to collect cotton plant image samples and preprocess the cotton plant images to obtain preprocessed image samples;

[0023] The second processing module is configured to mark the cotton bud positions on the preprocessed image samples to obtain marked image samples, and divide the marked image samples into a training set and a validation set;

[0024] The third processing module is configured to use the training set to train the top-down heatmap network; use the trained top-down heatmap network to process the labeled image samples in the verification set to obtain the positions of the two-dimensional key points of the cotton terminal buds, and verify the trained top-down heatmap network;

[0025] The fourth processing module is configured to extract the three-dimensional coordinates of the cotton terminal bud by applying a three-dimensional coordinate extraction method based on ROS according to the positions of the two-dimensional key points of the cotton terminal bud.

[0026] According to the system of the second aspect of the present invention, the first processing module is specifically configured to collect cotton plant image samples and preprocess the cotton plant images to obtain preprocessed image samples including:

[0027] Among the cotton plant image samples, image samples whose clarity of cotton terminal buds reaches a predefined index and whose image samples contain only one cotton plant in one frame are selected.

[0028] According to the system of the second aspect of the present invention, the second processing module is specifically configured as follows: marking the cotton bud position of the preprocessed image sample includes:

[0029] The position of the cotton apical bud in the preprocessed image is marked with dots, and the auxiliary algorithm is used to identify and select 1-2 nodes at the lower end of the apical bud for marking.

[0030] According to the system of the second aspect of the present invention, the fourth processing module is specifically configured as follows: according to the position of the two-dimensional key point of the cotton terminal bud, applying the ROS-based three-dimensional coordinate extraction method to extract the three-dimensional coordinates of the cotton terminal bud comprises:

[0031] Under the condition that the depth is known through the depth camera:

[0032]

[0033] Among them, [X C Y C Z C ] represents the three-dimensional coordinates of cotton terminal bud; Indicates the camera internal parameters; [xy] T Represents the positions of the two-dimensional key points of the cotton terminal bud.

[0034] The third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any one of the methods for extracting three-dimensional coordinates of cotton terminal buds based on key point detection in the first aspect of the present disclosure are implemented.

[0035] The fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods for extracting three-dimensional coordinates of cotton terminal buds based on key point detection in the first aspect of the present disclosure.

[0036] In summary, the scheme proposed in the present invention designs a complete three-dimensional coordinate recognition technology for cotton terminal buds from sample-training-deployment-recognition. The key point recognition technology is used to identify the cotton terminal buds from the side and output the three-dimensional coordinates, which significantly improves the low recognition accuracy from the top and the incoherent output of the target detection model recognition of only two-dimensional coordinates, which is not conducive to mechanical operation. Thereby effectively improving the accuracy and efficiency of cotton terminal bud recognition, and providing precise positioning for subsequent mechanical topping. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 It is a flow chart of a method for extracting three-dimensional coordinates of cotton terminal buds based on key point detection according to an embodiment of the present invention;

[0039] Figure 2 A labeling interface of a sample labeling tool according to an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of the relationship between a camera coordinate system and an image coordinate system according to an embodiment of the present invention;

[0041] Figure 4 A structural diagram of a cotton terminal bud three-dimensional coordinate extraction system based on key point detection according to an embodiment of the present invention;

[0042] Figure 5 The figure is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0044] The first aspect of the present invention discloses a method for extracting three-dimensional coordinates of cotton terminal buds based on key point detection. Figure 1 FIG. 4 is a flow chart of a method for extracting three-dimensional coordinates of cotton terminal buds based on key point detection according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0045] Step S1, collecting cotton plant image samples, and preprocessing the cotton plant images to obtain preprocessed image samples;

[0046] Step S2, marking the cotton bud positions on the preprocessed image samples to obtain marked image samples, and dividing the marked image samples into a training set and a verification set;

[0047] Step S3, using the training set to train the top-down heatmap network; using the trained top-down heatmap network to process the labeled image samples in the verification set to obtain the positions of the two-dimensional key points of the cotton terminal buds, and verifying the trained top-down heatmap network;

[0048] Top-down heatmap is a method based on human posture estimation, which predicts human posture by generating a heatmap of human key points.

[0049] Step S4: according to the positions of the two-dimensional key points of the cotton terminal bud, a three-dimensional coordinate extraction method based on ROS is applied to extract the three-dimensional coordinates of the cotton terminal bud.

[0050] ROS (Robot Operating System) provides a series of program libraries and tools to help software developers create robot application software.

[0051] In step S1, cotton plant image samples are collected and preprocessed to obtain preprocessed image samples.

[0052] In some embodiments, in step S1, collecting cotton plant image samples and preprocessing the cotton plant images to obtain preprocessed image samples includes:

[0053] Among the cotton plant image samples, image samples whose clarity of cotton terminal buds reaches a predefined index and whose image samples contain only one cotton plant in one frame are selected.

[0054] Specifically, the sample data obtained can be videos / images taken by different devices (including mobile phones, cameras, webcams, computers, etc.), and preprocessing operations such as video frame conversion and image adjustment to a uniform size are required. The specific process includes:

[0055] (1) In order to obtain a large amount of sample data in a short time, reduce the working time of sample collection staff, and improve work efficiency, the sample collection process can be carried out by shooting videos. In the preprocessing, the video frame rate is determined by the program and the video is converted according to the corresponding frame rate.

[0056] (2) If the sample data taken with different devices may have inconsistent resolutions, in order to facilitate subsequent network training, the sample size is adjusted during preprocessing. Of course, it can also be adjusted during the training process.

[0057] (3) The images converted from video frames may have problems such as no target images and high image repetition rate due to high frame rate. In the preprocessing process, it is also necessary to select the images to remove some no target images and multi-frame repeated images for selective retention.

[0058] (4) Among the cotton plant image samples, select image samples whose clarity of cotton apical buds reaches a predefined index and whose image samples contain only one cotton plant in one frame.

[0059] In step S2, the cotton bud positions are annotated on the preprocessed image samples to obtain annotated image samples, and the annotated image samples are divided into a training set and a validation set.

[0060] In some embodiments, in step S2, marking the cotton bud position of the preprocessed image sample includes:

[0061] The position of the cotton apical bud in the preprocessed image is marked with dots, and the auxiliary algorithm is used to identify and select 1-2 nodes at the lower end of the apical bud for marking.

[0062] Specifically, (1) for the identification of cotton terminal buds, a key point detection technology based on deep learning is used. Therefore, the annotation content should be the node position of the cotton terminal bud and some key nodes used to assist in locating the cotton terminal bud.

[0063] (2) The annotation work is performed using professional image annotation software or image annotation websites, such as Figure 2 As shown in the figure, the labeling platform is used to label the sample images collected in the field. The position of the cotton terminal bud in the image is marked with dots, and the algorithm is assisted in identification. 1-2 nodes at the lower end of the terminal bud can be selected for labeling. The purpose of such labeling is, first, the marked node position facilitates the algorithm to locate the terminal bud; second, in actual engineering practice, if the terminal bud cannot be identified, the identified node can be used as an alternative solution to topping work to prevent the phenomenon of cotton plants missing topping. Each sample image is individually labeled, and a single image is exported as a label json file.

[0064] After all sample images are labeled, they are randomly divided into training sets and validation sets according to a certain training-validation set ratio (usually 8:2 or 9:1). The training set is used for network training, and the validation set is used for model accuracy evaluation.

[0065] In step S3, the training set is used to train the top-down heatmap network; the trained top-down heatmap network is used to process the labeled image samples in the verification set to obtain the positions of the two-dimensional key points of the cotton terminal buds, and the trained top-down heatmap network is verified.

[0066] Specifically, the network model is a top-down heatmap structure, which decomposes the key point detection task into two steps: first, detecting the target position, then generating a heat map of the key points in each detected target area, and finally obtaining the position of each key point by decoding the heat map.

[0067] (1) In the feature extraction stage, HRNet is used to perform deep feature extraction of the input image. Then, the object detector (HRNet) is used to detect the position of each target cotton plant in the image. The detector will give the bounding box of each detected cotton plant.

[0068] (2) Key point heat map generation: Each detected target example is cropped and usually resized to a preset fixed size for subsequent processing. This can reduce background interference and concentrate resources on processing the details of a single target. A heat map of key points is generated for each detected cotton plant area. This heat map is usually a two-dimensional image, and each pixel represents the confidence of the corresponding key point. In heat map encoding, the MSRAHeatmap heat map encoding method is adopted, which uses a Gaussian distribution to represent the position of key points. Specifically, for each key point, MSRAHeatmap will generate a Gaussian distribution at its position to represent the probability of the existence of this key point, usually represented by a pixel-level heat map.

[0069] (3) Heat map decoding: decode the generated heat map to obtain the precise location of each key point. The goal of decoding is to find the most likely location of each key point from the heat map. This method uses maximum pooling for heat map decoding. Max-pooling: perform maximum pooling on the heat map of each key point and find the maximum value in the heat map as the location of the key point.

[0070] In network training, a variety of data enhancement strategies are used, including random horizontal flipping, random cropping and other operations. The loss function used is KeypointMSELoss as the loss function for key point regression. The Adam optimizer is used, the initial learning rate is 5e-4, the learning rate strategy uses a combination of LinearLR and MultiStepLR for adaptive adjustment, and the Dropout (random dropout probability) is 0.1 during training. The learning rate is reduced by adaptive learning rate adjustment to make the model converge. Setting the random dropout probability can speed up training and improve the generalization ability of the network to a certain extent.

[0071] The verification uses the evaluation indicators of the coco evaluation system, including AP (Average Precision) average precision, AR (Average Recall) average recall rate, PR curve, and mAP (multi-category average precision) as accuracy evaluation indicators.

[0072] By comparing with the sample labels, we can get the total number of correct extractions (true positives TP), the total number of incorrect extractions (false positives FP), and the total number of missed points (false negatives FN) of the test results, so that the precision and recall rates can be calculated as follows:

[0073]

[0074] AP is the average precision, which is used to measure the detection accuracy of the model for each key point. AR is the average recall rate, which is used to measure the proportion of real key points that the model can detect. The PR curve is the precision-recall curve, which shows the relationship between precision and recall under different thresholds. mAP is the multi-category average precision, which is used to measure the average detection accuracy of the model for multiple key points.

[0075] In step S4, according to the positions of the two-dimensional key points of the cotton terminal bud, a three-dimensional coordinate extraction method based on ROS is applied to extract the three-dimensional coordinates of the cotton terminal bud.

[0076] In some embodiments, in step S4, applying a three-dimensional coordinate extraction method based on ROS according to the position of the two-dimensional key point of the cotton terminal bud to extract the three-dimensional coordinates of the cotton terminal bud comprises:

[0077] Under the condition that the depth is known through the depth camera:

[0078]

[0079] Among them, [X C Y C Z C ] represents the three-dimensional coordinates of cotton terminal bud; Indicates the camera internal parameters; [xy] T Represents the positions of the two-dimensional key points of the cotton terminal bud.

[0080] Specifically, the three-dimensional coordinates of the terminal bud in the camera coordinate system are extracted: the extracted two-dimensional coordinates are expanded to the three-dimensional coordinates in the camera coordinate system.

[0081] First, the imaging principle of the depth camera is as follows Figure 3 As shown, the object is imaged through the pinhole and recorded by the digital unit to form an image. Therefore, the working principle of the depth camera is essentially the mutual conversion between the camera coordinate system and the image coordinate system (and the pixel coordinate system). The specific relationship between them is:

[0082]

[0083] Where f represents the focal length of the camera;

[0084] Therefore, according to the above formula, the three-dimensional coordinates can be inversely calculated under the condition that the depth is known through the depth camera. C , image coordinates (pixel coordinates), camera internal parameters, you can find the coordinates in the camera coordinate system:

[0085] Specific embodiments

[0087] (1) The depth camera is connected to the computer, the depth camera is started through the ROS system, and the camera signal is broadcast within the system.

[0088] (2) A key point detection model based on deep learning is used to obtain camera signals and perform inference and prediction on the image / video to obtain the two-dimensional key point information of the cotton plant in the image / video.

[0089] (3) The two-dimensional key point information of cotton and the depth map of the image / video are jointly used and input into the three-dimensional coordinate extraction module of the present invention to extract the three-dimensional coordinates. The annotated image / video information and the three-dimensional coordinates of the top bud of the cotton plant in the image / video in the camera coordinate system are output in real time to complete the three-dimensional coordinate extraction of the top bud of cotton based on key point detection.

[0090] In summary, the scheme proposed in the present invention designs a complete three-dimensional coordinate recognition technology for cotton terminal buds from sample-training-deployment-recognition. The key point recognition technology is used to identify the cotton terminal buds from the side and output the three-dimensional coordinates, which significantly improves the low recognition accuracy from the top and the incoherent output of the target detection model recognition of only two-dimensional coordinates, which is not conducive to mechanical operation. Thereby effectively improving the accuracy and efficiency of cotton terminal bud recognition, and providing precise positioning for subsequent mechanical topping.

[0091] The second aspect of the present invention discloses a cotton terminal bud three-dimensional coordinate extraction system based on key point detection. Figure 4 FIG. 4 is a structural diagram of a cotton terminal bud three-dimensional coordinate extraction system based on key point detection according to an embodiment of the present invention; Figure 4 As shown, the system 100 includes:

[0092] The first processing module 101 is configured to collect cotton plant image samples and preprocess the cotton plant images to obtain preprocessed image samples;

[0093] The second processing module 102 is configured to annotate the cotton bud positions on the preprocessed image samples to obtain annotated image samples, and divide the annotated image samples into a training set and a validation set;

[0094] The third processing module 103 is configured to use the training set to train the top-down heatmap network; use the trained top-down heatmap network to process the labeled image samples in the verification set to obtain the positions of the two-dimensional key points of the cotton terminal buds, and verify the trained top-down heatmap network;

[0095] The fourth processing module 104 is configured to extract the three-dimensional coordinates of the cotton terminal bud by applying a three-dimensional coordinate extraction method based on ROS according to the positions of the two-dimensional key points of the cotton terminal bud.

[0096] According to the system of the second aspect of the present invention, the first processing module 101 is specifically configured to collect cotton plant image samples and preprocess the cotton plant images to obtain preprocessed image samples including:

[0097] Among the cotton plant image samples, image samples whose clarity of cotton terminal buds reaches a predefined index and whose image samples contain only one cotton plant in one frame are selected.

[0098] Specifically, the sample data obtained can be videos / images taken by different devices (including mobile phones, cameras, webcams, computers, etc.), and preprocessing operations such as video frame conversion and image adjustment to a uniform size are required. The specific process includes:

[0099] (1) In order to obtain a large amount of sample data in a short time, reduce the working time of sample collection staff, and improve work efficiency, the sample collection process can be carried out by shooting videos. In the preprocessing, the video frame rate is determined by the program and the video is converted according to the corresponding frame rate.

[0100] (2) If the sample data taken with different devices may have inconsistent resolutions, in order to facilitate subsequent network training, the sample size is adjusted during preprocessing. Of course, it can also be adjusted during the training process.

[0101] (3) The images converted from video frames may have problems such as no target images and high image repetition rate due to high frame rate. In the preprocessing process, it is also necessary to select the images to remove some no target images and multi-frame repeated images for selective retention.

[0102] (4) Among the cotton plant image samples, select image samples whose clarity of cotton apical buds reaches a predefined index and whose image samples contain only one cotton plant in one frame.

[0103] According to the system of the second aspect of the present invention, the second processing module 102 is specifically configured as follows: marking the cotton bud position of the preprocessed image sample includes:

[0104] The position of the cotton apical bud in the preprocessed image is marked with dots, and the auxiliary algorithm is used to identify and select 1-2 nodes at the lower end of the apical bud for marking.

[0105] Specifically, (1) for the identification of cotton terminal buds, a key point detection technology based on deep learning is used. Therefore, the annotation content should be the node position of the cotton terminal bud and some key nodes used to assist in locating the cotton terminal bud.

[0106] (2) The annotation work is performed using professional image annotation software or image annotation websites, such as Figure 2 As shown in the figure, the labeling platform is used to label the sample images collected in the field. The position of the cotton terminal bud in the image is marked with dots, and the algorithm is assisted in identification. 1-2 nodes at the lower end of the terminal bud can be selected for labeling. The purpose of such labeling is, first, the marked node position facilitates the algorithm to locate the terminal bud; second, in actual engineering practice, if the terminal bud cannot be identified, the identified node can be used as an alternative solution to topping work to prevent the phenomenon of cotton plants missing topping. Each sample image is individually labeled, and a single image is exported as a label json file.

[0107] After all sample images are labeled, they are randomly divided into training sets and validation sets according to a certain training-validation set ratio (usually 8:2 or 9:1). The training set is used for network training, and the validation set is used for model accuracy evaluation.

[0108] According to the system of the second aspect of the present invention, the third processing module 103 is specifically configured as a network model of a top-down heatmap structure, which decomposes the key point detection task into two steps: first, detecting the target position, then generating a heat map of the key points in each detected target area, and finally obtaining the position of each key point by decoding the heat map.

[0109] (1) In the feature extraction stage, HRNet is used to perform deep feature extraction of the input image. Then, the object detector (HRNet) is used to detect the position of each target cotton plant in the image. The detector will give the bounding box of each detected cotton plant.

[0110] (2) Key point heat map generation: Each detected target example is cropped and usually resized to a preset fixed size for subsequent processing. This can reduce background interference and concentrate resources on processing the details of a single target. A heat map of key points is generated for each detected cotton plant area. This heat map is usually a two-dimensional image, and each pixel represents the confidence of the corresponding key point. In heat map encoding, the MSRAHeatmap heat map encoding method is adopted, which uses a Gaussian distribution to represent the position of key points. Specifically, for each key point, MSRAHeatmap will generate a Gaussian distribution at its position to represent the probability of the existence of this key point, usually represented by a pixel-level heat map.

[0111] (3) Heat map decoding: decode the generated heat map to obtain the precise location of each key point. The goal of decoding is to find the most likely location of each key point from the heat map. This method uses maximum pooling for heat map decoding. Max-pooling: perform maximum pooling on the heat map of each key point and find the maximum value in the heat map as the location of the key point.

[0112] In network training, a variety of data enhancement strategies are used, including random horizontal flipping, random cropping and other operations. The loss function used is KeypointMSELoss as the loss function for key point regression. The Adam optimizer is used, the initial learning rate is 5e-4, the learning rate strategy uses a combination of LinearLR and MultiStepLR for adaptive adjustment, and the Dropout (random dropout probability) is 0.1 during training. The learning rate is reduced by adaptive learning rate adjustment to make the model converge. Setting the random dropout probability can speed up training and improve the generalization ability of the network to a certain extent.

[0113] The verification uses the evaluation indicators of the coco evaluation system, including AP (Average Precision) average precision, AR (Average Recall) average recall rate, PR curve, and mAP (multi-category average precision) as accuracy evaluation indicators.

[0114] By comparing with the sample labels, we can get the total number of correct extractions (true positives TP), the total number of incorrect extractions (false positives FP), and the total number of missed points (false negatives FN) of the test results, so that the precision and recall rates can be calculated as follows:

[0115]

[0116] AP is the average precision, which is used to measure the detection accuracy of the model for each key point. AR is the average recall rate, which is used to measure the proportion of real key points that the model can detect. The PR curve is the precision-recall curve, which shows the relationship between precision and recall under different thresholds. mAP is the multi-category average precision, which is used to measure the average detection accuracy of the model for multiple key points.

[0117] According to the system of the second aspect of the present invention, the fourth processing module 104 is specifically configured to extract the three-dimensional coordinates of the cotton terminal bud by applying the ROS-based three-dimensional coordinate extraction method according to the position of the two-dimensional key point of the cotton terminal bud, including:

[0118] Under the condition that the depth is known through the depth camera:

[0119]

[0120] Among them, [X C Y C Z C ] represents the three-dimensional coordinates of cotton terminal bud; Indicates the camera internal parameters; [xy] T Represents the positions of the two-dimensional key points of the cotton terminal bud.

[0121] Specifically, the three-dimensional coordinates of the terminal bud in the camera coordinate system are extracted: the extracted two-dimensional coordinates are expanded to the three-dimensional coordinates in the camera coordinate system.

[0122] First, the imaging principle of the depth camera is as follows Figure 3 As shown, the object is imaged through the pinhole and recorded by the digital unit to form an image. Therefore, the working principle of the depth camera is essentially the mutual conversion between the camera coordinate system and the image coordinate system (and the pixel coordinate system). The specific relationship between them is:

[0123]

[0124] Where f represents the focal length of the camera;

[0125] Therefore, according to the above formula, the three-dimensional coordinates can be inversely calculated under the condition that the depth is known through the depth camera. C , image coordinates (pixel coordinates), camera internal parameters, you can find the coordinates in the camera coordinate system:

[0126]

[0127] The third aspect of the present invention discloses an electronic device. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any one of the methods for extracting three-dimensional coordinates of cotton terminal buds based on key point detection disclosed in the first aspect of the present invention are implemented.

[0128] Figure 5 is a structural diagram of an electronic device according to an embodiment of the present invention, such as Figure 5 As shown, the electronic device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, near field communication (NFC) or other technologies. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covered on the display screen, or a button, a trackball or a touch pad set on the housing of the electronic device, or an external keyboard, touch pad or mouse, etc.

[0129] Those skilled in the art will understand that Figure 5The structure shown in the figure is only a structural diagram of the part related to the technical solution of the present disclosure, and does not constitute a limitation on the electronic device to which the technical solution of the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0130] The fourth aspect of the present invention discloses a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the method for extracting three-dimensional coordinates of cotton terminal buds based on key point detection in any one of the first aspects of the present invention are implemented.

[0131] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.

Claims

1. A method for extracting three-dimensional coordinates of cotton terminal buds based on key point detection, characterized in that: The method comprises: Step S1, collecting cotton plant image samples, and preprocessing the cotton plant images to obtain preprocessed image samples; Step S2, marking the cotton bud positions on the preprocessed image samples to obtain marked image samples, and dividing the marked image samples into a training set and a verification set; Step S3, using the training set to train the top-down heatmap network; using the trained top-down heatmap network to process the labeled image samples in the verification set to obtain the positions of the two-dimensional key points of the cotton terminal buds, and verifying the trained top-down heatmap network; Step S4: according to the positions of the two-dimensional key points of the cotton terminal bud, a three-dimensional coordinate extraction method based on ROS is applied to extract the three-dimensional coordinates of the cotton terminal bud.

2. The method for extracting three-dimensional coordinates of cotton terminal buds based on key point detection according to claim 1, characterized in that: In the step S1, the cotton plant image samples are collected and preprocessed to obtain the preprocessed image samples, including: Among the cotton plant image samples, image samples whose clarity of cotton terminal buds reaches a predefined index and whose image samples contain only one cotton plant in one frame are selected.

3. The method for extracting three-dimensional coordinates of cotton terminal buds based on key point detection according to claim 1, characterized in that: In the step S2, marking the cotton bud positions on the preprocessed image samples includes: The position of the cotton apical bud in the preprocessed image is marked with dots, and the auxiliary algorithm is used to identify and select 1-2 nodes at the lower end of the apical bud for marking.

4. The method for extracting three-dimensional coordinates of cotton terminal buds based on key point detection according to claim 1, characterized in that: In step S4, applying a three-dimensional coordinate extraction method based on ROS according to the position of the two-dimensional key point of the cotton terminal bud to extract the three-dimensional coordinates of the cotton terminal bud includes: Under the condition that the depth is known through the depth camera: Among them, [X C Y C Z C ] represents the three-dimensional coordinates of cotton terminal bud; Indicates the camera internal parameters; [xy] T Represents the positions of the two-dimensional key points of the cotton terminal bud.

5. A cotton terminal bud three-dimensional coordinate extraction system based on key point detection, characterized in that: The system comprises: A first processing module is configured to collect cotton plant image samples and preprocess the cotton plant images to obtain preprocessed image samples; The second processing module is configured to mark the cotton bud positions on the preprocessed image samples to obtain marked image samples, and divide the marked image samples into a training set and a validation set; The third processing module is configured to use the training set to train the top-down heatmap network; use the trained top-down heatmap network to process the labeled image samples in the verification set to obtain the positions of the two-dimensional key points of the cotton terminal buds, and verify the trained top-down heatmap network; The fourth processing module is configured to extract the three-dimensional coordinates of the cotton terminal bud by applying a three-dimensional coordinate extraction method based on ROS according to the positions of the two-dimensional key points of the cotton terminal bud.

6. The cotton terminal bud three-dimensional coordinate extraction system based on key point detection according to claim 5, characterized in that: The first processing module is specifically configured to collect cotton plant image samples and preprocess the cotton plant images to obtain preprocessed image samples including: Among the cotton plant image samples, image samples whose clarity of cotton terminal buds reaches a predefined index and whose image samples contain only one cotton plant in one frame are selected.

7. The cotton terminal bud three-dimensional coordinate extraction system based on key point detection according to claim 5, characterized in that: The second processing module is specifically configured as follows: marking the cotton bud position on the preprocessed image sample includes: The position of the cotton apical bud in the preprocessed image is marked with dots, and the auxiliary algorithm is used to identify and select 1-2 nodes at the lower end of the apical bud for marking.

8. The cotton terminal bud three-dimensional coordinate extraction system based on key point detection according to claim 5, characterized in that: The fourth processing module is specifically configured as follows: according to the position of the two-dimensional key point of the cotton terminal bud, applying the ROS-based three-dimensional coordinate extraction method to extract the three-dimensional coordinates of the cotton terminal bud includes: Under the condition that the depth is known through the depth camera: Among them, [X C Y C Z C ] represents the three-dimensional coordinates of cotton terminal bud; Indicates the camera internal parameters; [xy] T Represents the positions of the two-dimensional key points of the cotton terminal bud.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the steps in the method for extracting three-dimensional coordinates of cotton terminal buds based on key point detection described in any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the method for extracting three-dimensional coordinates of cotton terminal buds based on key point detection according to any one of claims 1 to 4 are implemented.

Images