A tree top detection method based on image processing, an electronic device and a storage medium

Abstract

The application provides a tree top detection method based on image processing, an electronic device and a storage medium, wherein the method comprises: acquiring a to-be-reconstructed image set, the to-be-reconstructed image set comprising tree top images taken by a drone after preprocessing; performing super-resolution reconstruction on the to-be-reconstructed image set to obtain a reconstructed image set; screening a test image set from the reconstructed image set; using a Tood-SPCS model to perform tree top detection on the test image set, and performing slice-assisted reasoning at the same time to obtain a tree top detection result; wherein the Tood-SPCS model is a target detection model based on a Swin-Transformer network. The to-be-reconstructed image set is subjected to super-resolution reconstruction, and the definition of the to-be-reconstructed image set is improved before tree top recognition. The small image obtained through slice-assisted reasoning has more context information, which can enhance the ability to obtain local information, thereby improving the accuracy of tree top detection.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a treetop detection method, electronic device, and storage medium based on image processing. Background Technology

[0002] In recent years, the planting scale of fruit trees such as lychee trees has been continuously expanding. During the lychee tree planting process, the new shoots that grow have a significant impact on lychee yield. For example, in spring, new shoots and flower spikes grow at the same time. The more new shoots grow, the fewer flower spikes grow, thus reducing lychee yield. Therefore, the identification of new lychee tree shoots is of great significance for orchard management and improving lychee yield.

[0003] Currently, methods for detecting litchi trees fall into two categories. The first method involves manually inspecting orchards and identifying new shoots. This method is limited by factors such as orchard topography, area, and tree height, resulting in low accuracy. The second method utilizes UAV remote sensing technology. For example, CN114266975A discloses UAV remote sensing images of orchard fruit collection, constructing an image database, and dividing the database into training, validation, and test sets. A litchi canopy segmentation model is used to segment the training set images, obtaining canopy segmentation results, which are then validated on the validation set. A litchi fruit detection model divides the canopy segmentation results into dense and non-dense fruit areas. Super-resolution operations are performed on the dense fruit areas to obtain super-resolution dense fruit images. Secondary feature extraction and target localization are then performed on these super-resolution dense fruit images to detect the number of fruits in the dense fruit areas. This method does not detect new shoots of litchi trees. Since targets such as new shoots or fruits account for a very small proportion in UAV remote sensing images, the accuracy of the extracted features is low. This method performs super-resolution reconstruction after segmentation and target detection, resulting in low accuracy of the detection results.

[0004] In summary, existing methods for detecting litchi trees are limited by factors such as orchard topography, area, and tree height, or by the small proportion of the target in UAV remote sensing images, resulting in low accuracy of detection results. Summary of the Invention

[0005] To overcome the problems existing in related technologies, one of the objectives of this invention is to provide a treetop detection method based on image processing. This method performs super-resolution reconstruction on an image set captured by a drone before treetop detection, which improves the clarity of the image set. On this basis, an improved detection model and auxiliary slicing technology are used to identify treetops, thereby improving the accuracy of treetop identification.

[0006] A treetop detection method based on image processing includes:

[0007] Obtain a set of images to be reconstructed, which includes pre-processed images of treetops taken by a drone;

[0008] Super-resolution reconstruction is performed on the image set to be reconstructed to obtain the reconstructed image set;

[0009] A test image set is selected from the reconstructed image set, and the Tood-SPCS model is used to detect treetops in the test image set. Simultaneously, slice-assisted inference is performed to obtain the treetop detection results. The Tood-SPCS model is an object detection model based on the Swin-Transformer network.

[0010] In a preferred embodiment of the present invention, after obtaining the reconstructed image set, the method further includes:

[0011] Construct the Tooth-SPC model to be trained;

[0012] The reconstructed image set is divided into a training image set, a validation image set, and a test image set;

[0013] Set training parameters, and use the training parameters and the training image set to train the Tood-SPC model to be trained, so as to obtain the trained Tood-SPC model.

[0014] The trained Tood-SPC model is sliced ​​for fine-tuning to obtain the Tood-SPCS model.

[0015] In a preferred embodiment of the present invention, constructing the Toood-SPC model to be trained includes:

[0016] Replace the backbone network of the Tood model with the aforementioned Swin-Transformer network;

[0017] Replace the neck network of the Tood model with a path aggregation feature pyramid network;

[0018] Replace the GIoU loss function of the Toood model with the CIoU loss function.

[0019] In a preferred embodiment of the present invention, the step of performing super-resolution reconstruction on the image set to be reconstructed to obtain the reconstructed image set includes:

[0020] The image set to be reconstructed is cropped to obtain a cropped image set;

[0021] Images containing tree branches and shoots are selected from the cropped image set, and all images containing tree branches and shoots are combined into a target image set;

[0022] The Real-ESRGAN method is used to enlarge each image in the target image set by a factor of four to obtain the reconstructed image set.

[0023] In a preferred embodiment of the present invention, training the Toood-SPC model to be trained using the training parameters and the training image set to obtain a trained Toood-SPC model includes:

[0024] The training image set is labeled to obtain the labeled image set;

[0025] The labeled image set is augmented to obtain an augmented image set; wherein the data augmentation includes any one or more of translation, rotation, sharpening, flipping, and blurring;

[0026] The enhanced image set is input into the Tood-SPC model to be trained, and the Tood-SPC model is trained according to the training parameters;

[0027] Calculate the CIoU loss function value, and determine whether the performance of the trained Tood-SPC model meets the standard based on the CIoU loss function value and the classification result. If so, stop training and obtain the trained Tood-SPC model.

[0028] In a preferred embodiment of the present invention, the step of determining whether the performance of the trained Tooth-SPC model meets the standard based on the CIoU loss function value and the classification result includes:

[0029] The intersection-union ratio (CIoU) of the predicted bounding box and the ground truth bounding box is obtained based on the CIoU loss function value.

[0030] The validation image set is input into the trained Toood-SPC model to obtain classification results; the accuracy is calculated based on the classification results, and the average precision is calculated based on the accuracy and the total number of samples.

[0031] If the average precision is greater than the average precision threshold and the cross-union ratio is greater than the cross-union ratio threshold, the training epoch is detected. If the training epoch is greater than or equal to the training epoch threshold, then the performance of the Tooth-SPC model in training meets the standard.

[0032] In a preferred embodiment of the present invention, the step of performing slice-assisted fine-tuning on the trained Tood-SPC model to obtain the Tood-SPCS model includes:

[0033] Set slice parameters, and use the slice parameters and the training parameters to perform auxiliary fine-tuning training on the trained Tooth-SPC model to obtain an auxiliary fine-tuning model; wherein, the slice parameters include slice size and slice mode;

[0034] The auxiliary fine-tuning model with the best performance is selected based on the average accuracy and the crossover ratio, and the auxiliary fine-tuning model with the best performance is selected as the Tood-SPCS model.

[0035] In a preferred embodiment of the present invention, the step of using the Tood-SPCS model to detect treetops in the test image set, and performing slice-assisted inference simultaneously with treetop detection to obtain treetop detection results, includes:

[0036] Set slicing parameters, and segment the i-th image in the test image set according to the slicing parameters to obtain multiple segmented regions; wherein the test image set includes N images, N≥i;

[0037] Treetop detection is performed on each segmented region of the i-th image in the test image set using the Tood-SPCS model.

[0038] If i+1 is greater than N, stop segmentation and treetop detection and obtain the treetop detection result; if i+1 is less than or equal to N, segment the (i+1)th image in the test image set according to the slicing parameters to obtain multiple segmentation regions.

[0039] Treetop detection is performed on each segmented region of the (i+1)th image in the test image set using the Tood-SPCS model.

[0040] A second objective of this invention is to provide an electronic device, comprising:

[0041] Processor; and

[0042] The memory stores executable code, which, when executed by the processor, causes the processor to perform the aforementioned image processing-based treetop detection method.

[0043] A third objective of this invention is to provide a non-transitory machine-readable storage medium storing executable code thereon, which, when executed by a processor of an electronic device, causes the processor to perform the aforementioned treetop detection method based on image processing.

[0044] The beneficial effects of this invention are as follows:

[0045] This invention provides a treetop detection method based on image processing, comprising: acquiring a set of images to be reconstructed, including pre-processed treetop images captured by a drone; performing super-resolution reconstruction on the image set to be reconstructed, improving the clarity of the image set before treetop recognition, resulting in a reconstructed image set; selecting a test image set from the reconstructed image set, and using the Toood-SPCS model to perform treetop detection on the test image set, the Toood-SPCS model being a target detection model based on the Swin-Transformer network. The Swin-transformer network uses a hierarchical construction method similar to that in convolutional neural networks, extracting visual features at different levels, enabling the model to flexibly handle images of different sizes, making it more suitable for tasks such as segmentation and detection; and performing slice-assisted inference simultaneously with treetop detection, using slice-assisted inference to segment each test image in the test image set into multiple smaller images, and the Toood-SPCS model performing treetop detection on each smaller image. The smaller images obtained through slice-assisted inference have more contextual information, which can enhance the ability to obtain local information, thereby improving the accuracy of treetop detection. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the treetop detection method based on image processing provided by the present invention;

[0047] Figure 2 This is a schematic diagram of the process for slice-assisted fine-tuning of a trained Tooth-SPC model provided by the present invention;

[0048] Figure 3 This is a schematic diagram of the process for super-resolution reconstruction of the image set to be reconstructed provided by the present invention;

[0049] Figure 4 This is a schematic diagram of the training process for the Tooth-SPC model provided by the present invention.

[0050] Figure 5 This is a schematic diagram of the process of using the Tood-SPCS model to detect treetops on a test image set, provided by the present invention.

[0051] Figure 6 This is a schematic diagram of a scene for detecting treetops provided by the present invention. Detailed Implementation

[0052] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0053] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0054] It should be understood that although the terms "first," "second," "third," etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, features defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0055] Example 1

[0056] like Figure 1 As shown, the treetop detection method based on image processing provided in this embodiment includes the following steps S1-S3:

[0057] S1: Obtain the image set to be reconstructed, which includes pre-processed images of treetops taken by a drone.

[0058] Control the drone to fly at a set altitude and photograph the lychee trees. As an example, use a DJI Phantom 4 RTK drone and control it to fly at an altitude of 6 to 8 meters to take aerial photos of the lychee trees in the orchard.

[0059] Multiple images captured by the drone are preprocessed. This preprocessing includes removing unusable images due to factors such as wind and light intensity, and denoising the remaining images. For example, when the wind is strong, the lychee tree sways, causing motion blur during drone capture, resulting in 10 blurry images out of 100. As the drone flies continuously, the shooting angle changes from one location to another, leading to increased light intensity corresponding to the shooting angle, resulting in 5 images with strong light intensity out of 100. After removing the 10 blurry images and the 5 images with strong light intensity, denoising is performed on the remaining 85 images to obtain the image set to be reconstructed.

[0060] S2: Perform super-resolution reconstruction on the image set to be reconstructed to obtain the reconstructed image set.

[0061] Image resolution is a parameter used to evaluate the richness of detail contained in an image; high-resolution images have a larger pixel density and more texture details. Super-resolution reconstruction restores a low-resolution image to its corresponding high-resolution image. Super-resolution reconstruction transforms the set of images to be reconstructed into a clearer set of reconstructed images. Performing treetop detection on this reconstructed image set can improve the accuracy of the treetop detection results. It is worth noting that in this embodiment, treetop detection refers to detecting new treetops that grow on the lychee trees throughout the four seasons in the image.

[0062] S3: Select a test image set from the reconstructed image set, and use the Tood-SPCS model to perform treetop detection on the test image set. At the same time as treetop detection, slice-assisted inference is performed to obtain the treetop detection result; wherein, the Tood-SPCS model is a target detection model based on the Swin-Transformer network.

[0063] The reconstructed image set is divided into a test image set, a training image set, and a validation image set. Preferably, the reconstructed image set is divided into the test image set, the training image set, and the validation image set according to an image quantity ratio of 8:1:1.

[0064] Slice-assisted inference is used to segment each test image in the test image set into multiple smaller images, and the Tood-SPCS model performs treetop detection on each smaller image. As an example, slice-assisted inference is used to segment a test image into multiple smaller images, and the Tood-SPCS model performs treetop detection on all smaller images in that test image. Slice-assisted inference is then used to segment the next test image into multiple smaller images, and the Tood-SPCS model performs treetop detection on all smaller images in that test image.

[0065] The original Tood model's backbone network was a ResNet network based on a CNN architecture. ResNet, with its extensive convolutional operations, loses feature information about small objects, hindering feature extraction from the new shoots of the lychee tree. The Swin-transformer network, however, uses a hierarchical construction method similar to that in convolutional neural networks, extracting visual features at different levels. This allows the Tood-SPCS model to flexibly handle images of varying sizes, making it more suitable for tasks such as image segmentation and object detection. Furthermore, the Swin-transformer introduces a sliding window mechanism, confining attention computation to a single window rather than the entire image, significantly reducing computational cost.

[0066] This embodiment provides a treetop detection method based on image processing, including acquiring a set of images to be reconstructed, which includes pre-processed treetop images taken by a drone. Super-resolution reconstruction is performed on the image set to be reconstructed, improving its clarity before treetop recognition, resulting in a reconstructed image set. A test image set is selected from the reconstructed image set, and the Tood-SPCS model is used to detect treetops on the test image set. The Tood-SPCS model is an object detection model based on the Swin-Transformer network. The Swin-Transformer network uses a hierarchical construction method similar to that in convolutional neural networks, extracting visual features at different levels, allowing the model to flexibly handle images of different sizes, making it more suitable for tasks such as segmentation and detection. Slice-assisted inference is performed simultaneously with treetop detection, segmenting each test image in the test image set into multiple smaller images. The Tood-SPCS model then performs treetop detection on each of these smaller images. The smaller images obtained through slice-assisted inference have more contextual information, enhancing the ability to acquire local information and thus improving the accuracy of treetop detection.

[0067] Example 2

[0068] like Figure 2 As shown, this embodiment only describes the differences from Embodiment 1; the remaining technical features are the same as those in the above embodiment. After obtaining the reconstructed image set, the following steps are also included:

[0069] S21': Construct the Tooth-SPC model to be trained.

[0070] Replace the backbone network of the Toood model with the Swin-Transformer network, replace the neck network of the Toood model with the path aggregation feature pyramid network, and replace the GIoU loss function of the Toood model with the CIoU loss function.

[0071] The original FPN in the Tood model propagates from top to bottom, requiring traversal of multiple network layers from the bottom to the top, which may lead to the loss of feature information and multi-scale information of the input image during training. PAFPN improves upon FPN by adding a downsampling module and an additional 3×3 convolutional layer. By constructing a bottom-up feature fusion network, it reduces the propagation path of the main feature layers, enhances the multi-scale fusion information of FPN, better preserves shallow feature information, and improves the localization ability of the entire feature layer.

[0072] When the ground truth bounding box completely encloses the predicted bounding box, the IoU and GIoU loss functions have the same value. In this case, the GIoU loss function degenerates into the IoU loss function, making it impossible to distinguish their relative positional relationship. The CIoU loss function's parameters include the overlap area between the ground truth bounding box and the predicted bounding box, the distance between their center points, and the aspect ratio, making the bounding box regression more stable and improving the accuracy of object detection. The formula for calculating CIoU is as follows:

[0073] ;

[0074] ;

[0075] ;

[0076] The three terms of CIoU correspond exactly to IoU, namely, intersection-to-union ratio, center-to-center distance, and aspect ratio. The first aspect ratio related parameter, This is the second aspect ratio related parameter. This indicates the width of the prediction box. Indicates the height of the prediction box. This represents the width of the actual bounding box. This represents the height of the actual bounding box. The role of the Swin-Transformer network has already been explained in step S3, and will not be repeated here.

[0077]

[0078] The table above shows the performance metrics of the Toood-SPC model and the Toood model. It can be seen that replacing the backbone network of the Toood model with a Swin-Transformer network, replacing the neck network of the Toood model with FPN (Feature Pyramid Networks) instead of PPN (Path Aggregation Feature Pyramid Networks), and replacing the GIoU loss function of the Toood model with the CIoU loss function all improve the performance metrics of the Toood model. The Toood-SPC model to be trained, which simultaneously replaces the backbone network, neck network, and GIoU loss function of the Toood model using the above methods, exhibits the best performance metrics.

[0079] S22': Divide the reconstructed image set into a training image set, a validation image set, and a test image set.

[0080] The reconstructed image set is divided into a training image set, a validation image set, and a test image set in an 8:1:1 ratio. For example, the reconstructed image set has a total of 100 images. Images 1 to 80 are assigned to the training image set, images 81 to 90 are assigned to the validation image set, and images 91 to 100 are assigned to the test image set.

[0081] S23': Set training parameters, and use the training parameters and the training image set to train the Tood-SPC model to obtain the trained Tood-SPC model.

[0082] The training parameters include the learning rate, gradient descent method, and training epochs. Based on the training parameters, the Tood-SPC model to be trained is trained on a graphics card such as an RTX 3090. After training, the trained Tood-SPC model, along with its corresponding performance metrics and inference weight file, is obtained.

[0083]

[0084] The table above compares the performance of the trained Tood-SPC model provided in this embodiment with other models, where Tood-SPC(ours) represents the trained Tood-SPC model. It can be seen that the trained Tood-SPC model outperforms other existing models in detecting new shoots of litchi trees.

[0085] As an example, the training image set consists of 1728 images, and the validation image set consists of 216 images. The learning rate is set to 0.02, and the gradient descent method used is SGD (stochastic gradient descent), with 36 training epochs. The Tood-SPC model is trained on an RTX3090 using the above training parameters, resulting in the trained Tood-SPC model.

[0086] S24': Perform slice-assisted fine-tuning on the trained Tood-SPC model to obtain the Tood-SPCS model.

[0087] Slicing the trained Tood-SPC model for fine-tuning can further improve the accuracy of the trained Tood-SPC model in recognizing and locating new shoots of lychee trees in images.

[0088] This embodiment replaces the backbone network of the Tood-SPC model with a Swin-Transformer network, reducing computational cost while extracting visual features at different levels. The neck network of the Tood model is replaced with a path aggregation feature pyramid network, improving the ability to fuse multi-scale information and enabling the feature layer to better locate the treetops of the lychee tree. The GIoU loss function of the Tood model is replaced with a CIoU loss function, making the bounding box regression more stable and improving the accuracy of object detection.

[0089] Example 3

[0090] like Figure 3 As shown, this embodiment only describes the differences from Embodiment 1, and the remaining technical features are the same as those in the above embodiment.

[0091] Super-resolution reconstruction is performed on the image set to be reconstructed to obtain a reconstructed image set, including:

[0092] S21: The image set to be reconstructed is cropped to obtain a cropped image set.

[0093] Each image in the image set to be reconstructed is cropped to 1800×1200 to obtain the cropped image set.

[0094] S22: Select images containing tree branches and shoots from the cropped image set, and combine all the images containing tree branches and shoots into a target image set.

[0095] Images containing tree branches and shoots can be selected manually or through object detection methods; no specific method is required here.

[0096] S23: Use the Real-ESRGAN method to enlarge each image in the target image set by four times to obtain the reconstructed image set.

[0097] The Real-ESRGAN method is used to double the length and width of each image in the target image set to increase the resolution of each image, resulting in the reconstructed image set.

[0098] like Figure 4 As shown, training the Tooth-SPC model to be trained using the training parameters and the training image set to obtain the trained Tooth-SPC model includes:

[0099] S231': Label the training image set to obtain a labeled image set.

[0100] The LabelImg tool was used to annotate the branches and shoots of litchi trees in the training image set. The annotated results were then converted into COCO format annotation files to obtain the annotated image set.

[0101] S232': Perform data augmentation on the labeled image set to obtain an enhanced image set; wherein the data augmentation includes any one or more of translation, rotation, sharpening, flipping, and blurring.

[0102] As an example, in the labeled image set The images are sequentially translated, rotated, and sharpened to the labeled image set. Flip the image in the labeled image set. The images are blurred to increase the number of images in the labeled image set, thus obtaining the labeled image set.

[0103] S233': Input the enhanced image set into the Tood-SPC model to be trained, and train the Tood-SPC model according to the training parameters.

[0104] The augmented image set contains a larger number of images, resulting in a stronger generalization ability for the trained Toood-SPC model compared to training with a labeled image set. The Toood-SPC model is trained using parameters including the learning rate, gradient descent method, and number of training epochs. Training stops when the number of training epochs reaches a threshold, yielding the trained Toood-SPC model. Preferably, the training epoch threshold is set to 36.

[0105] S234': Calculate the CIoU loss function value. Based on the CIoU loss function value and the classification result, determine whether the performance of the trained Tood-SPC model meets the standard. If so, stop training and obtain the trained Tood-SPC model.

[0106] The intersection-union ratio (CIoU) between the predicted bounding box and the ground truth bounding box is obtained based on the CIoU loss function value. The formula for calculating the CIoU loss function value is as follows:

[0107] ;

[0108] Wherein, IoU is the intersection-union ratio between the predicted bounding box and the ground truth bounding box.

[0109] The accuracy is calculated based on the classification results, and the average precision is calculated based on the accuracy and the total number of samples.

[0110] The classification results include TP, TN, FP, and FN. TP indicates that the model predicted a positive sample as a true positive, i.e., the number of samples where the model predicted a positive result and the actual value is positive. TN indicates that the model predicted a negative sample as a true positive, i.e., the number of samples where the model predicted a positive result and the actual value is negative. FP indicates that the model predicted a negative sample as a true positive, i.e., the number of samples where the model predicted a negative result and the actual value is negative. FN indicates that the model predicted a positive sample as a false negative, i.e., the number of samples where the model predicted a negative result and the actual value is positive.

[0111] Precision, recall, and average precision are calculated using the following formulas:

[0112] ;

[0113] ;

[0114] ;

[0115] Where P represents precision, R represents recall, and AP represents average precision. This represents the total number of samples.

[0116] If the average precision is greater than the average precision threshold and the cross-union ratio is greater than the cross-union ratio threshold, the training epoch is detected. If the training epoch is greater than or equal to the training epoch threshold, then the performance of the Tooth-SPC model in training meets the standard.

[0117] When the Cross-Union Ratio (CUI) is greater than the CUI threshold, it indicates that the trained Tood-SPC model is relatively accurate in locating new shoots of the litchi tree; when the Average Precision (ARP) is greater than the ARP threshold, it indicates that the trained Tood-SPC model is relatively accurate in identifying new shoots of the litchi tree. Combining ARP, CUI, and the number of training epochs can determine whether the trained Tood-SPC model meets the standards. If any one or more of these criteria are not met, the Tood-SPC model should be trained again until all three criteria are met.

[0118] The table below shows the results of the Tood-SPC model and the Tood-SPCS model in detecting new shoots of litchi trees in the image:

[0119]

[0120] Wherein, the false negative rate = 1 - recall rate, the original represents the dataset without super-resolution processing, and the super-resolution represents the dataset after super-resolution processing. It can be seen that after using super-resolution processing, both the Tood-SPC model and the Tood-SPCS model achieved higher accuracy and lower false negative rates. Slice-assisted processing includes slice-assisted inference and slice-assisted fine-tuning; after using slice-assisted processing, both the Tood-SPC model and the Tood-SPCS model achieved higher accuracy and lower false negative rates.

[0121] This embodiment augments the labeled image set, expanding the number of images in the set. The resulting augmented image set has stronger generalization ability and can better adapt to various application scenarios. During the training of the Tood-SPC model, the CIoU loss function value is calculated, and the intersection-union ratio (CIoU) between the predicted and ground truth bounding boxes is obtained through the CIoU loss function value. Accuracy is calculated based on the classification results, and the ratio of accuracy to the total number of samples is used as the mean precision. The CIoU reflects the accuracy of the Tood-SPC model in predicting the location of new shoots on a litchi tree, while the mean precision reflects the accuracy of the Tood-SPC model in identifying new shoots. Using mean precision, CIoU, and the number of training epochs allows for a comprehensive evaluation of the performance of the Tood-SPC model during training.

[0122] Example 4

[0123] This embodiment only describes the differences from Embodiment 1; the remaining technical features are the same as those in the above embodiments. The step of performing slice-assisted fine-tuning on the trained Tood-SPC model to obtain the Tood-SPCS model includes:

[0124] S241': Set the slice parameters, and use the slice parameters and the training parameters to perform auxiliary fine-tuning training on the trained Tooth-SPC model to obtain an auxiliary fine-tuning model; wherein, the slice parameters include slice size and slice mode.

[0125] Training parameters include the learning rate and the number of training epochs, while slicing parameters include the slice size and slice pattern. Below is a comparison table of the Tood-SPCS and Tood-SPC models under different evaluation metrics:

[0126]

[0127] As can be seen from the table, when comparing the Toood-SPC model and the Toood-SPCS model using the evaluation metrics AP95, AP50, APs, APm, and APl, the Toood-SPCS model outperforms the Toood-SPC model in all aspects.

[0128] S242': Select the auxiliary fine-tuning model with the best performance based on the average accuracy and the crossover ratio, and use the auxiliary fine-tuning model with the best performance as the Tood-SPCS model.

[0129] The auxiliary fine-tuning model that maximizes both average precision and crossover ratio (CRR) can be used as the Toood-SPCS model. Alternatively, the auxiliary fine-tuning model that maximizes either average precision or CRR can be used as the Toood-SPCS model. After selecting the best-performing auxiliary fine-tuning model, save the configuration file used during its training.

[0130] like Figure 5 As shown, the Treetop Detection method uses the Tooth-SPCS model to perform treetop detection on the test image set, and performs slice-assisted inference simultaneously to obtain the treetop detection results, including:

[0131] S32: Set slicing parameters, and segment the i-th image in the test image set according to the slicing parameters to obtain multiple segmented regions; wherein, the test image set includes N images, N≥i.

[0132] Before step S32, step S31 is included: selecting a test image set from the reconstructed image set. The window size is set, and the window is slid sequentially to segment each image in the test image set into multiple segmentation regions.

[0133] S33: Use the Tood-SPCS model to perform treetop detection on each segmented region of the i-th image in the test image set.

[0134] Figure 6 This is a schematic diagram of a scene for detecting treetops provided by the present invention, as shown below. Figure 6 As shown, the size of each segmented region in the i-th image is small. Performing treetop detection within a smaller region can obtain more contextual information, thereby enhancing the ability to acquire local information.

[0135] S34: Detect whether i+1 is greater than N. If so, stop segmentation and treetop detection and obtain the treetop detection result; if i+1 is less than or equal to N, segment the i+1th image in the test image set according to the slice parameters to obtain multiple segmentation regions.

[0136] When i+1 is greater than N, it means that all images in the test image set have completed treetop detection. Mark the positions of the new branches of the litchi tree detected in all images with rectangles, and mark the confidence level corresponding to the rectangles near the rectangles to obtain the treetop detection results.

[0137] When i+1 is less than or equal to N, it means that some images in the test image set have not yet completed treetop detection. At this time, the i+1th image in the test image set is segmented according to the slicing parameters to obtain multiple segmentation regions.

[0138] S35: Use the Tood-SPCS model to perform treetop detection on each segmented region of the (i+1)th image in the test image set.

[0139] After performing treetop detection on all segmented regions of the (i+1)th image, it is checked whether i+2 is greater than N. If so, segmentation and treetop detection are stopped, and the treetop detection result is obtained. If i+2 is less than or equal to N, the (i+2)th image in the test image set is segmented according to the slicing parameters to obtain multiple segmented regions.

[0140] This embodiment further improves the Toood-SPC model's ability to identify and locate new shoots of lychee trees in images by performing slice-assisted fine-tuning on the trained model. Average precision reflects the fine-tuning model's ability to identify new shoots of lychee trees in images, while the intersection-over-union (IoU) ratio reflects its ability to locate them. Using the fine-tuning model with the highest average precision and / or IoU ratio as the Toood-SPCS model ensures that the Toood-SPCS model has optimal performance in identifying and locating new shoots of lychee trees in images.

[0141] Example 5

[0142] Corresponding to Embodiments 1-4, this application also provides an electronic device, which includes a memory and a processor. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0143] Memory can include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM can store static data or instructions required by the processor or other modules of the computer. Permanent storage devices can be read-write storage devices. Permanent storage devices can be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., hard disks, optical disks, and / or flash memory) as permanent storage devices. In other embodiments, permanent storage devices can be removable storage devices (e.g., floppy disks and / or optical drives). System memory can be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory can store some or all of the instructions and data required by the processor during operation. Furthermore, memory can include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and can also be hard disks and / or optical disks. In some implementations, the memory may include removable storage devices that are readable and / or writable, such as laser discs (CDs), read-only digital multifunction optical discs (e.g., DVD-ROMs, dual-layer DVD-ROMs), read-only Blu-ray discs, ultra-high density optical discs, flash memory cards (e.g., SD cards, mini SD cards, Micro-SD cards, etc.), magnetic floppy disks, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0144] The memory stores executable code, which, when processed by the processor, can cause the processor to execute some or all of the methods described above.

[0145] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings. In the description of this application, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is usually based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this application and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this application; the directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0146] For ease of description, spatial relative terms such as "above," "on top of," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation beyond the orientation of the device as described in the figures. For example, if the device in the figures were inverted, a device described as "above" or "on top of" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0147] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore cannot be construed as limiting the scope of protection of this application.

[0148] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A treetop detection method based on image processing, characterized in that, include: Obtain a set of images to be reconstructed, which includes pre-processed images of treetops taken by a drone; Super-resolution reconstruction is performed on the image set to be reconstructed to obtain the reconstructed image set; A test image set is selected from the reconstructed image set, and the Tood-SPCS model is used to detect treetops in the test image set. Slicing-assisted inference is performed simultaneously with treetop detection to obtain the treetop detection results. The Tood-SPCS model is an object detection model based on the Swin-Transformer network. After obtaining the reconstructed image set, the process also includes: Construct the Tooth-SPC model to be trained; The reconstructed image set is divided into a training image set, a validation image set, and a test image set; Set training parameters, and use the training parameters and the training image set to train the Tood-SPC model to be trained, so as to obtain the trained Tood-SPC model. The trained Tood-SPC model is sliced ​​for fine-tuning to obtain the Tood-SPCS model. The construction of the Toood-SPC model to be trained includes: Replace the backbone network of the Tood model with the described Swin-Transformer network; Replace the neck network of the Tood model with a path aggregation feature pyramid network; Replace the GIoU loss function of the Toood model with the CIoU loss function; The Tood-SPCS model is used to detect treetops in the test image set. Simultaneously, slice-assisted inference is performed to obtain the treetop detection results, including: Set slicing parameters, and segment the i-th image in the test image set according to the slicing parameters to obtain multiple segmented regions; wherein the test image set includes N images, N≥i; Treetop detection is performed on each segmented region of the i-th image in the test image set using the Tood-SPCS model. If i+1 is greater than N, stop segmentation and treetop detection and obtain the treetop detection result; if i+1 is less than or equal to N, segment the (i+1)th image in the test image set according to the slicing parameters to obtain multiple segmentation regions. Treetop detection is performed on each segmented region of the (i+1)th image in the test image set using the Tood-SPCS model.

2. The treetop detection method based on image processing according to claim 1, characterized in that, The process of performing super-resolution reconstruction on the image set to be reconstructed to obtain the reconstructed image set includes: The image set to be reconstructed is cropped to obtain a cropped image set; Images containing tree branches and shoots are selected from the cropped image set, and all images containing tree branches and shoots are combined into a target image set; The Real-ESRGAN method is used to enlarge each image in the target image set by a factor of four to obtain the reconstructed image set.

3. The treetop detection method based on image processing according to claim 1, characterized in that, The step of training the Toood-SPC model to be trained using the training parameters and the training image set to obtain the trained Toood-SPC model includes: The training image set is labeled to obtain the labeled image set; The labeled image set is augmented to obtain an augmented image set; wherein the data augmentation includes any one or more of translation, rotation, sharpening, flipping, and blurring; The enhanced image set is input into the Tood-SPC model to be trained, and the Tood-SPC model is trained according to the training parameters; Calculate the CIoU loss function value, and determine whether the performance of the trained Tood-SPC model meets the standard based on the CIoU loss function value and the classification result. If so, stop training and obtain the trained Tood-SPC model.

4. The treetop detection method based on image processing according to claim 3, characterized in that, The step of determining whether the performance of the trained Tooth-SPC model meets the standard based on the CIoU loss function value and classification results includes: The intersection-union ratio (CIoU) of the predicted bounding box and the ground truth bounding box is obtained based on the CIoU loss function value. The validation image set is input into the trained Toood-SPC model to obtain classification results; the accuracy is calculated based on the classification results, and the average precision is calculated based on the accuracy and the total number of samples. If the average precision is greater than the average precision threshold and the cross-union ratio is greater than the cross-union ratio threshold, the training epoch is detected. If the training epoch is greater than or equal to the training epoch threshold, then the performance of the Tooth-SPC model in training meets the standard.

5. The treetop detection method based on image processing according to claim 4, characterized in that, The step of performing slice-assisted fine-tuning on the trained Tood-SPC model to obtain the Tood-SPCS model includes: Set slice parameters, and use the slice parameters and the training parameters to perform auxiliary fine-tuning training on the trained Tooth-SPC model to obtain an auxiliary fine-tuning model; wherein, the slice parameters include slice size and slice mode; The auxiliary fine-tuning model with the best performance is selected based on the average accuracy and the crossover ratio, and the auxiliary fine-tuning model with the best performance is selected as the Tood-SPCS model.

6. An electronic device, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the image processing-based treetop detection method as described in any one of claims 1-5.

7. A non-transitory machine-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the treetop detection method based on image processing as described in any one of claims 1-5.

Images