Unmanned aerial vehicle target detection method and device based on scale adaptive identification, and medium

Abstract

The invention discloses an unmanned aerial vehicle target detection method and device based on scale adaptive recognition and a medium, and relates to the field of machine learning, and the method comprises the steps: carrying out the data collection based on an unmanned aerial vehicle, and obtaining video data corresponding to a target scene; analyzing based on the video data to obtain image data of a continuous sequence, and labeling the image data to generate a training data set; performing data enhancement on the image data in the training data set; and training the convolutional neural network through the training data set after data enhancement, and performing target detection through the trained convolutional neural network. The image is used as input, and then the feature extraction fusion part is used for feature extraction. A multi-scale convolution strategy is adopted, convolution operations of various different sizes are specifically covered, and feature fusion is carried out on feature maps of different scales to obtain a feature combination.

Description

Technical Field

[0001] The present application relates to the field of machine learning, and specifically to a method, device and medium for detecting unmanned aerial vehicle targets based on scale adaptive recognition. Background Art

[0002] UAV target detection technology has been widely used and developed in recent years. It has become an indispensable and important tool in many fields such as traffic monitoring, power inspection, agricultural monitoring, and disaster relief. For example, in traffic monitoring, drones can efficiently monitor traffic flow, identify violations, and improve road safety management. In power inspection, drones can quickly detect problems such as broken wires and equipment failures to ensure the stable operation of the power system. In agricultural monitoring, drones can obtain farmland information in real time, help farmers accurately fertilize and irrigate, and improve agricultural production efficiency. In disaster relief, drones can quickly enter the disaster area, provide on-site information, and provide strong support for rescuers. These applications demonstrate the huge potential and value of drone target detection technology in various fields.

[0003] UAV target detection technology has corresponding advantages. First, UAVs can scan the target area over a large area in a short time, quickly obtain target information, and significantly improve detection efficiency. Second, UAV target detection is highly safe. In complex, dangerous or difficult-to-enter environments, UAVs can replace manual target detection, effectively avoiding casualties and property losses. In addition, UAV target detection also has the advantages of flexibility and data richness. UAVs are not restricted by terrain, landforms and other conditions, can fly freely in the air, flexibly adjust the detection angle and height, and adapt to various complex scenes. At the same time, UAVs can obtain high-resolution image and video data, provide richer information for target detection, and help improve the accuracy and reliability of detection.

[0004] Although drone target detection technology has a wide range of applications and advantages, it also has some disadvantages. First of all, data acquisition is a major challenge faced by drone target detection. In the drone inspection scenario, the complexity and variability of the scene pose great challenges to data collection and annotation. Secondly, the difficulty of target detection is also one of the problems that need to be solved in drone target detection. Since drones usually monitor and shoot from a bird's-eye view, the feature extraction angle is too single. At the same time, the target to be detected is relatively small, with fewer pixels and single features, which can easily lead to missed detection or false detection. Summary of the invention

[0005] In order to solve the above problems, this application proposes a UAV target detection method based on scale adaptive recognition, including:

[0006] Data collection is performed based on drones to obtain video data corresponding to the target scene;

[0007] Parsing the video data to obtain a continuous sequence of image data, annotating the image data, and generating a training data set;

[0008] Performing data enhancement on the image data in the training data set;

[0009] The convolutional neural network is trained through the data-enhanced training data set, and the trained convolutional neural network is used for target detection;

[0010] Among them, the convolutional neural network includes a feature extraction and fusion part and a target detection part; the feature extraction and fusion part includes multiple convolution kernels of different scales, and the target detection part includes multiple parallel detection heads, each detection head independently receives different feature combinations, and the feature combination is obtained by fusion of the outputs of multiple convolution kernels of different scales.

[0011] On the other hand, the embodiment of the present application also provides a drone target detection device based on scale adaptive recognition, including:

[0012] at least one processor; and,

[0013] a memory communicatively connected to the at least one processor; wherein,

[0014] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the drone target detection method based on scale adaptive recognition as described in the above example.

[0015] On the other hand, an embodiment of the present application further provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to be: the drone target detection method based on scale adaptive recognition as described in the above example.

[0016] The UAV target detection method based on scale adaptive recognition proposed in this application can bring the following beneficial effects:

[0017] In the feature extraction and fusion part, the image is used as input, and then the feature extraction and fusion part is used to extract features. A multi-scale convolution strategy is adopted, which specifically covers convolution operations of various sizes, and feature maps of different scales are fused to obtain feature combinations.

[0018] In the object detection part, multiple detection heads working in parallel are used. The detection heads are closely connected with the feature extraction part and are responsible for converting the extracted features into specific prediction information. Each detection head module independently receives specific level features from the feature extraction part and performs the object detection task accordingly.

[0019] After the detection is completed, each module will output information including the target category and precise coordinate position, and finally summarize to form a complete target detection result, thereby overcoming the limitation of the uneven scale of UAV target detection, so as to improve the accuracy and robustness of UAV target detection and improve the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 Schematic diagram of the process of the UAV target detection method based on scale adaptive recognition in the embodiment of the present application;

[0022] Figure 2 This is a schematic diagram of the architecture of a convolutional neural network in an embodiment of the present application;

[0023] Figure 3 Schematic diagram of a drone target detection device based on scale adaptive recognition in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0025] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0026] During the flight, the distance between the drone and the detection target often changes greatly, which makes the shooting angle of the detection target also changeable. This complex shooting condition causes the image to contain both near and far targets, resulting in the phenomenon of target scale imbalance. The scale imbalance problem not only increases the difficulty of target detection, but also significantly affects the accuracy and efficiency of detection.

[0027] Specifically, the scale imbalance problem is mainly reflected in the following aspects: For large-sized targets, due to their rich features, the detection algorithm can usually extract features and identify them well. However, for small-sized targets, due to their small size and unclear features, the detection algorithm often finds it difficult to extract effective features for identification, resulting in poor detection results. Secondly, the scale imbalance problem may also cause the detection algorithm to make false detections or missed detections when dealing with multi-scale targets. When there are targets of multiple scales in the image at the same time, the detection algorithm may not be able to accurately determine the true scale of the target, resulting in erroneous detection results.

[0028] like Figure 1 As shown, the embodiment of the present application provides a drone target detection method based on scale adaptive recognition, comprising:

[0029] S101: Collect data based on the drone to obtain video data corresponding to the target scene.

[0030] First, preliminary data collection is required, using drones to fly and shoot target scenes to obtain rich field data sets. The target scenes are set based on requirements.

[0031] Through the flexible flight of the drone, various details of the scene can be captured efficiently and video data can be obtained, laying the foundation for subsequent data processing and analysis.

[0032] S102: parsing the video data to obtain a continuous sequence of image data, and labeling the image data to generate a training data set.

[0033] Next is the data processing stage. The live video captured by the drone is parsed frame by frame, and images are captured at a certain frequency (for example, 30 frames per second) to ensure the continuity and integrity of the image sequence and obtain a continuous sequence of image data.

[0034] Then, use an image annotation tool (e.g., labelimg) to annotate each frame of the captured image data with a rectangular box. During the annotation process, it is necessary to ensure that each detected target is accurately circled with a rectangular box and attached with corresponding label information as training data to generate a training data set.

[0035] Finally, the training data set is divided into stages. The labeled data set is divided into training data set, validation set and test set according to a certain ratio. The training set is used for the initial training of the model, and the validation set is used to adjust the model parameters during the training process to improve the generalization ability of the model. The test set is used to comprehensively evaluate the performance of the model after the model training is completed.

[0036] S103: Perform data enhancement on the image data in the training data set.

[0037] Data augmentation effectively increases the size and diversity of a dataset by performing a series of transformations on the samples of the original dataset. This process is crucial to improving the generalization ability of the model, especially when dealing with complex tasks such as depth estimation. It can significantly reduce the risk of overfitting and make the model more adaptable and robust when facing unseen data.

[0038] First, the image data is color normalized. Color normalization eliminates redundant information in the image data by standardizing the RGB image, for example, subtracting the mean and dividing by the standard deviation, which accelerates the model training process and improves training efficiency.

[0039] The following are some specific data enhancement methods. In actual operation, you can choose one or more of the following methods for data enhancement.

[0040] The first method is to scale the image data in the training dataset using random numbers within a first preset range. For example, by scaling the color image using a random number s in the range of [1,1.5], the visual changes of the target object at different distances can be simulated, so that the model can learn to recognize objects at different scales and estimate their depth.

[0041] The second method is to rotate the image using a random number within a second preset range. For example, the rotation operation rotates the color and depth images with a random degree r within the range of [-5,5] degrees, which helps the model learn the features of the detected target at different angles, thereby enhancing its adaptability to changes in the object's posture.

[0042] The third method is to adjust the image attributes by a random number within a third preset range. The image attributes include at least one of brightness, contrast, and saturation. For example, the color jitter operation adjusts the brightness, contrast, and saturation of the color image. These parameters are multiplied by k in the range of [0.6, 1.4] to simulate image changes under different lighting conditions, making the model more sensitive to lighting changes and improving its depth estimation accuracy in complex lighting environments.

[0043] The fourth method is to determine whether to flip the image data based on the preset probability, and to flip the image data that needs to be flipped through the nearest neighbor sampling interpolation method. For example, the flip operation flips the color and depth images horizontally with a probability of 50%, and uses the nearest neighbor sampling interpolation method to further increase the diversity of the data set, allowing the model to learn the feature representations of the target object in different directions. This operation not only helps to improve the generalization ability of the model, but also reduces the cost and time of data collection to a certain extent.

[0044] Based on actual conditions, by rationally selecting and combining different data augmentation methods, the size and diversity of the data set can be effectively expanded, providing richer and more comprehensive data support for model training and optimization.

[0045] S104: Train the convolutional neural network using the data-enhanced training data set, and perform target detection using the trained convolutional neural network.

[0046] Among them, the convolutional neural network includes a feature extraction and fusion part and a target detection part; the feature extraction and fusion part includes multiple convolution kernels of different scales, and the target detection part includes multiple parallel detection heads, each detection head independently receives different feature combinations, and the feature combination is obtained by fusion of the outputs of multiple convolution kernels of different scales.

[0047] Specifically, Figure 2 As shown, the feature extraction and fusion part may include a pre-trained ResNet backbone network, which is mainly used for feature extraction and fusion. The feature extraction and fusion part includes multiple convolution kernels of different scales, and the convolution kernels are arranged in order from small to large: the first convolution kernel, the second convolution kernel, and the third convolution kernel. The multiple parallel detection heads are arranged in order from small to large: the first detection head, the second detection head, and the third detection head. For example, Figure 2 As shown, the scales of the first convolution kernel, the second convolution kernel, and the third convolution kernel are 1×1, 3×3, and 5×5, respectively. The scales of the first detection head, the second detection head, and the third detection head are 20×20, 40×40, and 80×80, respectively.

[0048] At this time, the first feature combination independently received by the first detection head is obtained by fusion of the outputs of the first convolution kernel (corresponding to 1×1) and the second convolution kernel (corresponding to 3×3). The second feature combination independently received by the second detection head is obtained by fusion of the outputs of the first convolution kernel (corresponding to 1×1) and the third convolution kernel (corresponding to 5×5). The third feature combination independently received by the third detection head is obtained by fusion of the outputs of the second convolution kernel (corresponding to 3×3) and the third convolution kernel (corresponding to 5×5).

[0049] A multi-scale convolution strategy with three different sizes of convolution kernels, 1×1, 3×3, and 5×5, is adopted. This choice not only covers different scale features from fine to coarse, but also can more effectively capture and fuse multi-scale information by processing these scales in parallel in the same network. In addition, the introduction of 1×1 convolution kernels may also be used for feature dimensionality reduction or increase nonlinearity, further enhancing feature representation capabilities.

[0050] Further, Figure 2For example, in the target detection part, a first feature combination is convolved (conv) to obtain a first convolution result, and the first convolution result is upsampled (upsample) to obtain a first upsampling result.

[0051] The second feature combination is fused (concat) with the first upsampling result, and convolved through the C3 module (mainly used to extract contextual information in the feature map, and increase the network's receptive field and feature expression ability through deep separable convolution operations, thereby improving the precision and accuracy of target detection) to obtain a second convolution result, and the second convolution result is upsampled to obtain a second upsampling result.

[0052] The third feature combination is fused with the second upsampling result, and the third target detection result with the largest scale (corresponding to the output of the third detection head, whose feature map scale corresponds to 80×80) is output through the C3 module, and convolution is performed to obtain the third convolution result.

[0053] The third convolution result is fused with the second convolution result, and the second target detection result of medium scale is output through the C3 module (corresponding to the output of the second detection head, and the scale of its feature map corresponds to 40×40), and convolution is performed to obtain the fourth convolution result.

[0054] The fourth convolution result is fused with the first convolution result, and the C3 module outputs the first target detection result with the smallest scale (corresponding to the output of the first detection head, whose feature map scale corresponds to 20×20).

[0055] There are three detection heads and they are set in parallel. Corresponding to the three fused (concat) features, the features that have been concatted with 3x3 and 5x5 convolution kernels are integrated into the third detection head, the features that have been concatted with 1x1 and 5x5 convolution kernels are integrated into the second detection head, and the features that have been concatted with 1x1 and 3x3 convolution kernels are integrated into the first detection head. The detection head is used to obtain prediction information at different scales from the features, and generate the coordinate position of the target category set based on the prediction information at different scales to obtain the target detection result.

[0056] The detection head uses three parallel modules. Each module specifically incorporates feature information of different scales and different convolution kernel combinations, which brings significant advantages, which are specifically reflected in the following aspects:

[0057] 1. By combining convolution kernels of different sizes (for example, 3x3, 5x5, and 1x1), each detection head can capture features of different scales and patterns in the image. The 3x3 convolution kernel can capture local context information, the 5x5 convolution kernel can capture wider regional features, and the 1x1 convolution kernel is often used to change the number of channels of the feature map or perform feature fusion. This diverse combination of features enables the model to understand the image content more comprehensively, thereby improving the accuracy of target detection.

[0058] 2. Objects of different scales occupy different amounts of space in an image, so scale adaptability is crucial for object detection models. By setting up three detection heads in parallel, each detection head can more accurately capture and predict objects of different scales based on its own feature combination. This design enables the model to process small, medium, and large objects at the same time, thereby improving the model's detection performance for objects of different scales.

[0059] 3. Since each detection head incorporates feature information of different scales and different convolution kernel combinations, the model can learn richer feature representations during training. This rich feature representation not only improves the model's detection accuracy for known targets, but also enhances the model's generalization ability for unknown targets. Therefore, this design enables the model to maintain good detection performance when facing new or unseen targets.

[0060] Based on this, the detection head adopts three parallel modules and specifically integrates feature information of different scales and different convolution kernel combinations, which not only increases the diversity of features and improves the scale adaptability of the model, but also enhances the generalization ability of the model.

[0061] In the feature extraction and fusion part, the image is used as input, and then the feature extraction and fusion part is used to extract features. A multi-scale convolution strategy is adopted, which specifically covers convolution operations of various sizes, and feature maps of different scales are fused to obtain feature combinations.

[0062] In the object detection part, multiple detection heads working in parallel are used. The detection heads are closely connected with the feature extraction part and are responsible for converting the extracted features into specific prediction information. Each detection head module independently receives specific level features from the feature extraction part and performs the object detection task accordingly.

[0063] After the detection is completed, each module will output information including the target category and precise coordinate position, and finally summarize to form a complete target detection result, thereby overcoming the limitation of the uneven scale of UAV target detection, so as to improve the accuracy and robustness of UAV target detection and improve the accuracy of target detection.

[0064] In one embodiment, when there are multiple detection heads in the convolutional neural network, each detection head can output a corresponding detection result. At this time, the final output result needs to be determined based on the multiple detection results. Figure 2 For example, when there are three detection heads, three target detection results will be output respectively.

[0065] Specifically, the target detection results outputted by each detection head are determined. The target detection results are usually selected in the form of a rectangular frame.

[0066] Based on the volume of the detection target, the flight altitude of the drone, and the shooting focal length of the drone, the proportion of the detection target in the image is estimated to obtain a first estimation result. Generally speaking, the volume of the detection target is relatively fixed, and as the flight altitude of the drone and the shooting focal length of the drone change, the visual range of the drone will also change, and the proportion of the detection target in the image will also change. Therefore, the proportion can be estimated to obtain a first estimation result.

[0067] At this time, the confidence scores of the target detection results output by the convolutional neural network are compensated according to the first estimation result. Among them, multiple estimation ranges are pre-set, and the confidence scores (referred to as confidence) of the detection heads corresponding to each estimation range are determined according to the estimation range hit by the first estimation result. Figure 2 Taking the structure of the convolutional neural network in as an example, it has three detection heads, and the scales are as follows from small to large: the first detection head, the second detection head, and the third detection head. At this time, three corresponding estimation ranges are set respectively, from small to large: the first estimation range, the second estimation range, and the third estimation range.

[0068] For different detection heads, the larger the scale, the more suitable it is for detecting large targets. Therefore, when detecting large targets, the confidence of the detection head with a large scale is higher and more trustworthy. Conversely, when detecting small targets, the confidence of the detection head with a small scale is higher.

[0069] At this time, the approximate size of the detection target in the image can be determined by the first estimated result, and the confidence of each detection head can be determined by the estimated range of the hit. For example, when the second estimated range is hit, the confidence of the second detection head is the highest. At this time, if it is closer to the first estimated range, the confidence of the first detection head is second, and the confidence of the third detection head is the lowest, so that the original confidence score is compensated according to the estimated range hit by the first estimated result.

[0070] According to the compensated confidence score, the target detection results are sorted, and each target detection result is selected in order from high to low according to the sorting result, so as to delete the target detection results whose overlap with the currently selected target detection result is higher than a preset threshold (also called IoU threshold) from the remaining target detection results.

[0071] The remaining target detection results are taken as the final target detection results.

[0072] Still Figure 2 Take the convolutional neural network in as an example, assuming that the ranking results according to the confidence score are: second target detection result, first target detection result, third target detection result. At this time, the second target detection result is first selected, and the overlap between it and the second target detection result is determined among the remaining first target detection results and third target detection results. The overlap is determined by the intersection and union ratio of the two coordinates.

[0073] If the overlap of the first target detection result is higher than the preset threshold, it is deleted. If the overlap of the third target detection result is lower than the preset threshold, it is retained.

[0074] Since the first target detection result has been deleted, the third target detection result is selected in order and the overlap between it and the remaining target detection results is calculated. Since the overlap between it and the first target detection result has been calculated and there are no other remaining target detection results, the third target detection result is retained.

[0075] At this time, the remaining target detection results are the first target detection result and the third target detection result, which are retained as the final target detection results.

[0076] Furthermore, when determining whether the overlap is higher than a preset threshold, the setting of the preset threshold is also important in target detection, and determines whether the corresponding detection result is retained.

[0077] Therefore, firstly, the default preset threshold corresponding to the overlap degree is determined, which is usually 0.5, and can also be set according to the actual situation. Then, the preset threshold is compensated based on one or more of the following methods to make the final detection result more accurate.

[0078] 1. Based on the first estimation result, the preset threshold is compensated; wherein, the larger the first estimation result is, the higher the preset threshold is. Generally speaking, for larger targets, a higher preset threshold is usually set because large targets are usually not easy to overlap. For small targets, a lower preset threshold may be required to avoid removing too many boxes.

[0079] 2. Based on the type of the detection target and the type of the target scene, the density of the detection target in the target scene is estimated by outputting the large language model to obtain a second estimation result. Based on the second estimation result, a preset threshold is compensated; wherein, the larger the second estimation result, the lower the preset threshold.

[0080] In datasets with sparse targets (e.g., vehicle detection in street scenes), a higher preset threshold (e.g., 0.7 or 0.8) is usually set because there is less overlap between targets. In datasets with dense targets (e.g., crowd detection on pedestrian streets), a lower preset threshold (e.g., 0.3 or 0.4) can be selected to retain more candidate boxes and avoid accidentally deleting important targets.

[0081] 3. Compensate the preset threshold based on the degree of compensation of the confidence score; the higher the degree of compensation, the lower the preset threshold. The confidence distribution of the model prediction will affect the selection of the preset threshold. If the confidence output of the model is more accurate, a higher preset threshold can be set, because high confidence boxes can usually accurately mark the target. On the contrary, if the model's prediction is not very stable or there are many false positives, setting a lower preset threshold can avoid too many boxes being removed.

[0082] For the above embodiment, when the degree of compensation for the confidence score is lower, it is believed that the proportion of the detection target obtained based on the volume of the detection target, the flight altitude of the drone, and the shooting focal length of the drone is closer to the default situation, which means that the situation faced in this case is the most common situation. Therefore, at this time, it can be considered that the results predicted by the model are more trustworthy, and a higher preset threshold can be set. Conversely, the higher the degree of compensation for the confidence score, the lower the preset threshold.

[0083] In one embodiment, during actual shooting and recognition, due to the high speed of the drone, the captured image may be blurred, resulting in incoherent recognition of the detected target and failure to detect content in some frame images, affecting subsequent detection of the target trajectory.

[0084] Therefore, at this time, a continuous sequence of multiple frame images is determined. At this time, if it is determined that the detection target is detected in a frame image with a ratio higher than a preset ratio (for example, 95%) in the multiple frame images, it is considered that the detection target exists. At this time, if it is determined that the flight speed of the drone is higher than the preset speed and / or the acceleration is higher than the preset acceleration when acquiring the multiple frame images, it is considered that the detection target is not recognized due to the image blur caused by its excessive flight speed or excessive acceleration. At this time, a specified frame image in which the detection target is not detected is acquired in the multiple frame images.

[0085] Then, based on the coordinates of the detected target in the adjacent frames of the specified frame image, coordinate completion is performed in the specified frame image. For example, the coordinates of the adjacent frames are averaged to obtain the coordinates in the specified frame image. If there are no coordinates in the adjacent frame image, the nearest frame image with the detected target is found, and the coordinates in the specified frame image are obtained by calculating the weighted average.

[0086] like Figure 3 As shown, the embodiment of the present application also provides a drone target detection device based on scale adaptive recognition, including:

[0087] at least one processor; and,

[0088] a memory communicatively connected to the at least one processor; wherein,

[0089] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the drone target detection method based on scale adaptive recognition as described in any of the above embodiments.

[0090] An embodiment of the present application also provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to be: the drone target detection method based on scale adaptive recognition as described in any of the above embodiments.

[0091] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0092] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0093] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0094] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0095] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0097] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0098] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0099] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0100] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0101] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A UAV target detection method based on scale adaptive recognition, characterized in that: include: Data collection is performed based on drones to obtain video data corresponding to the target scene; Parsing the video data to obtain a continuous sequence of image data, annotating the image data, and generating a training data set; Performing data enhancement on the image data in the training data set; The convolutional neural network is trained through the data-enhanced training data set, and the trained convolutional neural network is used for target detection; Among them, the convolutional neural network includes a feature extraction and fusion part and a target detection part; the feature extraction and fusion part includes multiple convolution kernels of different scales, and the target detection part includes multiple parallel detection heads, each detection head independently receives different feature combinations, and the feature combination is obtained by fusion of the outputs of multiple convolution kernels of different scales.

2. The method according to claim 1, characterized in that: Performing data enhancement on the image data in the training data set specifically includes: Perform color normalization on image data; For the image data in the training data set, scaling is performed by using a random number within a first preset range; and / or Rotating the image using a random number within a second preset range; and / or Adjusting image attributes by using a random number within a third preset range; wherein the image attributes include at least one of brightness, contrast, and saturation; and / or Based on the preset probability, it is determined whether to perform flip processing on the image data, and the image data that needs to be flipped is flipped by using the nearest neighbor sampling interpolation method.

3. The method according to claim 1, characterized in that The feature extraction and fusion part includes a plurality of convolution kernels of different scales, and the convolution kernels are arranged in descending order of scale: a first convolution kernel, a second convolution kernel, and a third convolution kernel; The multiple parallel detection heads are arranged in order from small to large in size: a first detection head, a second detection head, and a third detection head; The first feature combination independently received by the first detection head is obtained by fusion of outputs of the first convolution kernel and the second convolution kernel; The second feature combination independently received by the second detection head is obtained by fusing the outputs of the first convolution kernel and the third convolution kernel; The third feature combination independently received by the third detection head is obtained by fusion of the outputs of the second convolution kernel and the third convolution kernel.

4. The method according to claim 3, characterized in that: Target detection is performed through the trained convolutional neural network, including: Convolving the first feature combination to obtain a first convolution result, and upsampling the first convolution result to obtain a first upsampling result; The second feature combination is merged with the first upsampling result, and convolution is performed through the C3 module to obtain a second convolution result, and the second convolution result is upsampled to obtain a second upsampling result; The third feature combination is combined with the second upsampling result, and the third target detection result with the largest scale is output through the C3 module respectively, and convolution is performed to obtain a third convolution result; The third convolution result is merged with the second convolution result, and the second target detection result with a medium scale is output through the C3 module respectively, and convolution is performed to obtain a fourth convolution result; The fourth convolution result is fused with the first convolution result, and the first target detection result with the smallest scale is output through the C3 module.

5. The method according to claim 3, characterized in that: The scales of the first convolution kernel, the second convolution kernel, and the third convolution kernel are 1×1, 3×3, and 5×5 respectively; The sizes of the first detection head, the second detection head, and the third detection head are 20×20, 40×40, and 80×80, respectively.

6. The method according to claim 1, characterized in that Target detection is performed through the trained convolutional neural network, including: Determine the target detection results outputted by each detection head respectively; Based on the volume of the detection target, the flight altitude of the drone, and the shooting focal length of the drone, the proportion of the detection target in the image is estimated to obtain a first estimation result; According to the first estimation result, the confidence score of each target detection result output by the convolutional neural network is compensated; wherein a plurality of estimation ranges are pre-set, and the confidence score of the detection head corresponding to each estimation range is determined according to the estimation range hit by the first estimation result; According to the compensated confidence score, the target detection results are sorted, and each target detection result is selected in order from high to low according to the sorting result, so as to delete the target detection results whose overlap with the currently selected target detection result is higher than a preset threshold from the remaining target detection results; The remaining target detection results are taken as the final target detection results.

7. The method according to claim 6, characterized in that Before deleting the target detection results whose overlap with the currently selected target detection result is higher than a preset threshold among the remaining target detection results, the method further includes: Determine a default preset threshold value corresponding to the overlap; Based on the first estimation result, compensating the preset threshold; wherein the larger the first estimation result is, the higher the preset threshold is; and / or Based on the type of the detection target and the type of the target scene, the density of the detection target in the target scene is estimated by outputting the large language model to obtain a second estimation result; based on the second estimation result, the preset threshold is compensated; wherein the larger the second estimation result, the lower the preset threshold; and / or The preset threshold is compensated based on the compensation degree of the confidence score; wherein, the higher the compensation degree, the lower the preset threshold.

8. The method according to claim 1, characterized in that After performing target detection using the trained convolutional neural network, the method further includes: Determine a continuous sequence of multiple frame images; Determining, among the multiple frames of images, that the detection target is detected in the frame images exceeding a preset ratio; If it is determined that the flight speed of the drone when acquiring the multiple frames of images is higher than a preset speed and / or the acceleration is higher than a preset acceleration, acquiring a specified frame image in which the detection target is not detected in the multiple frames of images; Based on the coordinates of the detection target in the adjacent frames of the designated frame image, coordinate completion is performed in the designated frame image.

9. A drone target detection device based on scale adaptive recognition, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the drone target detection method based on scale adaptive recognition as described in any one of claims 1 to 8.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are set to: the drone target detection method based on scale adaptive recognition as described in any one of claims 1 to 8.

Images