UAV visual recognition method and device

Abstract

The present invention relates to a method and device for visual recognition of unmanned aerial vehicles, and the method comprises: collecting multimodal images through a multimodal sensing device of the unmanned aerial vehicle, performing image conversion on the multimodal images, and obtaining enhanced images; performing target detection on the enhanced images according to a preset target detection network, and obtaining preliminary detection targets; performing regional feature analysis on the preliminary detection targets, and obtaining corresponding target region feature information; obtaining environmental factors collected by the multimodal sensing device, and performing environmental tracking analysis on the preliminary detection targets according to the environmental factors, and obtaining target tracking results; performing multi-task analysis on the preliminary detection targets by combining the target region feature information and the target tracking results with a preset recognition model, and obtaining corresponding comprehensive recognition results. The present invention can improve the overall performance and reliability of visual recognition of unmanned aerial vehicles.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle visual recognition, and in particular to a method and device for unmanned aerial vehicle visual recognition. Background Art

[0002] The rapid development of drone technology has led to its widespread application in many fields, among which visual recognition capability is the key to drones performing various tasks. With the advancement of artificial intelligence and computer vision technology, how to improve the accuracy and efficiency of drone visual recognition to adapt to complex and changing environments and mission requirements has become one of the research focuses. Existing drone visual recognition methods usually rely on only a single image data source, such as a visible light camera or an infrared camera, while ignoring the advantages of multimodal image fusion. This single data source method may lead to inaccurate or unstable recognition results in certain complex environments, thus affecting the execution of drone missions. In addition, traditional target detection and tracking algorithms often fail to fully consider the dynamic changes of environmental factors, resulting in large differences in recognition performance in different scenarios. Summary of the invention

[0003] The main purpose of the present invention is to provide a drone visual recognition method, device, equipment and storage medium, which can improve the overall performance and reliability of drone visual recognition.

[0004] To achieve the above object, the present invention provides a UAV visual recognition method, comprising:

[0005] Collecting multimodal images through a multimodal sensing device of the drone, and performing image conversion on the multimodal images to obtain enhanced images;

[0006] Performing target detection on the enhanced image according to a preset target detection network to obtain a preliminary detection target;

[0007] Performing regional feature analysis on the preliminary detected target to obtain corresponding target region feature information;

[0008] Acquire the environmental factors collected by the multimodal sensing device, perform environmental tracking analysis on the preliminary detection target according to the environmental factors, and obtain a target tracking result;

[0009] The preset recognition model is combined with the target area feature information and the target tracking result to perform a multi-task analysis on the preliminary detected target to obtain a corresponding comprehensive recognition result.

[0010] Furthermore, the collecting of multimodal images by a multimodal sensing device of the drone and performing image conversion on the multimodal images to obtain enhanced images include:

[0011] Collecting multimodal images including visible light images, infrared images and range depth images through the multimodal sensing device of the drone;

[0012] Performing multi-scale fusion of the visible light image, the infrared image and the range depth image to obtain a preliminary fused image;

[0013] Performing multi-scale spatial pyramid processing on the preliminary fused image to obtain a multi-layer feature map;

[0014] Performing top-down feature propagation on the multi-layer feature maps to obtain an initial feature pyramid;

[0015] Performing a 1x1 convolution transformation on each layer of the feature map in the initial feature pyramid to obtain an adjusted feature map;

[0016] Upsampling and element-wise addition fusion of the adjusted feature maps of adjacent levels to obtain a fused feature map;

[0017] Performing 3x3 convolution processing on the fused feature map to obtain a smoothed feature map of the corresponding level;

[0018] The smoothed feature maps of all levels are fused using a preset feature enhancement algorithm to obtain the enhanced image.

[0019] Furthermore, performing target detection on the enhanced image according to a preset target detection network to obtain a preliminary detected target includes:

[0020] Inputting the enhanced image into a shallow feature extraction layer of the target detection network for visual extraction to obtain a low-level visual feature map;

[0021] Inputting the low-level visual feature map into the middle-level feature fusion layer of the target detection network for visual extraction to obtain a middle-level semantic feature map;

[0022] Inputting the intermediate semantic feature map into the deep semantic feature layer of the target detection network for visual extraction to obtain a high-level semantic feature map;

[0023] Performing feature fusion on the low-level visual feature map, the mid-level semantic feature map, and the high-level semantic feature map to obtain a multi-level visual feature map;

[0024] Generate an anchor frame according to the multi-level visual feature map to obtain an initial candidate region;

[0025] Performing feature alignment on the initial candidate region and the multi-level visual feature map to obtain candidate region features;

[0026] Inputting the candidate region features into the target classifier and bounding box regressor of the target detection network to obtain target refined bounding box coordinates;

[0027] Performing target screening on the target refined bounding box coordinates according to a preset confidence threshold to obtain a confidence target;

[0028] Eliminate overlapping detection frames of the confidence target to obtain a confidence detection result;

[0029] Mapping detection is performed on the enhanced image according to the confidence detection result to obtain a preliminary detection target.

[0030] Furthermore, the performing of regional feature analysis on the preliminary detected target to obtain corresponding target region feature information includes:

[0031] Performing region segmentation on the initially detected target to obtain a plurality of target sub-regions;

[0032] Extracting features from each of the target sub-regions according to a preset adaptive feature extraction algorithm to obtain a sub-region feature set;

[0033] According to a preset attention mechanism, weights are assigned to the sub-region feature sets to obtain a weighted feature set;

[0034] Performing topological feature conversion on the weighted feature set to obtain a topological feature representation of the target;

[0035] Matching the topological feature representation with a preset spatiotemporal feature template to obtain a spatiotemporal correlation score;

[0036] Screening the topological feature representation according to the spatiotemporal correlation score to obtain a key feature set;

[0037] Applying dynamic quantization coding to the key feature set to obtain a compressed feature vector;

[0038] The compressed feature vector is calibrated and calculated according to a preset self-calibration algorithm to obtain the target area feature information.

[0039] Furthermore, matching the topological feature representation with a preset spatiotemporal feature template to obtain a spatiotemporal correlation score includes:

[0040] Decomposing the topological feature representation in time and space dimensions to obtain a time feature sequence and a space feature matrix;

[0041] Calculating the similarity between the time feature sequence and the time template in the spatiotemporal feature template to obtain a time correlation score;

[0042] Performing a spatial frequency transformation on the spatial feature matrix to obtain a spatial spectrum diagram;

[0043] Performing a matching operation on the spatial spectrum graph and the spatial template in the spatiotemporal feature template according to a preset image matching algorithm to obtain a spatial correlation score;

[0044] Performing weighted fusion on the temporal correlation score and the spatial correlation score to obtain an initial temporal and spatial correlation score;

[0045] The time-space correlation scores are screened according to a preset threshold judgment rule to obtain a valid time-space correlation score;

[0046] Performing statistical analysis on the effective spatiotemporal correlation scores to obtain the spatiotemporal correlation scores;

[0047] Among them, the calculation formula for similarity calculation is:

[0048]

[0049] : the i-th value in the first time series, : the i-th value in the second time series,

[0050] : the mean of the first time series, : the mean of the second time series,

[0051] r: temporal correlation score, the range of r is [-1, 1], where 1 indicates a perfect positive correlation, 0 indicates no correlation, and -1 indicates a perfect negative correlation.

[0052] Furthermore, the acquiring of environmental factors collected by the multimodal sensing device and performing environmental tracking analysis on the preliminary detection target according to the environmental factors to obtain a target tracking result include:

[0053] Acquiring multi-dimensional environmental factor data including light intensity, weather conditions and terrain features collected by the multimodal sensing device;

[0054] Performing matrix conversion on the multi-dimensional environmental factor data to obtain an environmental feature matrix;

[0055] Performing environmental characteristic analysis on the preliminary detection target according to the environmental characteristic matrix to obtain environmental target characteristics;

[0056] Performing filtering position calculation on the environmental target features to obtain target predicted position information;

[0057] Performing target motion vector analysis on the target predicted position information to obtain a target motion trend;

[0058] The target movement trend is analyzed by trajectory analysis according to a preset spline interpolation algorithm to obtain the target tracking result.

[0059] Furthermore, the preset recognition model is combined with the target area feature information and the target tracking result to perform multi-task analysis on the preliminary detected target to obtain a corresponding comprehensive recognition result, including:

[0060] Extracting features from the target area feature information and the target tracking result to obtain a target feature vector;

[0061] Inputting the target feature vector into the first decision tree layer of the recognition model for preliminary classification to obtain a corresponding preliminary classification result;

[0062] Selecting a classification neural network layer corresponding to the recognition model according to the preliminary classification result, inputting the target feature vector into the classification neural network layer for initial recognition processing, and obtaining an initial recognition result;

[0063] Inputting the initial recognition result into the second-layer decision tree of the recognition model for fine-grained classification to obtain a fine-grained classification result;

[0064] Inputting the fine-grained classification result and the preliminary detection target into the task neural network layer of the recognition model, and performing a multi-task analysis including target classification, posture estimation and behavior recognition on the preliminary detection target according to the fine-grained classification result to obtain a multi-task analysis result;

[0065] Inputting the multi-task analysis results into the third-level decision tree of the recognition model for final decision analysis to obtain a decision result;

[0066] The decision result is subjected to time series analysis through the output layer of the recognition model to generate and output the comprehensive recognition result.

[0067] The present invention also provides a drone visual recognition device, which is applied to any of the above-mentioned drone visual recognition methods, comprising:

[0068] An acquisition module, wherein the acquisition module is used to acquire multimodal images through a multimodal sensing device of the drone, and perform image conversion on the multimodal images to obtain enhanced images;

[0069] An analysis module, the analysis module is used to perform target detection on the enhanced image according to a preset target detection network to obtain a preliminary detection target;

[0070] An association module, the association module is used to perform regional feature analysis on the preliminary detected target to obtain corresponding target region feature information;

[0071] A processing module, the processing module is used to obtain the environmental factors collected by the multimodal sensing device, perform environmental tracking analysis on the preliminary detection target according to the environmental factors, and obtain a target tracking result;

[0072] A control module is used to perform multi-task analysis on the preliminary detected target through a preset recognition model combined with the target area feature information and the target tracking result to obtain a corresponding comprehensive recognition result.

[0073] The present invention provides a method, device, equipment and storage medium for visual recognition of unmanned aerial vehicles, which have the following beneficial effects:

[0074] By converting and enhancing the data collected by the multimodal sensing device, the advantages of different sensors can be fully utilized, the accuracy and robustness of target detection can be improved, and more reliable basic data can be provided for subsequent analysis. According to the preset target detection network, the enhanced image is detected and feature recognition is performed to achieve refined recognition of the target, which helps to improve the accuracy and efficiency of recognition and adapt to complex and changing environmental requirements. By obtaining environmental factors and tracking and analyzing the characteristic information of the target area, dynamic tracking of the target can be achieved, the recognition performance of the system under different environmental conditions is improved, and the interference caused by environmental changes is reduced. The preset recognition model is used to perform multi-task analysis on the target tracking results, and the comprehensive recognition of the target is achieved, which can meet the needs of different mission scenarios and improve the flexibility and adaptability of the system. Through this method of multimodal image fusion and environmental factor consideration, the limitations of single data source and static recognition algorithm are effectively solved, the overall performance and reliability of UAV visual recognition are improved, and strong support is provided for UAVs to perform various tasks in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a flow chart of a UAV visual recognition method provided by the present invention;

[0076] Figure 2 This is a structural diagram of a UAV visual recognition device provided by the present invention;

[0077] Figure 3 It is a structural diagram of a UAV visual recognition device provided by the present invention.

[0078] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0080] The present invention is further described below in conjunction with the accompanying drawings and specific implementation methods.

[0081] Reference Figure 1 As shown, the present invention provides a drone visual recognition method, comprising:

[0082] Step S1: collecting multimodal images through the multimodal sensing device of the drone, performing image conversion on the multimodal images, and obtaining enhanced images;

[0083] Step S2: performing target detection on the enhanced image according to a preset target detection network to obtain a preliminary detection target;

[0084] Step S3: Perform regional feature analysis on the preliminary detected target to obtain corresponding target region feature information;

[0085] Step S4: Acquire environmental factors collected by the multimodal sensing device, perform environmental tracking analysis on the preliminary detection target based on the environmental factors, and obtain target tracking results;

[0086] Step S5: Perform multi-task analysis on the preliminary detected target through a preset recognition model combined with target area feature information and target tracking results to obtain a corresponding comprehensive recognition result.

[0087] Based on the above steps, the detailed process is as follows:

[0088] Step S1: The multimodal images of the target area are synchronously collected through the multimodal perception equipment (such as RGB camera, thermal imager, lidar, etc.) on the UAV.

[0089] The multimodal image is converted into an enhanced image through a preset image enhancement algorithm, such as histogram equalization, denoising, fusion, etc. The visibility and effective information of the image are improved. For example, by fusing the thermal image with the visible light image, a richer information image is generated to show the outline of the object and its temperature characteristics.

[0090] Through a series of image processing algorithms, the output enhanced image should have rich details and clear contrast, so that subsequent target detection can be carried out under better conditions. The effectiveness of the enhanced image will directly affect the accuracy and efficiency of target detection in subsequent steps.

[0091] Step S2: Detect the enhanced image based on the preset trained target detection network, take the enhanced image as input, and analyze the enhanced image using the target detection network. The target detection network outputs the bounding box, category label and confidence score of the detected target according to the category label set during training.

[0092] In the output of target detection, high-confidence detection results are screened out by setting a threshold to avoid possible false detections. For example, only detection results with a confidence level higher than 0.5 are selected to ensure that only relatively reliable detection targets are selected.

[0093] The preliminary detection target includes a set of target object information. This target object information, such as the location coordinates and category information of the object, will be passed to the next analysis step. By comparing the detection results of different types of targets, the important elements in the scene can be preliminarily determined.

[0094] Step S3: Use image processing techniques, such as edge detection, corner detection and region segmentation, to perform detailed analysis on each target region. This can be done with the help of feature maps, where the extracted features can include the target's shape, color distribution and relative contours.

[0095] When performing regional feature analysis on the initially detected target, the image data is used to perform regional segmentation to obtain an accurate description of each target area. For example, for animal detection, its type can be further accurately described by analyzing its body shape, body color and movement characteristics.

[0096] The output target area feature information, such as the target's descriptive features (such as color, shape), relative position and motion state, will be stored and passed to the next step of environmental tracking analysis.

[0097] Step S4:

[0098] Acquire environmental factors collected by multimodal sensing devices. Multimodal sensing devices not only collect information about target objects, but also need to monitor environmental factors in real time. These environmental factors include meteorological data (such as temperature, humidity, light intensity), geographic information (such as altitude, terrain), wind speed, background noise, and other external conditions that may affect target detection and tracking. The data of these environmental factors can be obtained through integrated sensors (such as meteorological sensors, GPS, IMU, etc.).

[0099] For the initially detected target, it is assumed that the detected target is an animal. The animal's behavior pattern or movement trend will be analyzed using the environmental data (such as temperature and humidity) that has been acquired. The activity status of the target under specific environmental factors will be determined through historical data.

[0100] The initial location information of the target is associated with environmental factors to make environmental tracking predictions. For example, if the target is observed to be active at a certain temperature, it is inferred that the animal will continue or move to other areas in the temperature environment.

[0101] Through environmental tracking analysis, the dynamic tracking results of the target are output. This includes the target's predicted trajectory, motion state (such as stop, moving direction, speed estimation), and correlation with environmental factors. For example, the tracking information of the detected animal is output, including its possible movement path in complex terrain. This information will provide more accurate and context-related data support for subsequent recognition analysis.

[0102] Step S5: Integrate the information from different steps to construct a comprehensive feature vector, which will contain the target area feature information (such as shape, color, texture), target tracking results (such as motion state and direction), and the impact data of environmental factors.

[0103] When performing multi-task analysis, the recognition model is able to handle the following tasks simultaneously:

[0104] Target category recognition: Determine the type of target, such as vehicle, animal or pedestrian.

[0105] Prediction of target status: Predict the future status of the target, such as whether the target is about to move, wait, or have other behavioral changes.

[0106] Decision making: Automatically select the best path or operation strategy based on target characteristics and environmental factors. For example, in a search and rescue mission, the search path can be adjusted according to the animal's movement trajectory and environmental changes.

[0107] Based on the results of all analyses, the model will generate a comprehensive recognition result, which includes the final category of the target, its dynamic state, and the prediction of its behavior.

[0108] The present invention provides a method for visual recognition of unmanned aerial vehicles. By performing image conversion and enhancement on data collected by a multimodal sensing device, the advantages of different sensors can be fully utilized, the accuracy and robustness of target detection can be improved, and more reliable basic data can be provided for subsequent analysis. According to a preset target detection network, the enhanced image is subjected to target detection and feature recognition, so as to achieve refined recognition of the target, which is helpful to improve the accuracy and efficiency of recognition and adapt to complex and changing environmental requirements. By acquiring environmental factors and tracking and analyzing the characteristic information of the target area, dynamic tracking of the target can be achieved, the recognition performance of the system under different environmental conditions is improved, and the interference caused by environmental changes is reduced. A preset recognition model is used to perform multi-task analysis on the target tracking results, so as to achieve comprehensive recognition of the target, meet the requirements of different task scenarios, and improve the flexibility and adaptability of the system. Through this method of multimodal image fusion and environmental factor consideration, the limitations of a single data source and a static recognition algorithm are effectively solved, the overall performance and reliability of the unmanned aerial vehicle visual recognition device are improved, and strong support is provided for the unmanned aerial vehicle to perform various tasks in a complex environment.

[0109] In one embodiment, a multimodal image is collected by a multimodal sensing device of a drone, and image conversion is performed on the multimodal image to obtain an enhanced image, including:

[0110] The drone's multimodal perception equipment collects multimodal images including visible light images, infrared images, and range depth images. Visible light images provide appearance details of the target, while infrared images present the thermal characteristics of the object. Range depth images are used to obtain three-dimensional information in the scene. This multimodal data collection ensures the integrity of target information in a changing environment.

[0111] The visible light image, infrared image and range depth image are fused at multiple scales to obtain a preliminary fused image. Multiscale fusion is to process various images at different resolutions in order to extract their respective features while retaining important information. In this process, the scales of various images are adjusted to adapt to a unified scale.

[0112] The preliminary fused image is processed with a multi-scale spatial pyramid to obtain a multi-layer feature map. The image is decomposed into features of different scales. These features can extract more detailed content, such as edges, corners, color distribution and other information. Each layer of feature map represents the features of the image at different scales, reflecting different levels of details.

[0113] After the multi-layer feature map is generated, it will be propagated from top to bottom. The purpose of this step is to propagate high-level semantic information to low-level features layer by layer, which helps to enhance the overall characteristics of the target and the richness of detailed information. The initial feature pyramid is thus formed, which contains various features from low to high.

[0114] Perform 1x1 convolution transformation on each layer of feature map, adjust the number of channels of the feature map and perform linear combination of local features.

[0115] After obtaining the adjusted feature map, the adjusted feature maps of adjacent levels will be upsampled and element-wise fused. The purpose of this process is to combine the high-resolution feature map with the low-resolution feature map, thereby effectively preserving the details and global information of target detection.

[0116] After the fusion feature map is completed, the feature map is processed by 3x3 convolution to obtain the smoothed feature map of the corresponding level. 3x3 convolution helps to extract more complex spatial features, reduce noise, and enhance important structures such as edges, making the image more delicate and clear.

[0117] The enhanced image is obtained by fusing the smoothed feature maps of all levels through a preset feature enhancement algorithm. This feature enhancement algorithm is designed to further improve the contrast, visual clarity and emphasis of important features of the image. For example, the use of technologies such as adaptive histogram equalization not only improves the recognizability of potential targets in the enhanced image, but also provides a more superior foundation for subsequent target detection, ensuring that drones can more accurately identify and track targets in complex environments.

[0118] This embodiment significantly improves the accuracy and robustness of target recognition through the effective extraction and enhancement of multiple image features. By performing multi-scale fusion of visible light images, infrared images, and range depth images, it is possible to fully explore target features under complex environmental conditions to ensure that the target is still clearly visible under the influence of different lighting and backgrounds. The combination of this multimodal data enables the system to obtain informative images at different times and in different environments, making the target recognition system more adaptable. By adopting a top-down feature propagation mechanism and a 1x1 convolution adjustment feature map design, effective support is provided for the correlation between features, which helps to retain the key details of the target in multi-level features, making the overall recognition result more accurate. This not only improves the efficiency of recognition, but also provides strong data support for subsequent target tracking.

[0119] In one embodiment, target detection is performed on the enhanced image according to a preset target detection network to obtain a preliminary detected target, including:

[0120] The enhanced image is input into the shallow feature extraction layer of the target detection network for visual extraction to obtain a low-level visual feature map. In this layer, the image undergoes a preliminary convolution operation to extract a low-level visual feature map, including edges, corners and other basic visual elements.

[0121] The low-level visual feature map is passed to the mid-level feature fusion layer of the object detection network. At this stage, the basic information in the low-level visual feature map is fused with the mid-level semantic information from other paths to extract the mid-level semantic feature map, which contains not only basic morphological information, but also understanding of object category, texture and background information.

[0122] The intermediate semantic feature map is input into the deep semantic feature layer of the target detection network. After being processed by a complex deep learning algorithm, a high-level semantic feature map is generated. The feature extraction at this stage can capture more complex scene structures and object semantics, such as the foreground and background information and relative position relationship of the target, providing a higher level of understanding of various targets in the chain.

[0123] The low-level visual feature map, the mid-level semantic feature map and the high-level semantic feature map are fused to obtain a multi-level visual feature map.

[0124] Anchor boxes are generated based on multi-level visual feature maps to obtain initial candidate regions. Anchor boxes are boxes of predetermined shapes and sizes based on feature maps, and their purpose is to capture the location of potential targets. These anchor boxes are adjusted in the image by interacting with actual objects in the feature map, providing a rough positioning for target detection.

[0125] Perform feature alignment on the initial candidate regions to ensure that the features of the candidate regions correspond to the coordinates of the multi-level visual feature map. Feature alignment is the process of ensuring the consistency of features of objects of different scales and categories, which can improve the accuracy of subsequent classification and bounding box regression.

[0126] The candidate region features are input into the target classifier and bounding box regressor of the target detection network to obtain the refined bounding box coordinates of the target. At this stage, the classifier classifies and identifies the candidate regions based on the target detection network, while the regressor is used to accurately adjust the anchor box to produce a bounding box that matches the shape and position of the target.

[0127] The refined bounding box coordinates of the target are filtered according to the preset confidence threshold to obtain the confidence target. The preset confidence threshold is to filter out those targets whose confidence is lower than a certain standard to ensure the reliability and accuracy of the final recognition result.

[0128] The subsequent overlapping detection box elimination process is designed to eliminate possible duplicate detections and ensure that each target is identified only once. This process can effectively reduce false positive results and improve the clarity of detection results.

[0129] Based on the confidence detection results, the enhanced image is mapped and detected to obtain the preliminary detection target, which contains positioning information and target category.

[0130] The target detection network based on deep learning in this embodiment can extract features from enhanced images layer by layer, and feature information from low-level to high-level can be reasonably integrated, which not only improves the accuracy of target recognition, but also enhances the understanding of complex scenes. Through the generation and fusion of multi-level visual feature maps, the system can capture subtle changes and contextual relationships of the target, ensuring effective target recognition in a dynamic environment. By adopting a confidence threshold screening mechanism and overlapping box elimination technology, this combined strategy effectively reduces false detections and duplicate detections, thereby improving the reliability of the detection results. This high-precision recognition capability enables drones to quickly and accurately locate and track targets in complex mission environments, greatly improving operational efficiency and response speed.

[0131] In one embodiment, regional feature analysis is performed on the preliminary detected target to obtain corresponding target region feature information, including:

[0132] The image processing algorithm is used to segment the target for preliminary detection. The segmented image generates multiple target sub-regions. The purpose of region segmentation is to refine the target region to make feature extraction more accurate. The segmentation method selected depends on the properties of the target and the complexity of the environment.

[0133] An adaptive feature extraction algorithm is applied to each target sub-region to obtain a sub-region feature set. The adaptive feature extraction algorithm can dynamically adjust the extraction parameters according to the changes in the brightness, color distribution and texture characteristics of the image, and can capture useful information reflecting the characteristics of the sub-region, such as shape, color and edge features.

[0134] The generated sub-region feature sets will be weighted according to the attention mechanism. The attention mechanism is a technology that simulates human visual habits and aims to reasonably adjust the attention of various types of information in the feature set to highlight key features. By assigning different weights to each feature, a weighted feature set is formed, allowing the system to focus on more recognizable areas.

[0135] The weighted feature set is then transformed through a topological feature conversion operation to generate a topological feature representation of the target. Topological feature conversion is a technique used to describe the form of an object in space. It can represent the geometric and structural characteristics of the object in a topological data format, thereby providing a higher-level information overview and maintaining the consistency and integrity of its spatial relationships.

[0136] The topological feature representation determines the spatiotemporal correlation score by matching with the preset spatiotemporal feature template. The spatiotemporal feature template is a predefined set that can describe the behavior of the target in the spatiotemporal dimension. It is mainly used to judge the similarity between the current target state and the template, so as to obtain the target's activity level in the spatiotemporal dimension.

[0137] According to the spatiotemporal correlation scores, the topological feature representations are filtered to obtain the key feature set. The screening process uses the scores to retain the features that are critical to target recognition and filter out redundant information to optimize the processing performance of subsequent steps. The key feature set contains the most important structural details and dynamic characteristics of the interaction between the target and the environment.

[0138] Dynamic quantization coding technology is applied to the key feature set to generate compressed feature vectors. Dynamic quantization coding is a data compression algorithm that aims to reduce the amount of feature data while maintaining the integrity and validity of the original information, so that resources can be saved and data quality can be guaranteed during storage and transmission.

[0139] The compressed feature vector is calibrated and calculated through the preset self-calibration algorithm to derive the feature information of the target area. The self-calibration algorithm is used to correct feature deviations caused by environmental changes or inaccurate feature extraction. It adaptively corrects errors by referring to known calibration data to ensure high accuracy and consistency of the output results. The entire process undergoes fine feature screening and conversion.

[0140] This embodiment achieves more efficient and accurate target recognition of drones in complex environments by performing regional feature analysis on the preliminary detected targets. By region segmentation, the target is decomposed into multiple sub-regions, which improves the accuracy of feature extraction and enables the system to capture subtle features of the target. The adaptive feature extraction algorithm dynamically adjusts parameters according to the characteristics of different sub-regions to ensure accurate extraction of key features in diverse environments and enhance the adaptability of the system. Through the attention mechanism, the focus on important features is effectively improved, the interference of redundant information is reduced, and the efficiency and accuracy of the recognition process are further optimized. The combination of topological feature conversion and spatiotemporal feature matching enables the system to better capture the dynamic changes of the target, ensuring that the recognition results are not only limited to static features, but also reflect the behavior patterns of the target in the spatiotemporal dimension.

[0141] In one embodiment, the topological feature representation is matched with a preset spatiotemporal feature template to obtain a spatiotemporal correlation score, including:

[0142] The topological feature representation is decomposed in time and space dimensions to generate a time feature sequence and a spatial feature matrix. This step analyzes the topological features and separates their information in time and space dimensions to form a time series representing the target behavior and a spatial feature matrix describing the target distribution.

[0143] The similarity between the time feature sequence and the time template in the spatiotemporal feature template is calculated to obtain the time correlation score. This calculation involves comparing the changes in the target in the time series with the preset time template to evaluate the similarity between the two. The time template is usually composed of historical data or known behavior patterns, which can provide a reference benchmark for subsequent judgments.

[0144] The spatial frequency transformation of the spatial feature matrix is ​​performed through Fourier transform to generate a spatial spectrum map. This change can also use other technical transformations to convert the features in the spatial domain into the frequency domain. The spatial spectrum map provides more intuitive spectral information for subsequent spatial matching, making it more efficient to evaluate the spatial distribution of targets in complex scenes.

[0145] According to the preset image matching algorithm, the spatial spectrum diagram is matched with the spatial template in the spatiotemporal feature template. Matching operations can also be performed through algorithms such as cross-correlation and Euclidean distance to evaluate the fit between the spatial features of the current target and the template.

[0146] The time correlation score and the space correlation score are weighted and fused to generate the initial time-space correlation score. Weighted fusion is based on previously determined rules, and usually assigns different weights to the influence of time and space to reflect the relative importance of time and space factors to target judgment in specific application scenarios.

[0147] According to the preset threshold judgment rule, the initial spatiotemporal correlation score is screened to obtain the effective spatiotemporal correlation score. This judgment rule is based on previous experience or set standards to ensure that only cases that reach a certain correlation score can be considered effective, avoiding the risk of misjudgment.

[0148] The effective spatiotemporal correlation scores after screening are statistically analyzed to derive the final spatiotemporal correlation scores. This step integrates and analyzes the correlation scores through statistical methods such as mean and variance to ensure that the final result reflects the overall performance of the target in time and space. The generated spatiotemporal correlation scores provide an important basis for subsequent target identification and analysis, ensuring that the UAV can fully evaluate the dynamic characteristics and behavior patterns of the target when performing tasks.

[0149] Among them, the calculation formula for similarity calculation is:

[0150]

[0151] : the i-th value in the first time series, : the i-th value in the second time series,

[0152] : the mean of the first time series, : the mean of the second time series,

[0153] r: temporal correlation score, the range of r is [-1, 1], where 1 indicates a perfect positive correlation, 0 indicates no correlation, and -1 indicates a perfect negative correlation.

[0154] This embodiment significantly improves the adaptability and recognition accuracy of the drone to the dynamic behavior of the target by matching the topological feature representation with the spatiotemporal feature template. Through the decomposition of the spatiotemporal dimension, the temporal features and spatial features can be clearly distinguished, so that the system can perform detailed analysis on the two types of features respectively. This analysis method enables the system to adapt to different types of targets and improves the flexibility of understanding target behavior. The weighted fusion technology of the temporal correlation score and the spatial correlation score is used to ensure that the influence of temporal and spatial features on the final judgment can be effectively balanced under different environmental conditions. This process greatly enhances the reliability of target detection, making the recognition algorithm less susceptible to noise interference when facing complex backgrounds and dynamic scenes. The preset threshold judgment rule can accurately identify key information from a large amount of data for the application of screening effective spatiotemporal correlation scores, avoid false alarms and omissions, and ensure the accuracy of the final recognition results.

[0155] In one embodiment, environmental factors collected by a multimodal sensing device are obtained, and environmental tracking analysis is performed on a preliminary detection target based on the environmental factors to obtain a target tracking result, including:

[0156] Multi-dimensional environmental factor data including light intensity, weather conditions and terrain features are collected by the drone’s multimodal sensing equipment.

[0157] The acquired multi-dimensional environmental factor data is transformed into a matrix to form a unified environmental feature matrix. The environmental feature matrix is ​​a structured representation that can integrate data of different types of environmental factors and integrate information such as light, weather, and terrain into an operational analysis basis.

[0158] The environmental feature matrix is ​​used to analyze the environmental features of the preliminary detection target and identify the target's performance under the current environmental conditions. This step combines the preliminary detection target with the environmental features to analyze the possible state changes and behavioral performance of the target under specific environmental conditions and generate environmental target feature information.

[0159] The environmental target features are filtered and calculated to obtain the predicted position information of the target. Filter position calculation is a mathematical method for reducing noise, which can effectively correct the position deviation caused by sensor errors and environmental changes, thereby improving the accuracy of target position estimation.

[0160] After obtaining the predicted position information, the target motion vector analysis is performed to extract the target's motion trend. Motion vector analysis is based on historical position change data, calculating the displacement and direction of the target over time and revealing the target's motion pattern. This process determines the target's motion pattern by analyzing its past trajectory.

[0161] According to the preset spline interpolation algorithm, the target motion trend is analyzed and the target tracking result is obtained. The spline interpolation algorithm is a mathematical method for smoothing data sets. By interpolating key points, the target's motion trajectory presents a more continuous change, thereby achieving the purpose of accurately predicting the target's future position.

[0162] This embodiment significantly improves the target tracking capability of the UAV in a dynamic environment by acquiring environmental factors collected by a multimodal sensing device. First, by integrating multidimensional environmental data such as light intensity, weather conditions, and terrain features, the system can fully understand the impact of the current environment on the target behavior. This integration process improves the accuracy of environmental analysis, so that the target tracking process fully considers the dynamic properties of environmental changes. The construction of the environmental feature matrix makes data processing more systematic and standardized, making subsequent analysis steps more efficient. The use of filtered position calculation technology not only reduces the noise interference caused by the environment, but also improves the prediction accuracy of the target position information, ensuring that the tracking results are more reliable. In addition, through the combination of motion vector analysis and spline interpolation algorithm, the system can track the movement trend of the target in real time and accurately determine its future movement trajectory. This accuracy is particularly critical in dynamic or uncertain environments, ensuring that the UAV adjusts the tracking strategy in time and reduces the possibility of mistracking.

[0163] In one embodiment, a multi-task analysis is performed on the preliminary detected target through a preset recognition model combined with target area feature information and target tracking results to obtain a corresponding comprehensive recognition result, including:

[0164] Feature extraction is performed on the target area feature information and target tracking results to generate a target feature vector. The target feature vector is a high-dimensional data set used to represent the target attributes, containing key information such as the size, shape, color, and motion state of the target, ensuring that subsequent analysis has a rich feature basis.

[0165] The target feature vector is input into the first decision tree layer of the recognition model for preliminary classification. In this stage, the first decision tree layer of the pre-trained recognition model is used to determine the major categories of the target, such as "person", "vehicle" or "animal". The preliminary classification results will provide directional information for subsequent segmentation recognition.

[0166] According to the preliminary classification results, the corresponding classification neural network layer is selected, and the target feature vector is input into the classification neural network layer for initial recognition processing. The classification neural network layer uses deep learning technology to extract more detailed feature information through complex nonlinear mapping of input features and generate initial recognition results.

[0167] The initial recognition results are passed to the second-level decision tree of the recognition model for fine-grained classification. This process analyzes the possible categories of the target in the initial recognition and performs more detailed classification, such as identifying individuals or specific models in a specific scene.

[0168] The fine-grained classification results are input into the task-specific neural network layer of the recognition model together with the preliminary detection targets for multi-task analysis. The multi-task analysis based on the fine-grained classification results in this stage includes target classification, posture estimation and behavior recognition. The generated multi-task analysis results will fully present the state and behavior characteristics of the target, providing comprehensive information support for the drone when performing tasks.

[0169] The multi-task analysis results are further processed and passed to the third-level decision tree of the recognition model for final decision analysis to obtain the decision result. This process brings together the analysis results of all the previous steps to ensure the accuracy and reliability of the final decision. This decision result is a comprehensive judgment of the target's comprehensive performance in the current environment.

[0170] The decision results are input to the output layer of the recognition model for time series analysis, thereby generating and outputting comprehensive recognition results. Time series analysis is based on time information and historical data, and can examine the persistence and change trends of targets in dynamic scenarios to ensure that the results are not just instantaneous performance, but reflect the overall behavior pattern of the target within a certain time range.

[0171] Reference Figure 2 As shown, the present invention also provides a drone visual recognition device, which is applied to any of the above drone visual recognition methods, including:

[0172] The acquisition module is used to acquire multimodal images through the multimodal sensing device of the UAV, and perform image conversion on the multimodal images to obtain enhanced images;

[0173] An analysis module, the analysis module is used to perform target detection on the enhanced image according to a preset target detection network to obtain a preliminary detection target;

[0174] The association module is used to perform regional feature analysis on the preliminary detected target to obtain corresponding target region feature information;

[0175] A processing module, which is used to obtain environmental factors collected by the multimodal sensing device, perform environmental tracking analysis on the preliminary detection target based on the environmental factors, and obtain the target tracking result;

[0176] The control module is used to perform multi-task analysis on the preliminary detected target through a preset recognition model combined with target area feature information and target tracking results to obtain a corresponding comprehensive recognition result.

[0177] The present invention provides a drone visual recognition device, which has the following beneficial effects:

[0178] By converting and enhancing the data collected by the multimodal perception device, the advantages of different sensors can be fully utilized, the accuracy and robustness of target detection can be improved, and more reliable basic data can be provided for subsequent analysis. According to the preset target detection network, the enhanced image is detected and feature recognition is performed to achieve refined recognition of the target, which helps to improve the accuracy and efficiency of recognition and adapt to complex and changing environmental requirements. By obtaining environmental factors and tracking and analyzing the characteristic information of the target area, dynamic tracking of the target can be achieved, the recognition performance of the system under different environmental conditions is improved, and the interference caused by environmental changes is reduced. The preset recognition model is used to perform multi-task analysis on the target tracking results, and the comprehensive recognition of the target is achieved, which can meet the needs of different mission scenarios and improve the flexibility and adaptability of the system. Through this method of multimodal image fusion and environmental factor consideration, the limitations of single data source and static recognition algorithm are effectively solved, the overall performance and reliability of UAV visual recognition equipment are improved, and strong support is provided for UAVs to perform various tasks in complex environments.

[0179] Reference Figure 3 As shown, the present invention also provides a drone visual recognition device, comprising:

[0180] Memory, used to store programs;

[0181] The processor is used to execute the program to implement each step of any one of the above-mentioned drone visual recognition methods.

[0182] In this embodiment, the processor and the memory may be connected via a bus or other means. The memory may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a read-only memory, a flash memory, a hard disk, or a solid-state drive. The processor may be a general-purpose processor, such as a central processing unit, a digital signal processor, an application-specific integrated circuit, or one or more integrated circuits configured to implement the embodiments of the present invention.

[0183] The present invention also provides a storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute any of the above methods.

[0184] It should be noted that technicians in the relevant technical field can clearly understand that for the convenience and conciseness of description, the specific working process of the system and each module described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0185] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A drone visual recognition method, characterized in that: include: Collecting multimodal images through a multimodal sensing device of the drone, and performing image conversion on the multimodal images to obtain enhanced images; Performing target detection on the enhanced image according to a preset target detection network to obtain a preliminary detection target; Performing regional feature analysis on the preliminary detected target to obtain corresponding target region feature information; Acquire the environmental factors collected by the multimodal sensing device, perform environmental tracking analysis on the preliminary detection target according to the environmental factors, and obtain a target tracking result; Performing a multi-task analysis on the preliminary detected target by combining the target area feature information and the target tracking result through a preset recognition model to obtain a corresponding comprehensive recognition result; The performing of regional feature analysis on the preliminary detected target to obtain corresponding target region feature information includes: Performing region segmentation on the initially detected target to obtain a plurality of target sub-regions; Extracting features from each of the target sub-regions according to a preset adaptive feature extraction algorithm to obtain a sub-region feature set; According to a preset attention mechanism, weights are assigned to the sub-region feature sets to obtain a weighted feature set; Performing topological feature conversion on the weighted feature set to obtain a topological feature representation of the target; Matching the topological feature representation with a preset spatiotemporal feature template to obtain a spatiotemporal correlation score; Screening the topological feature representation according to the spatiotemporal correlation score to obtain a key feature set; Applying dynamic quantization coding to the key feature set to obtain a compressed feature vector; Calibrate and calculate the compressed feature vector according to a preset self-calibration algorithm to obtain feature information of the target area; The acquiring of the environmental factors collected by the multimodal sensing device, and performing environmental tracking analysis on the preliminary detection target according to the environmental factors to obtain the target tracking result, includes: Acquiring multi-dimensional environmental factor data including light intensity, weather conditions and terrain features collected by the multimodal sensing device; Performing matrix conversion on the multi-dimensional environmental factor data to obtain an environmental feature matrix; Performing environmental characteristic analysis on the preliminary detection target according to the environmental characteristic matrix to obtain environmental target characteristics; Performing filtering position calculation on the environmental target features to obtain target predicted position information; Performing target motion vector analysis on the target predicted position information to obtain a target motion trend; Performing trajectory analysis on the target motion trend according to a preset spline interpolation algorithm to obtain the target tracking result; The multi-task analysis of the preliminary detected target is performed by combining the target area feature information and the target tracking result through the preset recognition model to obtain the corresponding comprehensive recognition result, including: Extracting features from the target area feature information and the target tracking result to obtain a target feature vector; Inputting the target feature vector into the first decision tree layer of the recognition model for preliminary classification to obtain a corresponding preliminary classification result; Selecting a classification neural network layer corresponding to the recognition model according to the preliminary classification result, inputting the target feature vector into the classification neural network layer for initial recognition processing, and obtaining an initial recognition result; Inputting the initial recognition result into the second-layer decision tree of the recognition model for fine-grained classification to obtain a fine-grained classification result; Inputting the fine-grained classification result and the preliminary detection target into the task neural network layer of the recognition model, and performing a multi-task analysis including target classification, posture estimation and behavior recognition on the preliminary detection target according to the fine-grained classification result to obtain a multi-task analysis result; Inputting the multi-task analysis results into the third-level decision tree of the recognition model for final decision analysis to obtain a decision result; The decision result is subjected to time series analysis through the output layer of the recognition model to generate and output the comprehensive recognition result.

2. The drone visual recognition method according to claim 1, characterized in that: The method of collecting a multimodal image by a multimodal sensing device of the drone and performing image conversion on the multimodal image to obtain an enhanced image includes: Collecting multimodal images including visible light images, infrared images and range depth images through the multimodal sensing device of the drone; Performing multi-scale fusion of the visible light image, the infrared image and the range depth image to obtain a preliminary fused image; Performing multi-scale spatial pyramid processing on the preliminary fused image to obtain a multi-layer feature map; Performing top-down feature propagation on the multi-layer feature maps to obtain an initial feature pyramid; Performing a 1x1 convolution transformation on each layer of the feature map in the initial feature pyramid to obtain an adjusted feature map; Upsampling and element-wise addition fusion of the adjusted feature maps of adjacent levels to obtain a fused feature map; Performing 3x3 convolution processing on the fused feature map to obtain a smoothed feature map of the corresponding level; The smoothed feature maps of all levels are fused using a preset feature enhancement algorithm to obtain the enhanced image.

3. The drone visual recognition method according to claim 1, characterized in that: The performing target detection on the enhanced image according to a preset target detection network to obtain a preliminary detection target includes: Inputting the enhanced image into a shallow feature extraction layer of the target detection network for visual extraction to obtain a low-level visual feature map; Inputting the low-level visual feature map into the middle-level feature fusion layer of the target detection network for visual extraction to obtain a middle-level semantic feature map; Inputting the intermediate semantic feature map into the deep semantic feature layer of the target detection network for visual extraction to obtain a high-level semantic feature map; Performing feature fusion on the low-level visual feature map, the mid-level semantic feature map, and the high-level semantic feature map to obtain a multi-level visual feature map; Generate an anchor frame according to the multi-level visual feature map to obtain an initial candidate region; Performing feature alignment on the initial candidate region and the multi-level visual feature map to obtain candidate region features; Inputting the candidate region features into the target classifier and bounding box regressor of the target detection network to obtain target refined bounding box coordinates; Performing target screening on the target refined bounding box coordinates according to a preset confidence threshold to obtain a confidence target; Eliminate overlapping detection frames of the confidence target to obtain a confidence detection result; Mapping detection is performed on the enhanced image according to the confidence detection result to obtain a preliminary detection target.

4. The drone visual recognition method according to claim 1, characterized in that: The step of matching the topological feature representation with a preset spatiotemporal feature template to obtain a spatiotemporal correlation score includes: Decomposing the topological feature representation in time and space dimensions to obtain a time feature sequence and a space feature matrix; Calculating the similarity between the time feature sequence and the time template in the spatiotemporal feature template to obtain a time correlation score; Performing a spatial frequency transformation on the spatial feature matrix to obtain a spatial spectrum diagram; Performing a matching operation on the spatial spectrum graph and the spatial template in the spatiotemporal feature template according to a preset image matching algorithm to obtain a spatial correlation score; Performing weighted fusion on the temporal correlation score and the spatial correlation score to obtain an initial temporal and spatial correlation score; The time-space correlation scores are screened according to a preset threshold judgment rule to obtain a valid time-space correlation score; Performing statistical analysis on the effective spatiotemporal correlation scores to obtain the spatiotemporal correlation scores; Among them, the calculation formula for similarity calculation is: : the i-th value in the first time series, : the i-th value in the second time series, : the mean of the first time series, : the mean of the second time series, r: temporal correlation score, the range of r is [-1, 1], where 1 indicates a perfect positive correlation, 0 indicates no correlation, and -1 indicates a perfect negative correlation.

5. A drone visual recognition device, characterized in that: The drone visual recognition method applied to any one of claims 1 to 4 above comprises: An acquisition module, wherein the acquisition module is used to acquire multimodal images through a multimodal sensing device of the drone, and perform image conversion on the multimodal images to obtain enhanced images; An analysis module, the analysis module is used to perform target detection on the enhanced image according to a preset target detection network to obtain a preliminary detection target; An association module, the association module is used to perform regional feature analysis on the preliminary detected target to obtain corresponding target region feature information; A processing module, the processing module is used to obtain environmental factors collected by the multimodal sensing device, perform environmental tracking analysis on the preliminary detection target according to the environmental factors, and obtain a target tracking result; A control module, the control module is used to perform multi-task analysis on the preliminary detected target through a preset recognition model combined with the target area feature information and the target tracking result to obtain a corresponding comprehensive recognition result; The performing of regional feature analysis on the preliminary detected target to obtain corresponding target region feature information includes: Performing region segmentation on the initially detected target to obtain a plurality of target sub-regions; Extracting features from each of the target sub-regions according to a preset adaptive feature extraction algorithm to obtain a sub-region feature set; According to a preset attention mechanism, weights are assigned to the sub-region feature sets to obtain a weighted feature set; Performing topological feature conversion on the weighted feature set to obtain a topological feature representation of the target; Matching the topological feature representation with a preset spatiotemporal feature template to obtain a spatiotemporal correlation score; Screening the topological feature representation according to the spatiotemporal correlation score to obtain a key feature set; Applying dynamic quantization coding to the key feature set to obtain a compressed feature vector; Calibrate and calculate the compressed feature vector according to a preset self-calibration algorithm to obtain feature information of the target area; The acquiring of the environmental factors collected by the multimodal sensing device, and performing environmental tracking analysis on the preliminary detection target according to the environmental factors to obtain the target tracking result, includes: Acquiring multi-dimensional environmental factor data including light intensity, weather conditions and terrain features collected by the multimodal sensing device; Performing matrix conversion on the multi-dimensional environmental factor data to obtain an environmental feature matrix; Performing environmental characteristic analysis on the preliminary detection target according to the environmental characteristic matrix to obtain environmental target characteristics; Performing filtering position calculation on the environmental target features to obtain target predicted position information; Performing target motion vector analysis on the target predicted position information to obtain a target motion trend; Performing trajectory analysis on the target motion trend according to a preset spline interpolation algorithm to obtain the target tracking result; The multi-task analysis of the preliminary detected target is performed by combining the target area feature information and the target tracking result through the preset recognition model to obtain the corresponding comprehensive recognition result, including: Extracting features from the target area feature information and the target tracking result to obtain a target feature vector; Inputting the target feature vector into the first decision tree layer of the recognition model for preliminary classification to obtain a corresponding preliminary classification result; Selecting a classification neural network layer corresponding to the recognition model according to the preliminary classification result, inputting the target feature vector into the classification neural network layer for initial recognition processing, and obtaining an initial recognition result; Inputting the initial recognition result into the second-layer decision tree of the recognition model for fine-grained classification to obtain a fine-grained classification result; Inputting the fine-grained classification result and the preliminary detection target into the task neural network layer of the recognition model, and performing a multi-task analysis including target classification, posture estimation and behavior recognition on the preliminary detection target according to the fine-grained classification result to obtain a multi-task analysis result; Inputting the multi-task analysis results into the third-level decision tree of the recognition model for final decision analysis to obtain a decision result; The decision result is subjected to time series analysis through the output layer of the recognition model to generate and output the comprehensive recognition result.

Images