Aerial video stream target recognition processing method and system

Abstract

The present invention provides an aerial video stream target recognition processing method and system, which relates to the fields of image processing, artificial intelligence and aerial monitoring technology. By preprocessing video, infrared and radar modal data, combined with adaptive weighted fusion technology, multi-stage visual feature extraction and global convolution optimization technology, the system can extract target features from multi-level and multi-modal data; further combined with historical motion trajectories and future motion predictions, the system can dynamically correct the target position, and finally ensure the consistency of multi-modal data through cross-validation. The present invention enhances the recognition accuracy and robustness of the system for weak and small targets, reduces missed detection and false detection, has strong adaptability and low computing resource requirements, and is suitable for target recognition and tracking tasks in complex dynamic environments.

Description

Technical Field

[0001] The present invention relates to the fields of image processing, artificial intelligence and aviation monitoring technology, and in particular to an aviation video stream target recognition processing method and system. Background Art

[0002] With the widespread application of drones, aerial reconnaissance and monitoring systems, real-time and accurate target recognition technology is crucial to aviation safety and mission execution. The existing technology (China invention application, publication number: CN116630849A, name: A method for improving the performance of an aerial video stream target recognition algorithm) mainly relies on deep learning algorithms, such as convolutional neural networks (CNNs) to extract features from images and identify targets. However, in aerial scenes, due to the small size of targets, weak signals (low signal-to-noise ratio), and the complexity of multimodal data, the existing technology faces the following shortcomings:

[0003] Existing technologies have limited detection capabilities for small targets, especially in complex backgrounds, and are prone to missed detections or false detections; existing technologies lack sufficient robustness when faced with dynamic environments and multimodal data in aerial scenarios; many algorithms are difficult to apply in resource-limited scenarios (such as satellites or mobile devices) due to their complex steps and reliance on large amounts of computing resources; existing technologies often require retraining models for specific scenarios or data, and lack plug-and-play flexibility. Summary of the invention

[0004] In view of the many problems existing in the above-mentioned prior art, the present invention provides an aerial video stream target recognition processing method and system, which can accurately recognize and track targets in real time in dynamic and complex aviation environments through technologies such as adaptive weighted fusion, multimodal data cross-validation, global convolution optimization and future motion prediction. The robustness and recognition accuracy of the system are significantly improved, especially in multimodal data and rapidly changing environments.

[0005] A method for identifying and processing an aerial video stream target comprises the following steps:

[0006] By preprocessing the video data, infrared data and radar data, each modal data is converted into corresponding modal feature data, and then fused through adaptive weighted fusion technology to generate fused modal feature data;

[0007] Based on the fusion modality feature data, low-level, medium-level and high-level multi-stage visual feature extraction is performed, and the visual features are globally enhanced and optimized through the global convolution optimization technology to generate globally optimized visual feature data;

[0008] By modeling the global optimized visual feature data and the historical motion trajectory of the target, the future motion trajectory of the target is predicted by combining the future motion prediction technology, and the current position of the target is corrected according to the target motion prediction data to generate the corrected target position data;

[0009] The corrected target position data is cross-validated with the video modal feature data, the infrared modal feature data and the radar modal feature data to generate final target position data and target category data, and the final target position data and target category data are output.

[0010] Preferably, the video data, infrared data and radar data are preprocessed, specifically including:

[0011] Removing noise from the video data by filtering algorithms to make the edge and detail information of the video data clearer;

[0012] Normalizing the pixel values ​​of the infrared data to enhance the contrast between different temperature regions and ensure that the infrared data can accurately reflect the temperature distribution;

[0013] The radar data is processed by a distance normalization algorithm to standardize the reflected signal strength of the radar data and enhance the resolution of the distance information.

[0014] Preferably, the adaptive weighted fusion technology determines the weighting factor by calculating the confidence of each modal feature data, and the confidence is calculated according to the signal strength and clarity of each modal data and the identifiability of the target in different modalities. The weighting factor is calculated as follows:

[0015] Using weighting factors , and They represent the weights of video modal feature data, infrared modal feature data, and radar modal feature data, respectively, and their calculation formulas are as follows:

[0016] ;

[0017] in, is the weighting factor of the video modality feature data; is the weighting factor of infrared modal characteristic data; is the weighting factor of radar modal characteristic data; Indicates the confidence of the video modality feature data; Indicates the confidence level of infrared modal feature data; Indicates the confidence level of radar modal characteristic data.

[0018] Preferably, based on the fusion modality feature data, low-level, mid-level and high-level multi-stage visual feature extraction is performed, specifically including:

[0019] Extracting low-level visual features through large-size convolution kernels, wherein the extracted low-level visual features mainly include the edges of objects and basic features of simple geometric shapes;

[0020] Extracting mid-level visual features through a medium-sized convolution kernel, wherein the extracted mid-level visual features include shape, texture, and detail information of the object;

[0021] High-level visual features are extracted through small-size convolution kernels. The extracted high-level visual features include the overall outline and complex shape of the target, ensuring that different visual features of the target are accurately captured at each level.

[0022] Preferably, the global convolution optimization technology uniformly optimizes low-level visual feature data, mid-level visual feature data and high-level visual feature data through a global convolution network. The global convolution network ensures the consistency of visual features at each level on a global scale by analyzing the overall spatial distribution of the target. The global convolution network covers the entire input feature space through convolution kernels, captures long-distance correlations in the scene, and generates globally optimized visual feature data.

[0023] Preferably, the historical motion trajectory modeling of the target is based on historical motion data, and is modeled by analyzing the dynamic characteristics of the target's position information, speed and motion direction in several past frames. The historical motion data includes each position, timestamp and motion vector of the target. By performing regression analysis on the historical motion data, the future motion trend of the target is predicted and the target motion trajectory characteristic data is generated.

[0024] Preferably, the future motion prediction technology performs spatiotemporal inference on the target's motion trajectory feature data, and the spatiotemporal inference is based on the target's historical motion pattern and environmental change factors, combined with multiple frames of historical data to analyze the target's speed changes and direction adjustments, to generate the target's future motion trajectory, and corrects the current target position according to the target's future motion trajectory to generate corrected target position data.

[0025] Preferably, the corrected target position data is cross-validated with the video modal feature data, the infrared modal feature data and the radar modal feature data, specifically including: by comparing the corrected target position data with the video modal feature data, the infrared modal feature data and the radar modal feature data, calculating the position deviation and shape similarity of the target in different modalities, ensuring that the position information of the target in each modality is consistent, and cross-validating the corrected target position data with the video modal feature data, the infrared modal feature data and the radar modal feature data, specifically including: including position correction and shape consistency correction, and generating modal similarity data.

[0026] Preferably, the shape similarity of the target in different modes is calculated using the cosine similarity formula. It represents the similarity between the corrected target position data and the modal feature data, and the calculation formula is:

[0027]

[0028] in, is the modal similarity; A vector representation of the corrected target position data; is the vector representation of modal characteristic data; Represents vector dot product operation; Represents the modulus of the vector; the cosine similarity is used to ensure that the shape information of the target in different modes remains consistent.

[0029] A system for implementing the method for processing target recognition in an aerial video stream, comprising:

[0030] A preprocessing module, used for receiving and processing the video data, infrared data and radar data, and converting the video data, infrared data and radar data into video modal feature data, infrared modal feature data and radar modal feature data respectively;

[0031] A weighted fusion module, used to generate fused modal feature data based on the video modal feature data, the infrared modal feature data and the radar modal feature data by using an adaptive weighted fusion technology;

[0032] A visual feature extraction and optimization module, which is used to perform low-level, mid-level and high-level multi-stage visual feature extraction based on the fused modality feature data, and globally enhance and optimize the extracted visual features through global convolution optimization technology to generate globally optimized visual feature data;

[0033] A motion prediction and correction module, which is used to model the target based on the global optimized visual feature data and the historical motion trajectory of the target, predict the future motion trajectory of the target in combination with the future motion prediction technology, and correct the current position of the target according to the target motion prediction data to generate corrected target position data;

[0034] A cross-validation module, used to cross-validate the corrected target position data with the video modality feature data, infrared modality feature data and radar modality feature data to generate final target position data and target category data;

[0035] The output module is used to output the final target position data and target category data for system control or visual display.

[0036] Compared with the prior art, the advantages and beneficial effects of the present invention are:

[0037] The present invention improves the accuracy and robustness of target recognition by calculating the modal feature weights of video, infrared and radar data and dynamically adjusting the contribution of each modal data;

[0038] The system of the present invention can gradually extract the edge, shape and complex contour of the target, ensuring accurate recognition of the target at different visual levels;

[0039] The present invention predicts the future motion trajectory of the target through the historical motion data of the target combined with environmental factors, thereby enhancing the adaptability of the system in a rapidly changing environment;

[0040] The present invention realizes globally consistent target recognition through global convolution optimization technology, which ensures the global consistency of different modal data and different levels of visual features and reduces the deviation between modalities;

[0041] By comparing multimodal data, the system of the present invention can effectively correct the position information deviation between modes and improve the overall recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of the method of the present invention;

[0043] Figure 2 A schematic diagram of future motion prediction and position correction in the method of the present invention;

[0044] Figure 3 Schematic diagram of multimodal data cross-validation in the method of the present invention;

[0045] Figure 4 It is a structural block diagram of the system of the present invention. DETAILED DESCRIPTION

[0046] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure.

[0047] like Figure 1 As shown, the method for processing target recognition in an aerial video stream comprises the following steps:

[0048] By preprocessing the video data, infrared data and radar data, each modal data is converted into corresponding modal feature data, and then fused through adaptive weighted fusion technology to generate fused modal feature data;

[0049] Preferably, the video data, infrared data and radar data are preprocessed, specifically including:

[0050] Removing noise from the video data by filtering algorithms to make the edge and detail information of the video data clearer;

[0051] Normalizing the pixel values ​​of the infrared data to enhance the contrast between different temperature regions and ensure that the infrared data can accurately reflect the temperature distribution;

[0052] The radar data is processed by a distance normalization algorithm to standardize the reflected signal strength of the radar data and enhance the resolution of the distance information.

[0053] The noise in the video data is removed by filtering algorithms. Due to the complex shooting environment, the aerial video stream is often affected by factors such as air disturbance and camera equipment noise, resulting in more noise in the image. These noises will affect the extraction of image edges and detail features, thereby affecting the accuracy of target recognition. Therefore, in the present invention, a filtering algorithm (such as Gaussian filtering, mean filtering, etc.) is used to denoise the video data. The core principle of this filtering algorithm is to smooth the abnormal pixels (i.e., noise) by adjusting the grayscale value of the pixels in the image while retaining the key edge information. In the specific operation, by selecting the appropriate filter window size and kernel function parameters, the optimal degree of retention of edge information is ensured. After this step of processing, the noise in the video data is greatly reduced, the edge information is clearer, and the detail features are more significant, which provides higher quality input data for subsequent visual feature extraction and target edge detection.

[0054] Normalize the pixel values ​​of infrared data. In the target recognition of aerial video streams, infrared data can provide information on temperature distribution and help identify targets based on thermal signals. However, due to the characteristics of infrared sensors and fluctuations in external ambient temperature, there may be significant temperature differences in infrared images in different scenes, which will cause changes in image brightness values. Therefore, it is necessary to normalize infrared images to unify the pixel value range in different scenes to enhance the contrast between different temperature areas. The principle of normalization is to linearly stretch the pixel values ​​of infrared images so that the pixel values ​​of the image are mapped in the grayscale range of 0 to 255, thereby enhancing the contrast between low-temperature and high-temperature areas and ensuring that the thermal signal of the target is more prominent in the image. This processing method can not only reduce the problem of uneven temperature distribution in the image, but also significantly enhance the difference between the target area and the background, thereby improving the visibility of the target. The effect of this normalization processing is manifested in practical applications as follows: the temperature contrast of the infrared image after normalization is enhanced, making low-thermal radiation targets and high-thermal radiation targets easier to distinguish visually, providing higher-resolution infrared features for subsequent modal fusion and target recognition.

[0055] Radar data is processed by a distance normalization algorithm. Radar data is mainly used to provide distance and azimuth information in aviation target recognition, and the intensity of the radar reflection signal is usually inversely proportional to the distance of the target. However, due to the large difference in the distances of different targets, unprocessed radar data may have uneven reflection intensity, which in turn affects the detection accuracy of the target. Therefore, the present invention uses a distance normalization algorithm to standardize radar data, unify the reflection signal intensity of targets at different distances, and enable the radar reflection characteristics of each target to be compared and analyzed under the same standard. The normalization process mainly adjusts the reflection signals of targets at different distances to a unified scale range by segmenting the reflection signal intensity of the target. In the specific implementation, the system first estimates the distance of the target, and then normalizes the reflection signal according to the estimated distance value. The algorithm effectively solves the problem of signal intensity difference caused by the target distance, enhances the resolution of the distance information, and ensures that in the subsequent multimodal fusion process, the radar features can be effectively matched and fused with the video and infrared features.

[0056] Preferably, the adaptive weighted fusion technology determines the weighting factor by calculating the confidence of each modal feature data, and the confidence is calculated according to the signal strength and clarity of each modal data and the identifiability of the target in different modalities. The weighting factor is calculated as follows:

[0057] Using weighting factors , and They represent the weights of video modal feature data, infrared modal feature data, and radar modal feature data, respectively, and their calculation formulas are as follows:

[0058] ;

[0059] in, is the weighting factor of the video modality feature data; is the weighting factor of infrared modal characteristic data; is the weighting factor of radar modal characteristic data; Indicates the confidence of the video modality feature data; Indicates the confidence level of infrared modal feature data; Indicates the confidence level of radar modal characteristic data.

[0060] The principle of adaptive weighted fusion is to determine the weighting factors of each modal feature data based on the confidence level. Confidence is defined in the present invention as the signal strength, clarity and identifiability of the target in different modalities in the modal data. These indicators are used to evaluate the contribution of different modalities to target recognition at a given moment, thereby dynamically adjusting the weight of each modal feature data in the final fusion.

[0061] In practical applications, video data provides information about the visual appearance of the target, and can usually capture the shape and outline of the target; infrared data captures the thermal characteristics of the target based on temperature distribution, and is suitable for target identification under poor lighting conditions; radar data is mainly used to provide distance and direction information. Due to the different characteristics of these modal data, their reliability in specific scenarios also varies. For example, video data is less effective in low-visibility environments (such as at night or in fog), while infrared data may be interfered by background interference in high-temperature environments. Therefore, the contribution of each modality in the final fusion must be determined based on its confidence.

[0062] In the present invention, the confidence , and The confidence levels of the video modality feature data, infrared modality feature data, and radar modality feature data are respectively. The confidence levels are calculated based on signal strength (such as image contrast, thermal signal strength, etc.), clarity (such as edge sharpness, target outline clarity), and target recognizability (whether the target can be reliably detected in each modality). These confidence levels are evaluated by analyzing the effectiveness of each modality in target recognition. For example, when visibility is low, the confidence level of infrared data is Higher, confidence level of video data Lower, while radar data The dynamic adjustment of confidence ensures that each modality data can optimally contribute to target recognition under different conditions.

[0063] The above formula represents the proportion of the confidence of each modal feature data in the overall weighting process, ensuring that the modal data with higher confidence has a greater proportion in the final fusion result. When high, the weighting factor of the video feature data It is also correspondingly higher, indicating that the contribution of video features in the fusion result increases; conversely, when the video data is unreliable, its weighting factor decreases accordingly.

[0064] At the application level, the weighting process effectively avoids the problem of fixed weights in traditional multimodal fusion by adaptively adjusting the contribution of each modal feature data. For example, in the case of good visibility during the day, the confidence of the video modal feature data is high, and the weight of infrared and radar data may be low; at night or in bad weather conditions, the confidence of infrared and radar data increases, and the weight of video data decreases. In this way, the system can automatically and dynamically adjust the weight according to environmental changes to ensure the robustness and accuracy of target recognition.

[0065] For example, in a low-visibility night flight scenario, the video data has a weak signal strength due to insufficient light, and the confidence The infrared data can capture the thermal characteristics of the target more clearly, so the confidence Ascending, radar data provides stable distance information, confidence Stable. Through the weighted formula, the weighting factors of infrared data and radar data increase accordingly, while the weighting factor of video data decreases. In this way, the system relies more on infrared and radar information for target recognition at this time, and the fused modal feature data finally generated can reliably reflect the position and thermal characteristics of the target. Even if the video signal is not good, the system can still accurately identify the target.

[0066] The system first calculates the confidence of video, infrared and radar data through independent processing modules. In each time frame, the system dynamically updates the confidence value based on the current environmental conditions, target characteristics and signal strength. , and Next, using the above weighted formula, the system calculates the weighted factor of each modality. Finally, the system multiplies the feature data of each modality by its corresponding weighted factor to generate the final fused modality feature data.

[0067] Based on the fusion modality feature data, low-level, medium-level and high-level multi-stage visual feature extraction is performed, and the visual features are globally enhanced and optimized through the global convolution optimization technology to generate globally optimized visual feature data;

[0068] Preferably, based on the fusion modality feature data, low-level, mid-level and high-level multi-stage visual feature extraction is performed, specifically including:

[0069] Extracting low-level visual features through large-size convolution kernels, wherein the extracted low-level visual features mainly include the edges of objects and basic features of simple geometric shapes;

[0070] Extracting mid-level visual features through a medium-sized convolution kernel, wherein the extracted mid-level visual features include shape, texture, and detail information of the object;

[0071] High-level visual features are extracted through small-size convolution kernels. The extracted high-level visual features include the overall outline and complex shape of the target, ensuring that different visual features of the target are accurately captured at each level.

[0072] Multi-stage visual feature extraction is a key technology to ensure that the target can be accurately captured at different visual levels. Since the targets involved in the aerial video stream usually have complex geometric structures, variable appearance features and unstable shooting angles, single-level feature extraction often cannot fully reflect the various features of the target. Therefore, the present invention adopts a hierarchical multi-stage visual feature extraction strategy, combined with convolution kernels of different sizes, to gradually extract various features of the target from low level to high level, ensuring that various target features can be accurately captured and enhanced in the entire processing chain.

[0073] The core idea of ​​multi-stage visual feature extraction is to use convolution kernels of different sizes to extract multi-level features of the target. Specifically, large-sized convolution kernels are used to extract low-level features, medium-sized convolution kernels are used to extract medium-level features, and small-sized convolution kernels are used to extract high-level features. This hierarchical convolution process enables the edge information, geometric shape, texture details, and complex structure of the target to be extracted and enhanced step by step.

[0074] Low-level features mainly include the edges and basic geometric shapes of objects. These features usually have a simple spatial distribution in the image, but are crucial for the positioning and preliminary recognition of the target. Through processing with large-size convolution kernels, the system can effectively capture the edge and contour features of the target in the video frame. The principle of large-size convolution kernels is to perform calculations through a larger range of pixel neighborhoods, so that a wider range of spatial edge information can be extracted. For example, larger structures such as the wingspan of aircraft in aerial video streams can be clearly extracted through processing with large-size convolution kernels. In specific operations, the neighborhood of each pixel point is weighted by convolution operations, and the size of the convolution kernel determines the range and accuracy of the captured edge details.

[0075] Medium-level features include the shape, texture, and detail information of an object. Through medium-sized convolution kernels, the system can better extract the internal detail features of the target, such as the texture and markings on the surface of the object. Medium-sized convolution kernels can both cover the local features of the target in space and retain the correlation between adjacent features. In practical applications, this level of feature extraction can help the system identify the surface material, texture pattern and other features of aviation targets. For example, when identifying an aircraft, information such as the paint on the wing and the reflective material on the aircraft surface can be clearly extracted through this convolution kernel processing, providing the system with richer detail features.

[0076] High-level features mainly include the overall outline and complex shape of the target, reflecting the global structure and morphology of the target. Through the processing of small-sized convolution kernels, the system can accurately capture smaller features to ensure the combination of global features and local details of the target. The role of small-sized convolution kernels is to perform fine calculations through smaller neighborhoods to capture the detailed features of the target and combine them with the overall outline. For example, for some small aerial targets, such as drones or aircraft attachments, the overall outline may not be obvious, but through the processing of small-sized convolution kernels, the system can accurately capture these smaller features and combine them with the high-level overall morphology to achieve accurate recognition.

[0077] In the actual system implementation, the multi-stage visual feature extraction module processes the input fusion modality feature data in sequence through a hierarchical convolutional network structure. The system first performs preliminary edge and contour extraction on the input visual data through a large-size convolution kernel, then further captures the shape and texture features of the target in a medium-size convolution kernel, and finally refines the complex morphology and global contour of the target through a small-size convolution kernel. The convolution operation at each stage is adaptively adjusted according to the spatial frequency characteristics of the input data and the specific target type to ensure that the extracted features are sufficiently representative and discriminative.

[0078] In order to enhance the connection between different levels, the system combines the feedback mechanism between feature layers in the multi-stage visual feature extraction process to ensure that low-level features can affect the extraction of high-level features, and high-level features can react to the enhancement of low-level features in combination with contextual information. This hierarchical convolution processing method not only improves the robustness of the system, but also provides sufficient feature information for complex targets (such as aircraft, multi-object recognition, etc.), making the final recognition results more accurate.

[0079] The effect of multi-stage visual feature extraction in practical applications is that the system can capture different feature dimensions of the target at various levels, and finally obtain a target feature map containing multiple layers of information. This feature map can accurately represent the edge, shape, texture and global contour of the target, ensuring that in the subsequent target classification and recognition stages, the system can make more detailed target judgments based on this feature map.

[0080] For example, the system can identify different types of aircraft in complex aviation scenarios. Even at long distances, low-level convolution kernels can extract the overall outline of the aircraft, and high-level convolution kernels can capture subtle structural features of the aircraft, ultimately helping the system to accurately identify targets in harsh environments.

[0081] Preferably, the global convolution optimization technology uniformly optimizes low-level visual feature data, mid-level visual feature data and high-level visual feature data through a global convolution network. The global convolution network ensures the consistency of visual features at each level on a global scale by analyzing the overall spatial distribution of the target. The global convolution network covers the entire input feature space through convolution kernels, captures long-distance correlations in the scene, and generates globally optimized visual feature data.

[0082] The core of global convolution optimization technology is to use global convolutional networks to uniformly process and optimize visual feature data at different levels. Targets in aerial video streams often have complex structures and diverse features. Especially when facing moving objects, occlusions or long-distance monitoring, the extraction of local features may not be sufficient to fully characterize the overall picture of the target. To this end, the global convolutional network covers the entire input feature space, analyzes the global spatial distribution of the target, and ensures that visual features at all levels are coordinated globally to make them consistent and global.

[0083] In a specific implementation, low-level visual feature data usually includes the edge and outline of the target, mid-level visual feature data reflects the geometric shape and texture of the target, and high-level visual feature data is used to capture the overall morphology and complex structure of the target. Although features at different levels can extract local information of the target respectively, in order to improve the global accuracy of target recognition, these levels of information must be integrated through a global convolutional network. In the present invention, the convolution kernel of the global convolutional network covers the entire input feature space, and through the operation of a larger receptive field, the data of features at each level can be fused and optimized on a global scale to capture long-distance correlation features in the scene. This long-distance correlation refers to the relationship between different parts of the target in the global space, especially in the case of long distance or partial occlusion, local features are often insufficient for effective recognition, and global optimization can enhance the interconnection between these features to ensure the integrity of the target.

[0084] For example, when identifying an aircraft in an aerial video stream, the aircraft's wingspan, fuselage, and tail may appear in different areas of the image, and a single local feature cannot effectively associate these parts. However, through the global convolutional network, the global convolution kernel can simultaneously capture the overall spatial layout of the aircraft, associate the features of different parts, and generate globally optimized visual feature data. These global features can more accurately describe the overall structure of the target, which helps to further improve the recognition accuracy of the target.

[0085] In the specific technical implementation, the structure of the global convolutional network is composed of several levels of convolutional layers and pooling layers. The receptive field of each convolution kernel gradually increases, eventually covering the entire input feature space. During the optimization process, the data of low-level features is first processed by the global convolutional network to extract its features associated with global information; then, the mid-level and high-level features are processed in turn through the same processing mechanism to ensure that at each level, various local features can be associated with global features.

[0086] The size and number of convolution kernels directly affect the effect of global optimization. In order to capture long-distance associations, the global convolution kernels in the present invention are usually large, with a wide receptive field, and can span multiple levels of feature information. In addition, the system dynamically adjusts the parameters of the convolution kernel through an adaptive convolution parameter adjustment mechanism to meet the needs of different scenarios. For example, in large-scale monitoring or complex backgrounds, the parameters of the convolution kernel will automatically adjust to adapt to the complexity of the background and the diversity of the target.

[0087] The application effect of global convolution optimization technology is reflected in the global consistency and long-distance correlation enhancement of target recognition. First, this technology can effectively integrate visual features at different levels, so that in the recognition process, the local features and global features of the target can cooperate with each other, avoiding problems such as incomplete local feature extraction and blurred global features. Secondly, by capturing long-distance correlation, this technology can significantly improve target recognition capabilities in complex scenes, especially for long-distance targets, partially occluded targets or multi-target scenes. The system can better handle the relative position information and morphological consistency of the target.

[0088] For example, in an aerial target recognition task, the system needs to identify an aircraft that is partially obscured by clouds. Relying solely on low-level edge features or mid-level texture features may not be able to fully depict the overall shape of the aircraft. Through the global convolutional network, the system can extract local information related to the aircraft from different feature levels, and integrate this local information globally to capture the long-range correlation features of the aircraft's fuselage, wingspan, and tail, thereby generating a globally optimized visual feature data. Ultimately, even in a complex environment, the system can still accurately identify the position, shape, and category of the aircraft.

[0089] The global convolution optimization technology in the present invention is not only applicable to standard aerial target recognition tasks, but can also be effectively applied to multi-target recognition, target tracking and other tasks. In these tasks, global optimization technology helps to improve the accuracy of target classification and tracking by integrating long-range correlation information to ensure the consistency of the target in spatial distribution. For example, in a multi-target scene, the relative position relationship between different targets and the global structural information of the target are very important. Through global convolution optimization, it can be ensured that the visual features of multiple targets are correctly associated and distinguished, thereby improving the recognition efficiency and accuracy of the system.

[0090] By modeling the global optimized visual feature data and the historical motion trajectory of the target, the future motion trajectory of the target is predicted in combination with the future motion prediction technology, and the current position of the target is corrected according to the target motion prediction data to generate the corrected target position data.

[0091] By modeling the global optimized visual feature data and the historical motion trajectory of the target, combined with future motion prediction technology, the prediction of the target's future motion trajectory is a key step to ensure accurate tracking and positioning of the target in a dynamic environment. This step is not only used in the target recognition system to locate the target in real time, but also to predict the target's future position, further improving the system's response speed and accuracy.

[0092] By modeling the global optimized visual feature data and the historical motion trajectory of the target, a prediction model is constructed using the speed, acceleration, direction and other information in the historical trajectory, and combined with future motion prediction technology, the possible position of the target at a future moment is inferred. Aerial targets usually have high speeds and irregular motion trajectories, which makes it difficult for traditional single-frame target detection technology to accurately track the target. Through the modeling of the motion trajectory, the system can capture the position, motion direction and other information of the target at multiple moments in the past, and predict the future motion state based on this information.

[0093] The globally optimized visual feature data provides accurate visual information of the target at the current moment, including the shape, position, and other key features of the target. This data is optimized by the global convolutional network, has spatial consistency, and can accurately reflect the global position of the target in the current frame. The use of this data ensures that the system can perform motion modeling based on the optimized target features, providing an accurate basis for subsequent trajectory prediction.

[0094] The historical motion trajectory refers to the motion state of the target in the past several frames, including its position, speed, direction and other information. The system extracts the dynamic features of the target by tracking the change pattern in the historical trajectory and associates it with the current global optimized visual feature data. Common modeling methods include trajectory modeling techniques based on Kalman filtering or long short-term memory network (LSTM). Kalman filtering corrects the future position of the target by continuously updating the estimated position and actual measurement value of the target; while the LSTM network can capture the temporal dependency in the target's motion, which is particularly suitable for modeling complex nonlinear motion trajectories.

[0095] After modeling the historical motion trajectory, the system uses future motion prediction technology to predict the future trajectory of the target. The principle of future motion prediction technology is to analyze the motion parameters of the target such as acceleration and direction change, combined with external environmental factors (such as wind speed, airflow, etc.), to infer the position of the target at the future moment. This technology often uses methods such as spatiotemporal convolutional networks (STCNN) or deep reinforcement learning to predict the possible position of the target in the next few seconds through the input of multiple frames of historical data. The spatiotemporal convolutional network can capture the change pattern of the target in the time and space dimensions, thereby generating a more accurate prediction of the future motion trajectory.

[0096] Based on the predicted future motion trajectory, the system will correct the target position at the current moment. This process mainly matches the predicted future trajectory with the current visual feature data to ensure that the actual position of the target is consistent with the trajectory prediction. In some complex scenes, the target may have incomplete visual features in the current frame due to occlusion or rapid movement. At this time, the correction of the motion trajectory can effectively compensate for the lack of visual features, thereby ensuring the system's continuous tracking of the target. For example, if a target is partially obscured by an obstacle in the current frame, the system can predict the target's next position based on its historical trajectory, and correct the position in the current frame to generate the final corrected target position data.

[0097] In aerial video streams, the movement of targets is usually irregular and has large changes, so this method is particularly important. Through trajectory modeling and future motion prediction, the system can maintain continuous tracking of the target when the target moves quickly or is partially occluded. For example, in complex airspace, multiple aircraft fly at different altitudes and orientations, and traditional visual tracking algorithms may lose the target due to sudden changes in speed or direction. Through motion trajectory modeling, the system can capture dynamic changes such as acceleration, deceleration, and steering of the aircraft, and then use prediction technology to calculate its position at the next moment in advance to ensure that the target will not be lost.

[0098] A typical application scenario is target identification and tracking of drones. During the flight of a drone, its speed and altitude change frequently and may be affected by factors such as wind speed and obstacles. Through historical motion trajectory modeling, the system can quickly understand the movement patterns of the drone, and combined with future motion prediction technology, adjust the monitoring range of the camera or radar in advance so that the system can continue to track the target. The corrected target position data ensures that the system can still accurately locate the target in a rapidly changing environment to avoid losing the target due to emergencies.

[0099] In actual applications, the system first generates the target's motion parameters, such as speed, acceleration, and direction change, by modeling the historical motion trajectory. Then, the system combines the global optimization visual feature data and uses future motion prediction technology to infer the target's possible future position. Based on this predicted trajectory, the system corrects the target's current position when visual information is missing or the environment changes suddenly, thereby generating corrected target position data. Ultimately, the corrected target position data provides more accurate information support for the system's target tracking, recognition, and decision-making.

[0100] In the specific implementation, the system first collects the historical motion trajectory data of the target, models it through time series networks such as LSTM, and extracts the motion law of the target. In each time frame, combined with the current global optimized visual feature data, the system uses a spatiotemporal convolutional network to predict the future motion trajectory of the target. By comparing the target position of the current frame and the predicted frame, the system corrects the current position of the target to ensure the continuity and consistency of the target position. This correction process is particularly important for the application of target recognition systems in complex environments, and can effectively deal with situations where the target is lost, blocked, or suddenly changed.

[0101] Preferably, the historical motion trajectory modeling of the target is based on historical motion data, and is modeled by analyzing the dynamic characteristics of the target's position information, speed and motion direction in several past frames. The historical motion data includes each position, timestamp and motion vector of the target. By performing regression analysis on the historical motion data, the future motion trend of the target is predicted and the target motion trajectory characteristic data is generated.

[0102] The core of target motion trajectory modeling is to use historical motion data to analyze the dynamic behavior of the target and apply it to the prediction of future motion trajectories. Specifically, historical motion data includes the target's position information, speed, motion direction and other dynamic characteristics in the past frames. By performing regression analysis on these historical data, the system can identify the target's motion pattern and generate target motion trajectory feature data, providing a reliable basis for the prediction of future motion.

[0103] The main components of historical motion data include each position of the target (specific position coordinates in a specific time frame), timestamp (indicating the time point corresponding to the position), and motion vector (dynamic information consisting of position, speed, and direction). These data are acquired in real time through the target detection and tracking module in the multi-frame video stream and stored in the system's motion trajectory database. After obtaining enough historical data, the system will analyze the target's motion behavior based on these data.

[0104] An important part of target motion trajectory modeling is to perform regression analysis on historical motion data to establish a mathematical model of the target motion trajectory. The principle of regression analysis is to predict the future position of the target by fitting a function curve that can describe the target motion pattern. For example, based on the target position coordinates in a multi-frame video stream, the system can analyze the target's speed change, acceleration trend, and direction change through linear regression or polynomial regression. This regression model can effectively capture the overall trend of the target motion trajectory and can adapt to the dynamic changes of the target speed and direction. The implementation of regression analysis can adapt to various complex motion patterns, such as uniform linear motion, non-uniform motion, or curved motion, thereby improving the prediction accuracy of the target's future motion trend.

[0105] For example, when tracking an aircraft, if the aircraft has maintained a relatively stable speed and direction in the past few frames, the system can use linear regression to predict its trajectory in the next few frames. However, if the aircraft makes a sharp turn or accelerates, the system can use quadratic or high-order polynomial regression to fit its trajectory changes, thereby better capturing the target's motion dynamics.

[0106] The motion vector information in the historical motion data plays an important role in target trajectory modeling. The motion vector contains the target's speed (i.e., the displacement change per unit time) and direction (determined by the target's motion path). By cumulatively analyzing the motion vectors in multiple frames, the system can identify whether there are regular changes in the target's motion direction and the changing trend of the target's speed. The analysis of motion vectors can help the system better understand the target's motion characteristics in a complex environment and provide richer data support for the regression model.

[0107] Based on modeling based on historical motion data, the system can predict the future motion trajectory of the target in combination with the results of regression analysis. In specific operations, the system will predict the specific position of the target in several future frames based on the motion trajectory feature data obtained by modeling. This prediction process is usually performed recursively, that is, the system will use the target position predicted at the previous moment as the input for the next moment, and continuously iterate and update the position coordinates and motion direction of the target. Through this recursive prediction, the system can grasp the position changes of the target at future moments in advance, ensuring that the target can still be accurately tracked when the target moves quickly or the environment is complex.

[0108] By modeling the target's motion trajectory, the system can effectively deal with the rapid movement, direction changes, or partial occlusion of the target in the aerial video stream. Specifically, the effect of this technology is mainly reflected in the following aspects:

[0109] When the target moves fast or is obstructed, it may be difficult for the system to accurately identify the target's position in each frame. By modeling the historical motion trajectory, the system can make predictions based on the target's motion trend. Even if the target is temporarily invisible in some frames, the system can still make reasonable inferences about the target's position through trajectory prediction, thereby maintaining continuous tracking of the target.

[0110] The target motion trajectory modeling combines historical position information with motion vectors, and can take into account changes in speed and direction when predicting the target's future position. Therefore, the system can more accurately predict the position of the target in different motion modes, and even if the target has complex motion behaviors such as acceleration, deceleration or turning, the system can still maintain high positioning accuracy within multiple frames.

[0111] In the target recognition of aerial video streams, the movement of the target is often affected by environmental factors, such as wind speed, airflow and other external interference. These factors may cause the trajectory of the target to change suddenly. Through trajectory modeling and motion vector analysis, the system can capture the dynamic changes in the target trajectory in real time and update the prediction model in time to ensure that the system can adapt to changes in the external environment and maintain accurate tracking of the target.

[0112] In the application of drone target recognition and tracking, the target motion trajectory modeling and prediction technology has significant advantages. Suppose the system tracks a drone in a complex aviation environment. In the initial stage, the system captures the position information of the drone in each frame through the video stream and generates its historical motion data. After several frames, the system uses linear regression to analyze the motion trajectory of the drone and finds that it moves in a certain direction at a constant speed. However, as the drone encounters changes in wind speed, the system detects that the speed and direction in its motion vector have changed significantly. At this time, the system remodels using a high-order polynomial regression model and predicts the possible turning path of the drone in the next few frames. Through this process, the system can not only accurately track the motion trajectory of the drone, but also predict its future position after turning in advance, thereby avoiding tracking interruptions caused by sudden acceleration or turning of the drone.

[0113] Preferably, Figure 2 As shown, the future motion prediction technology performs spatiotemporal inference on the target's motion trajectory feature data. The spatiotemporal inference is based on the target's historical motion pattern and environmental change factors, combined with multiple frames of historical data to analyze the target's speed changes and direction adjustments, generate the target's future motion trajectory, and correct the current target position according to the target's future motion trajectory to generate corrected target position data.

[0114] By performing spatiotemporal inference on the target's motion trajectory feature data, it is possible to predict the target's future motion trajectory and correct the current target position based on historical motion patterns and environmental change factors. The core of this technology is to combine the target's motion information in the past few frames with changes in the external environment to ensure the system's tracking accuracy in complex dynamic environments.

[0115] The system extracts feature data from the target's historical motion trajectory, which usually includes the target's position information, speed, acceleration and other dynamic parameters in the past time frames. These historical data are an important basis for predicting the target's future motion. The system infers the future motion trajectory by analyzing the target's motion trend in multiple frames. Spatiotemporal inference technology combines information from the time dimension and the space dimension for comprehensive analysis, considering not only the target's motion changes in time, but also its position adjustment in space. Through this process, the system can accurately determine the acceleration or deceleration state of the target, identify changes in the direction of motion, and generate a future motion trajectory that conforms to the actual situation.

[0116] When generating future motion trajectories, the system also incorporates current external environmental data. For example, targets in aviation scenes are often affected by factors such as airflow and wind speed. The system uses sensors or external data sources to obtain real-time environmental information and inputs this data into the prediction model. By analyzing the impact of environmental factors on the target, the system can dynamically adjust the motion prediction results. For example, when the wind speed increases, the system can determine that its flight path may deflect based on the target's historical motion trajectory, and thus make corresponding adjustments in future trajectory predictions. This prediction method that combines environmental change factors greatly improves the robustness of the system in complex environments and ensures the continuity and accuracy of target tracking.

[0117] Multi-frame historical data plays a key role in this process. The system accumulates and analyzes the motion information in multiple frames, captures the changing patterns of the target's speed, direction, etc., and makes recursive predictions based on this. Regression analysis or recursive neural networks (such as LSTM) are often used to process this kind of time series data, which can effectively capture the long-term change trend of the target and avoid the error accumulation problem in single-frame prediction. The prediction result of each frame will affect the calculation of the next frame, thus forming a continuous and dynamic prediction model to ensure that the system can continuously track the target.

[0118] After generating the future motion trajectory, the system will correct the current position of the target based on this trajectory and generate the corrected target position data. This correction process is particularly important, especially when the target is partially occluded or lost. By correcting the future position, the system can compensate for the tracking error caused by insufficient visual information in the current frame and ensure the stability of the tracking results. For example, when a drone enters an occluded area during flight, the system can correct the position judgment in the current frame based on its historical trajectory and predicted future position to ensure that the target is not lost or inaccurately positioned. Through this process, the system can effectively deal with complex situations such as rapid target movement, direction changes, or external environmental interference.

[0119] The corrected target position data is cross-validated with the video modal feature data, the infrared modal feature data and the radar modal feature data to generate final target position data and target category data, and the final target position data and target category data are output.

[0120] Preferably, Figure 3As shown, the corrected target position data is cross-validated with the video modal feature data, the infrared modal feature data and the radar modal feature data, specifically including: by comparing the corrected target position data with the video modal feature data, the infrared modal feature data and the radar modal feature data, the position deviation and shape similarity of the target in different modalities are calculated to ensure that the position information of the target in each modality is consistent, and the cross-validation step includes position correction and shape consistency correction to generate modal similarity data.

[0121] The core of cross-validation technology is to compare the corrected target position data with each modal feature data (including video modal feature data, infrared modal feature data and radar modal feature data) to calculate the position deviation and shape similarity of the target in each modality. The results of these comparisons are used to confirm whether the representation of the target in different sensors is consistent, especially whether the core information such as position and shape matches. Through this inter-modal verification, recognition errors caused by single modal data errors or noise interference can be discovered and corrected, thereby generating more accurate final target position data and target category data.

[0122] Corrected target position data is generated by the system based on future motion prediction technology, that is, the system has made preliminary corrections to the target position through the prediction algorithm. This corrected position data is the estimated value that the system currently believes is most consistent with the actual position of the target. However, in order to ensure the accuracy of this position data, the system also needs to further improve the accuracy of the position data by cross-validating with multimodal data.

[0123] Position deviation calculation is the first step of cross-validation. The system compares the corrected target position data with the target position information extracted under each modality and calculates the spatial difference between them. In specific operations, the system extracts the center point coordinates of the target from the video modality feature data, infrared modality feature data, and radar modality feature data, which are expressed as (Video modal position), (infrared modal position) and (Radar mode position). These position information and the corrected target position By comparing and calculating the Euclidean distance or other spatial distance indicators between each mode, the deviation between the corrected target position and the target positions of different modes is obtained.

[0124] ;

[0125] in, , , These are the differences between the corrected target position data and the position information in the video, infrared, and radar modes. Based on these deviation values, the system will evaluate whether the corrected target position is consistent with the position information in the different modal data.

[0126] In addition to position deviation, the system also calculates shape similarity. Shape similarity ensures the consistency of the target in multiple modes by comparing the shape features of the target in different modes. For example, the video mode can capture the external contour and structural features of the target, the infrared mode reflects the distribution of the target's heat signal, and the radar mode provides the distance and reflection characteristics of the target. The system compares the shape features in these modes and calculates their similarity to ensure that the targets in different modes are consistent in shape.

[0127] The system can use cosine similarity or overlap coefficient to measure the shape consistency of the target in different modalities. For example, suppose , and They are the target shape features in video, infrared and radar modes respectively. The system calculates the shape similarity using the following formula:

[0128] ;

[0129] High shape similarity indicates that the shape features of the target in each modality are highly consistent, which enhances the system's confidence in target recognition.

[0130] After the position deviation and shape similarity are calculated, the system will correct the target's position information based on these indicators. Position correction is mainly to correct the position deviation under each mode to ensure that the final target position can be unified in the spatial position under different modes. Shape consistency correction is used to ensure that the shape features of the target under different modes match consistently, especially when the modal data is disturbed by the environment. Shape correction can avoid recognition errors caused by distortion of a certain mode.

[0131] After dual correction of position and shape, the system finally outputs the target position data after multi-modal verification, that is, the final target position data, which represents the most accurate target position that the system believes at the current moment. In addition, the system also generates target category data based on multi-modal feature data. The target category data is obtained by fusing and identifying target features under different modalities to ensure that the identified target category is consistent and accurate.

[0132] Through cross-validation technology, the system can effectively improve the accuracy of target recognition and positioning, especially in complex aviation environments, where data sources of different modalities provide multi-dimensional feature information. Through the verification and correction of position and shape, the system can ensure that the position information of the target remains consistent under multi-modality, avoiding recognition errors caused by single modality errors. In addition, the application of cross-validation can also enhance the system's ability to distinguish different target categories. For example, when the shape of the target captured by the video modality is inconsistent with the infrared modality, the system can identify the source of the inconsistency through shape similarity correction, and make corrections to avoid misidentification.

[0133] In one embodiment, in a complex aerial tracking mission, the system is tracking multiple drones that are flying at different altitudes and directions. Due to airflow disturbances, the infrared signature of one of the drones is disturbed by the heat source at a certain moment and deviates. Through cross-validation, the system detects that the position information under the infrared mode does not match that under the video and radar modes, and the calculated shape similarity is also low. The system then adjusts the weight of the infrared mode, corrects the position of the drone based on the position information of the video and radar data, and finally generates more accurate target position data to ensure that the tracking of the drone is not affected by interference factors. Through this process, the system can stably and accurately output the final position and category data of each drone.

[0134] Preferably, the shape similarity of the target in different modes is calculated using the cosine similarity formula. It represents the similarity between the corrected target position data and the modal feature data, and the calculation formula is:

[0135] ;

[0136] in, is the modal similarity; A vector representation of the corrected target position data; is the vector representation of modal characteristic data; Represents vector dot product operation; Represents the modulus of the vector; the cosine similarity is used to ensure that the shape information of the target in different modes remains consistent. Used to standardize the lengths of two vectors to eliminate the effect of length on similarity calculation.

[0137] The system converts the corrected target position data and each modal feature data into vector form. Through this vectorization, the system can compare mathematically. For each frame of video, the system generates a vector representation of the corrected target position, which contains the specific position information of the target in the image (such as coordinates), while the modal feature data vector represents the characteristic performance of the target in different modalities. This representation method can intuitively capture the shape and position information of the target in each modality, making the subsequent similarity calculation more accurate.

[0138] The system calculates the cosine similarity between the corrected target position vector and the modal feature vector, and determines the degree of similarity by evaluating the angle between the two vectors. The smaller the angle, the closer the cosine similarity value is to 1, indicating that the two vectors are more consistent in direction in space, which in turn reflects the consistency of the shape and position features of the target in different modes. On the contrary, if the similarity value is lower than a certain threshold, the system will identify that there may be recognition errors or data interference in a certain mode of the target, thereby triggering the subsequent correction mechanism.

[0139] Through cosine similarity calculation, the system can effectively ensure the consistency of feature information of the target in different modalities, which is crucial for target recognition in complex scenes. For example, in an aerial surveillance mission, if the target position captured by the video modality deviates significantly from the infrared modality or radar modality, the system will identify this inconsistency through modal similarity calculation, thereby avoiding target positioning failure caused by modal data errors.

[0140] Specifically, in a typical application, assume that the system is processing multi-modal data from a drone simultaneously. In the video modality, the corrected drone position vector may be , and the infrared modal characteristic data vector is By calculating the cosine similarity, the system can derive a similarity score between the two. If the score is low, it means that the shape information of the drone in the infrared mode may be distorted or interfered. The system will adjust the weight of the infrared mode accordingly and recalculate the target position to ensure that the final output data accurately reflects the true position of the drone.

[0141] In addition, modal similarity calculation can also enhance the system's adaptability to complex scenarios. For example, in a group of aircraft, the target may have inconsistent features due to interference, occlusion or rapid changes. The application of cosine similarity allows the system to quickly identify and correct these inconsistencies, thereby effectively improving the recognition ability and tracking accuracy in a multi-target environment.

[0142] In actual implementation, the system will periodically perform modal similarity calculations, including vectorizing the corrected target position data and the corresponding modal feature data for each frame, and then calculating the cosine similarity to generate modal similarity results. This process will continue to ensure that in a dynamically changing environment, the system can evaluate the matching of the target position and modal data in real time, so as to quickly respond to environmental changes and make necessary target position corrections.

[0143] like Figure 4 As shown, a system for implementing the aerial video stream target recognition processing method comprises:

[0144] The preprocessing module is used to receive and process video data, infrared data and radar data, and convert the video data, infrared data and radar data into video modal feature data, infrared modal feature data and radar modal feature data respectively; this module is mainly responsible for receiving video, infrared and radar data and performing modal conversion on these data. Each data source represents different modal information: video data captures the visible light image of the target, infrared data reflects the temperature distribution, and radar data is used to capture distance and speed information. Through preprocessing, the system converts the raw data into their respective modal feature data (video modal features, infrared modal features, radar modal features). Preprocessing includes noise removal, normalization processing, etc., so that the data can adapt to the subsequent processing flow.

[0145] The weighted fusion module is used to generate fused modal feature data based on the video modal feature data, infrared modal feature data and radar modal feature data using adaptive weighted fusion technology; in this module, the system uses adaptive weighted fusion technology to fuse the feature data of each modality into a comprehensive feature representation. By calculating the confidence of each modal feature data, the weighting factor of each modality is determined to generate more accurate fused modal feature data. The weighting factor is determined based on the signal strength, clarity and target recognizability of each modality to ensure that the contribution ratio of each modal data meets the weight requirements in actual applications.

[0146] The visual feature extraction and optimization module is used to perform low-level, medium-level and high-level multi-stage visual feature extraction based on the fused modal feature data, and globally enhance and optimize the extracted visual features through global convolution optimization technology to generate globally optimized visual feature data; the fused modal feature data is used as input, and the system gradually extracts from low-level (such as target edges), medium-level (such as shape and texture) to high-level (target overall contour and complex features) through multi-stage visual feature extraction technology. During the extraction process, global convolution optimization technology is used to optimize these visual features for global consistency to ensure that the visual features can reflect the global characteristics of the target in terms of spatial distribution. This process improves the system's ability to capture details of the target and enhances the expression effect of the features.

[0147] The motion prediction and correction module is used to build a model based on the global optimized visual feature data and the historical motion trajectory of the target, predict the future motion trajectory of the target in combination with future motion prediction technology, and correct the current position of the target according to the target motion prediction data to generate corrected target position data; based on the globally optimized visual feature data, the system builds a model in combination with the historical motion trajectory of the target, and uses future motion prediction technology to infer the future motion trajectory of the target. Motion prediction generates the future motion trend of the target by analyzing the historical position and speed changes of the target, and corrects the current position information of the target based on this trend to ensure that the system can still accurately track the target when it moves quickly. Finally, the module outputs the corrected target position data to reduce the positioning error caused by dynamic changes.

[0148] The cross-validation module is used to cross-validate the corrected target position data with the video modal feature data, infrared modal feature data, and radar modal feature data to generate the final target position data and target category data; this module ensures the consistency of multi-modal data by cross-validating the corrected target position data with the video modal features, infrared modal features, and radar modal features. The system discovers and corrects the recognition deviations under different modalities by comparing the target position information and shape similarity under each modality. This module generates the final target position data and target category data, providing the system with high-precision recognition results.

[0149] The output module is used to output the final target position data and target category data for system control or visual display. The final generated target position and target category data are output by the output module for system control or visual display. Through this module, the system can display the position and category information of the target in real time to support decision-making and control operations in aviation missions.

[0150] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for object recognition and processing of an aerial video stream, characterized in that: The following steps are involved: By preprocessing the video data, infrared data and radar data, each modal data is converted into corresponding modal feature data, and then fused through adaptive weighted fusion technology to generate fused modal feature data; Based on the fusion modality feature data, low-level, medium-level and high-level multi-stage visual feature extraction is performed, and the visual features are globally enhanced and optimized through the global convolution optimization technology to generate globally optimized visual feature data; By modeling the global optimized visual feature data and the historical motion trajectory of the target, the future motion trajectory of the target is predicted by combining the future motion prediction technology, and the current position of the target is corrected according to the target motion prediction data to generate the corrected target position data; Cross-validating the corrected target position data with the video modality feature data, the infrared modality feature data, and the radar modality feature data to generate final target position data and target category data, and outputting the final target position data and target category data; Based on the fusion modality feature data, low-level, mid-level and high-level multi-stage visual feature extraction is performed, including: Extracting low-level visual features through large-size convolution kernels, wherein the extracted low-level visual features include the edges of objects and basic features of simple geometric shapes; Extracting mid-level visual features through a medium-sized convolution kernel, wherein the extracted mid-level visual features include shape, texture, and detail information of the object; The high-level visual features are extracted by using a small-size convolution kernel. The extracted high-level visual features include the overall outline and complex shape of the target, ensuring that different visual features of the target are accurately captured at each level. The global convolution optimization technology uniformly optimizes low-level visual feature data, mid-level visual feature data and high-level visual feature data through a global convolution network. The global convolution network ensures the consistency of visual features at each level in the global range by analyzing the overall spatial distribution of the target. The global convolution network covers the entire input feature space through the convolution kernel, captures the long-distance association in the scene, and generates globally optimized visual feature data. The historical motion trajectory modeling of the target is based on the historical motion data. It is modeled by analyzing the dynamic characteristics of the target's position information, speed and motion direction in several past frames. The historical motion data includes each position, timestamp and motion vector of the target. By performing regression analysis on the historical motion data, the future motion trend of the target is predicted and the target motion trajectory feature data is generated.

2. The method for processing target recognition in an aerial video stream according to claim 1, characterized in that: Preprocess the video data, infrared data and radar data, including: Removing noise from the video data by filtering algorithms to make the edge and detail information of the video data clearer; Normalizing the pixel values ​​of the infrared data to enhance the contrast between different temperature regions and ensure that the infrared data can accurately reflect the temperature distribution; The radar data is processed by a distance normalization algorithm to standardize the reflected signal strength of the radar data and enhance the resolution of the distance information.

3. The method for object recognition and processing of aerial video stream according to claim 1, characterized in that: The adaptive weighted fusion technology determines the weighting factor by calculating the confidence of each modal feature data. The confidence is calculated according to the signal strength and clarity of each modal data and the identifiability of the target in different modalities. The weighting factor is calculated as follows: Using weighting factors , and They represent the weights of video modal feature data, infrared modal feature data, and radar modal feature data, respectively, and their calculation formulas are as follows: ； in, is the weighting factor of the video modality feature data; is the weighting factor of infrared modal characteristic data; is the weighting factor of radar modal characteristic data; Indicates the confidence of the video modality feature data; Indicates the confidence level of infrared modal feature data; Indicates the confidence level of radar modal characteristic data.

4. The method for object recognition and processing of aerial video stream according to claim 1, characterized in that: The future motion prediction technology performs spatiotemporal inference on the target's motion trajectory feature data. The spatiotemporal inference is based on the target's historical motion pattern and environmental change factors, combined with multiple frames of historical data to analyze the target's speed changes and direction adjustments, generate the target's future motion trajectory, and correct the current target position according to the target's future motion trajectory to generate corrected target position data.

5. The method for object recognition and processing of aerial video stream according to claim 1, characterized in that: Cross-validation of the corrected target position data with the video modal feature data, infrared modal feature data, and radar modal feature data includes: By comparing the corrected target position data with the video modal feature data, infrared modal feature data and radar modal feature data, the position deviation and shape similarity of the target in different modalities are calculated to ensure that the position information of the target in each modality is consistent. Cross-validation includes position correction and shape consistency correction to generate modal similarity data.

6. The method for processing target recognition in an aerial video stream according to claim 5, characterized in that: The shape similarity of the target in different modes is calculated using the cosine similarity formula. It represents the similarity between the corrected target position data and the modal feature data, and the calculation formula is: ； in, is the modal similarity; A vector representation of the corrected target position data; is the vector representation of modal characteristic data; Represents vector dot product operation; Represents the modulus of the vector; the cosine similarity is used to ensure that the shape information of the target in different modes remains consistent.

7. A system for implementing the method for object recognition and processing of aerial video streams according to any one of claims 1 to 6, characterized in that: include: A preprocessing module, used for receiving and processing the video data, infrared data and radar data, and converting the video data, infrared data and radar data into video modal feature data, infrared modal feature data and radar modal feature data respectively; A weighted fusion module, used to generate fused modal feature data based on the video modal feature data, the infrared modal feature data and the radar modal feature data by using an adaptive weighted fusion technology; A visual feature extraction and optimization module, which is used to perform low-level, mid-level and high-level multi-stage visual feature extraction based on the fused modality feature data, and globally enhance and optimize the extracted visual features through global convolution optimization technology to generate globally optimized visual feature data; A motion prediction and correction module, which is used to model the target based on the global optimized visual feature data and the historical motion trajectory of the target, predict the future motion trajectory of the target in combination with the future motion prediction technology, and correct the current position of the target according to the target motion prediction data to generate corrected target position data; A cross-validation module, used to cross-validate the corrected target position data with the video modality feature data, infrared modality feature data and radar modality feature data to generate final target position data and target category data; The output module is used to output the final target position data and target category data for system control or visual display.

Images