Unmanned aerial vehicle global perception method and system based on four-dimensional imaging and deep learning

Abstract

The application provides a kind of unmanned aerial vehicle global perception method and system based on four-dimensional imaging and deep learning, it is related to unmanned aerial vehicle perception technical field, first, the original detection signal of target area is acquired by four-dimensional imaging millimeter wave radar carried by unmanned aerial vehicle, four-dimensional imaging enhancement data is generated after multimodal signal complementary enhancement processing;Then the four-dimensional imaging enhancement data is input into the deep learning dynamic semantic modeling network, and a global semantic feature set is generated;Then the real-time flight state data of unmanned aerial vehicle is acquired, and the global semantic feature set is processed by space-time bidirectional association fusion, and space-time bidirectional association data is generated;Based on the space-time bidirectional association data, a dynamic environment semantic feature map is constructed;Finally, the global scene situation deduction and the dynamic division of safety boundary are carried out through the dynamic environment semantic feature map, and the global perception decision result covering the dynamic and static target distribution of target area, motion evolution trend and unmanned aerial vehicle safe flight airspace is output, to improve the global perception ability of unmanned aerial vehicle.

Description

Technical Field

[0001] This invention relates to the field of UAV perception technology, and more specifically, to a method and system for UAV global perception based on four-dimensional imaging and deep learning. Background Technology

[0002] In the field of drone applications, all-domain perception capability is a key factor in ensuring the safe and efficient execution of drone missions. Traditional drone perception methods mainly rely on single types of sensors, such as optical cameras and lidar. While optical cameras can provide rich visual information, their image quality is severely affected under complex environmental conditions, such as low light, fog, rain, and snow, leading to a significant decrease in the accuracy of target recognition and perception. Although lidar can obtain relatively accurate distance information, its detection performance is poor when facing transparent objects or objects with extremely low reflectivity, and its high cost limits its application in cost-sensitive scenarios.

[0003] Furthermore, most existing UAV perception methods based on multi-sensor fusion simply stitch or overlay data from different sensors without fully considering the inherent correlations and complementarities between different sensor data, making it difficult to achieve comprehensive and accurate all-domain perception. Moreover, when dealing with dynamic targets, traditional methods often lack in-depth analysis and understanding of the target's dynamic semantics, making it impossible to effectively predict the target's movement trend and failing to meet the needs of UAVs for safe flight in complex dynamic environments. Summary of the Invention

[0004] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide a method for all-domain perception of unmanned aerial vehicles based on four-dimensional imaging and deep learning, the method comprising: The original detection signal of the target area is obtained by the four-dimensional imaging millimeter-wave radar carried by the UAV, and the four-dimensional imaging enhanced data containing distance dimension information, azimuth dimension information, height dimension information, and velocity dimension information is generated through multi-modal signal complementary enhancement processing. The four-dimensional imaging enhancement data is input into a deep learning dynamic semantic modeling network. Through feature evolution analysis and dynamic target trajectory association processing, a global semantic feature set containing dynamic target semantic evolution features and static target semantic fixed features is generated. Acquire real-time flight status data of the UAV, and perform spatiotemporal bidirectional correlation and fusion processing on the real-time flight status data and the global semantic feature set to generate spatiotemporal bidirectional correlation data; Based on the aforementioned spatiotemporal bidirectional correlation data, a dynamic environmental semantic feature map is constructed, which includes the target's dynamic evolution relationship, the spatial semantic correlation relationship, and the UAV's own position correlation relationship. The dynamic environment semantic feature map is used to perform full-domain scene situation inference and dynamic division of safety boundaries, and output full-domain perception decision results covering the distribution of dynamic and static targets in the target area, motion evolution trend, and safe flight airspace of UAVs.

[0005] Furthermore, embodiments of the present invention also provide a UAV global perception system based on four-dimensional imaging and deep learning, characterized in that it includes: A processor; a machine-readable storage medium for storing machine-executable instructions of the processor; wherein the processor is configured to execute the above-described UAV global perception method based on four-dimensional imaging and deep learning by executing the machine-executable instructions.

[0006] In another aspect, embodiments of the present invention also provide a computer program product, the computer program product including machine-executable instructions, the machine-executable instructions being stored in a computer-readable storage medium, the processor of the UAV global perception system based on four-dimensional imaging and deep learning reading the machine-executable instructions from the computer-readable storage medium, the processor executing the machine-executable instructions, causing the UAV global perception system based on four-dimensional imaging and deep learning to execute the aforementioned UAV global perception method based on four-dimensional imaging and deep learning.

[0007] Based on the above, the original detection signals of the target area are acquired by the four-dimensional imaging millimeter-wave radar carried by the UAV, and enhanced data containing four-dimensional information including distance, azimuth, altitude, and velocity is generated through multi-modal signal complementarity enhancement processing. This fully utilizes the advantage of millimeter-wave radar to work stably in complex environments. At the same time, multi-modal signal complementarity enhances the richness and accuracy of the data. The four-dimensional imaging enhanced data is input into a deep learning dynamic semantic modeling network to generate a global semantic feature set containing dynamic target semantic evolution features and static target semantic fixed features. Real-time flight status data of the UAV is acquired and fused with the global semantic feature set through spatiotemporal bidirectional correlation to generate spatiotemporal bidirectional correlation data. This fully considers the spatiotemporal relationship between the UAV's own state and target features. Based on the spatiotemporal bidirectional correlation data, a dynamic environmental semantic feature map is constructed, which can comprehensively and dynamically reflect the complex environmental information of the target area. Finally, the dynamic environmental semantic feature map is used to perform global scene situation inference and dynamic division of safety boundaries, and outputs global perception decision results covering the distribution of dynamic and static targets in the target area, motion evolution trends, and safe flight airspace of the UAV, which significantly improves the global perception capability and safety performance of the UAV. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of the execution flow of the UAV global perception method based on four-dimensional imaging and deep learning provided in the embodiments of the present invention.

[0009] Figure 2 This is a schematic diagram of exemplary hardware and software components of an unmanned aerial vehicle (UAV) global perception system based on four-dimensional imaging and deep learning, provided in an embodiment of the present invention. Detailed Implementation

[0010] Figure 1 This is a flowchart illustrating a method for UAV global perception based on four-dimensional imaging and deep learning, provided in one embodiment of the present invention. A detailed description follows.

[0011] Step S110: Obtain the original detection signal of the target area through the four-dimensional imaging millimeter-wave radar carried by the UAV, and generate four-dimensional imaging enhancement data containing distance dimension information, azimuth dimension information, height dimension information, and velocity dimension information through multi-modal signal complementary enhancement processing.

[0012] This embodiment uses a drone patrol monitoring scenario in an urban area as an example. In this scenario, the drone needs to comprehensively perceive targets such as roads, buildings, pedestrians, and vehicles in the city. The four-dimensional imaging millimeter-wave radar carried by the drone can transmit and receive millimeter-wave signals, and obtain multi-dimensional information about the targets by processing these signals. Due to the complexity of the urban environment and the presence of various interferences and reflections, multi-modal signal complementary enhancement processing is required to improve the quality and accuracy of the data, ultimately generating four-dimensional imaging enhanced data containing information in four dimensions: distance, azimuth, altitude, and velocity.

[0013] Step S111: Control the UAV to cruise the target area along a full-coverage flight path, and simultaneously activate the onboard four-dimensional imaging millimeter-wave radar to enter the multi-band detection mode and transmit multi-band millimeter-wave detection signals to the target area.

[0014] In urban patrol and monitoring scenarios, the planning of full-coverage flight paths needs to consider the city's layout and monitoring requirements. Typically, a flight path is designed to cover the entire target area based on its size and shape, such as a grid or spiral pattern. The drone flies along the pre-set path, ensuring a comprehensive scan of the target area. Simultaneously, a four-dimensional imaging millimeter-wave radar is activated, entering multi-band detection mode. Multi-band detection mode means the radar emits millimeter-wave signals of different frequencies. Different frequencies have different characteristics in terms of penetration and resolution, adapting to the detection needs of different types of targets in the city. For example, lower-frequency signals may have stronger penetration, suitable for detecting obscured targets; higher-frequency signals may have higher resolution, enabling clearer distinction of target details.

[0015] Step S112: Receive multi-band echo signals reflected by various objects within the target area, and integrate all echo signals to form a raw detection signal set, which contains complete echo data of different frequency bands and different reflection angles.

[0016] When a drone flies along a planned path and transmits multi-band millimeter-wave signals, these signals encounter various objects within the target area, such as buildings, vehicles, and pedestrians, and are reflected back. The radar system receives these reflected echo signals. Due to the diversity and complexity of the targets, the echo signals also possess different characteristics, including different frequency bands and different reflection angles. All received echo signals are collected and organized, classified and stored according to characteristics such as frequency band and reflection angle, forming a raw detection signal set. This raw detection signal set contains rich echo data from the target area.

[0017] Step S113: Perform multi-band signal noise reduction processing on the original detection signal set, and separate the environmental interference component and effective reflection component in the signal through adaptive signal filtering technology, while retaining the effective signal carrying target information.

[0018] The original detection signal set inevitably contains various environmental interferences, such as electromagnetic noise and ground clutter, which can affect the accurate extraction of target signals. Adaptive signal filtering technology can automatically adjust the filter parameters based on the statistical characteristics of the signal, thereby effectively separating interference components from effective reflection components. First, the signals in each frequency band of the original detection signal set are analyzed to determine the characteristics and distribution of noise. Then, based on these analysis results, an adaptive filter is designed. During signal processing, the filter can monitor signal changes in real time, and automatically adjust the filter parameters to suppress interference when it is detected. Through this method, environmental interference can be effectively removed, the effective signal carrying target information can be retained, and the signal-to-noise ratio can be improved.

[0019] Step S114: Perform multi-view signal complementary fusion on the noise-reduced effective signal, and superimpose the effective signals of different frequency bands and different reflection angles according to the spatial position correspondence to generate the fused effective signal.

[0020] Effective signals from different frequency bands and reflection angles reflect different aspects of a target's characteristics. The purpose of multi-view signal complementary fusion is to integrate these signals from different perspectives to obtain more comprehensive and accurate target information. First, it is necessary to determine the spatial correspondence between effective signals from different frequency bands and reflection angles. This requires aligning the spatial positions of different signals using coordinate transformation and registration techniques. Then, the signals at corresponding positions are superimposed. During superposition, different weights can be assigned based on the reliability and importance of signals from different frequency bands. For example, signals from certain frequency bands that perform better in detecting specific types of targets can be given higher weights. Through this method, the fused effective signal can comprehensively utilize the advantages of signals from different frequency bands and reflection angles, improving the accuracy and reliability of target detection.

[0021] Step S115: Extract distance-related signal components from the fused effective signal, calculate the relative distance parameters between each reflection point and the UAV through signal propagation delay analysis, and generate distance dimension information.

[0022] The fused effective signal contains information related to the target distance, namely the signal propagation delay. The propagation delay is the time it takes for the signal to travel from the UAV's emission point to its reflection from the target and back to the UAV. Since the speed of electromagnetic waves in air is constant, the distance between the target and the UAV can be calculated by measuring the propagation delay. For each reflection point, the signal emission and reception times are recorded, and the difference between the two is the propagation delay. Then, based on the relationship that distance equals the propagation speed multiplied by half the propagation delay (because the signal travels a round trip), the relative distance parameter between each reflection point and the UAV is calculated. The relative distance parameters of all reflection points are then organized according to their spatial locations to generate distance dimension information.

[0023] Step S116: Extract the azimuth-related signal components from the fused effective signal, determine the projection direction parameters of each reflection point on the horizontal plane based on the phase difference analysis of the radar array antenna, and generate azimuth dimension information.

[0024] Radar typically employs an array antenna, composed of multiple antenna elements. When the signal reflected from a target reaches different antenna elements, a phase difference arises due to the varying path lengths. By analyzing these phase differences, the azimuth of the target on the horizontal plane can be determined. The azimuth-related signal component is extracted from the fused effective signal, and the phase of this signal component received by different elements in the radar array antenna is compared. Based on the array antenna's geometry and phase difference information, relevant algorithms (such as beamforming algorithms) are used to calculate the projection direction parameter (azimuth angle) of each reflection point on the horizontal plane. Finally, the azimuth angle parameters of all reflection points are integrated to generate azimuth dimension information.

[0025] Step S117: Extract the height-related signal components from the fused effective signal, combine the real-time flight altitude data of the UAV with the vertical angle offset information of the signal, calculate the actual altitude parameters of each reflection point, and generate altitude dimension information.

[0026] The fused effective signal also includes components related to the target altitude. The UAV's real-time flight altitude data can be obtained through its onboard GPS or barometric altimeter. The signal vertical angle offset information refers to the angle between the signal in the vertical direction and the UAV's horizontal line, which can be measured using radar pitch angle. For each reflection point, its actual altitude is calculated based on its relative distance parameters, the signal vertical angle offset information, and the UAV's real-time flight altitude data. For example, if the UAV's flight altitude is H, the relative distance to the reflection point is D, and the signal vertical angle offset is θ, then the altitude of the reflection point can be calculated by subtracting D from H and multiplying by sinθ (assuming θ is the pitch angle). After processing the actual altitude parameters of all reflection points, altitude dimension information is generated.

[0027] Step S118: Extract the velocity-related signal components from the fused effective signal, calculate the motion velocity parameters of each reflection point relative to the UAV through Doppler frequency offset analysis, and generate velocity dimension information.

[0028] When there is relative motion between the target and the drone, the frequency of the reflected signal changes, resulting in Doppler frequency shift. The magnitude of the Doppler frequency shift is related to the target's velocity relative to the drone. Velocity-related signal components are extracted from the fused effective signal, and spectral analysis is performed to determine the numerical value of the Doppler frequency shift. Based on the relationship between the Doppler frequency shift and relative velocity, the velocity parameters of each reflection point relative to the drone are calculated, including the magnitude and direction of the velocity. For example, a positive Doppler frequency shift indicates that the target is approaching the drone; a negative value indicates that the target is moving away from the drone. Integrating these velocity parameters generates velocity dimension information.

[0029] Step S119: Perform dynamic compensation processing on the distance dimension information, orientation dimension information, height dimension information, and velocity dimension information. Then, associate the compensated distance dimension information, orientation dimension information, height dimension information, and velocity dimension information one-to-one with spatial coordinates and time nodes to generate four-dimensional imaging enhancement data with spatial and temporal correspondence.

[0030] During flight, the drone's own motion (such as flight speed and attitude changes) affects information in dimensions such as distance, azimuth, altitude, and velocity. Dynamic compensation processing aims to eliminate these effects and improve data accuracy. For example, the drone's flight speed can cause errors in distance measurement, requiring correction of the distance dimension information based on the drone's speed and flight direction; attitude changes (such as pitch and roll) affect azimuth and altitude measurements, requiring calibration using attitude sensor data. After dynamic compensation, the four dimensions of information are correlated according to their corresponding spatial coordinates (distance, azimuth, and altitude) and time nodes. Each time node corresponds to a set of spatial coordinates and velocity information, thereby generating four-dimensional imaging augmentation data, which can dynamically reflect the state of the target within the target area at different times and spatial locations.

[0031] Step S120: Input the four-dimensional imaging enhancement data into a deep learning dynamic semantic modeling network, and generate a global semantic feature set containing dynamic target semantic evolution features and static target semantic fixed features through feature evolution analysis and dynamic target trajectory association processing.

[0032] In urban patrol and monitoring scenarios, four-dimensional imaging augmented data contains a wealth of information about targets. Deep learning dynamic semantic modeling networks can perform in-depth analysis of this data to extract the semantic features of the targets. Through feature evolution analysis, the changes in target features over time can be tracked, distinguishing between dynamic and static targets. Dynamic target trajectory association processing can link the features of the same dynamic target at different points in time, forming a continuous trajectory. The final generated global semantic feature set contains both the semantic evolution features of dynamic targets and the fixed semantic features of static targets.

[0033] Step S121: Call the pre-trained deep learning dynamic semantic modeling network, which includes a multi-scale feature extraction layer, a dynamic feature evolution layer, and a semantic association layer, and is trained using multiple four-dimensional imaging sample data and semantic label data.

[0034] This deep learning dynamic semantic modeling network is pre-trained and specifically designed for processing four-dimensional imaging data in urban patrol monitoring scenarios. The network structure includes a multi-scale feature extraction layer, a dynamic feature evolution layer, and a semantic association layer. The multi-scale feature extraction layer extracts features at different scales from the input data to accommodate targets of varying sizes and distances. The dynamic feature evolution layer employs a temporal recurrent network structure, enabling it to analyze feature changes over time and capture the movement trends of dynamic targets. The semantic association layer associates the extracted features with a pre-defined semantic labeling system, assigning semantic meaning to the features. During training, a large amount of four-dimensional imaging sample data is used, with each sample labeled with a corresponding semantic label, such as "vehicle," "pedestrian," and "building." The network parameters are continuously adjusted using a backpropagation algorithm, enabling the network to accurately extract semantic features from the input data.

[0035] Step S122: Divide the four-dimensional imaging enhancement data into continuous time-series data segments according to the time sequence. Each time-series data segment contains complete four-dimensional imaging enhancement data within a preset time length.

[0036] Four-dimensional imaging augmentation data is acquired continuously over time. To facilitate temporal analysis by the network, it needs to be divided into continuous temporal data segments. The selection of the preset time length needs to consider the target's motion speed and the temporal resolution of the monitoring. For example, for vehicles and pedestrians in a city, the preset time length can be set to a period of time that can capture obvious changes in their motion. Each temporal data segment contains the distance, orientation, height, and velocity information of all targets within that time period. This information will be input as a whole into the deep learning network for processing.

[0037] Step S123: Input each time series data segment into the multi-scale feature extraction layer of the deep learning dynamic semantic modeling network, and extract the local detail features and global context features of each time series data segment through multi-layer convolution operations with different receptive fields.

[0038] The multi-scale feature extraction layer consists of multiple convolutional layers, each employing a different kernel size to capture features within different receptive fields. Smaller kernels extract local details of the target, such as vehicle shapes and pedestrian outlines; larger kernels capture global contextual features, such as relative positions between targets and road orientation. For each time-series data segment, the data is first processed through multiple convolutional layers, each typically followed by an activation function and a pooling layer. The activation function introduces non-linear transformations to enhance the network's expressive power; the pooling layer reduces the dimensionality of the feature maps, decreasing computational cost. After multiple convolutional operations, multiple feature maps at different scales are obtained. These feature maps are then fused using methods such as concatenation to form a feature representation that simultaneously contains local details and global context.

[0039] Step S124: Input local detail features and global context features into the dynamic feature evolution layer, perform evolution analysis on the features of continuous time-series data segments through a time-series recurrent network, track the change trajectory of features over time, and filter out dynamic feature groups that exhibit continuous changes and static feature groups that remain stable.

[0040] Local detail features and global contextual features are input into the dynamic feature evolution layer. This layer uses a temporal recurrent network (such as an LSTM network) to process the temporal data. The LSTM network has a memory function, capable of remembering feature information from previous temporal data segments and associating it with the features of the current temporal data segment. By processing the features of consecutive temporal data segments, the LSTM network can capture how features change over time. Features whose feature values ​​continuously change across multiple consecutive temporal data segments are categorized into dynamic feature groups, such as moving vehicles and walking pedestrians; while features whose feature values ​​remain largely unchanged are categorized into static feature groups, such as buildings and streetlights.

[0041] For example, step S1241: Organize the local detail features and global context features from the multi-scale feature extraction layer according to the order of the time-series data segments to generate a feature sequence with time-series labels.

[0042] The local detail features and global context features output by the multi-scale feature extraction layer are specific to each time-series data segment. These features are arranged sequentially according to the acquisition order of the time-series data segments, and a corresponding time-series marker, such as a timestamp or sequence number, is added to each feature. For example, the feature marker for the first time-series data segment is t1, the second is t2, and so on, forming a feature sequence with time-series markers. In this way, the network can clearly define the temporal order of each feature when processing it, facilitating time-series analysis.

[0043] Step S1242: Input the feature sequence with temporal markers into the temporal recurrent network unit in the dynamic feature evolution layer. The temporal recurrent network unit contains multiple neuron nodes with memory function.

[0044] The feature sequence with temporal labels is input into a temporal recurrent network unit. This unit consists of multiple neurons with memory capabilities, such as LSTM neurons. These neurons can perform calculations based on the input features and their own memory state, and output the corresponding results. LSTM neurons control the flow of information and memory updates through gating mechanisms (input gate, forget gate, output gate), thereby effectively processing long-term temporal data and avoiding gradient vanishing or gradient exploding problems.

[0045] Step S1243: The feature sequence with time-series labels is processed sequentially by the neuron nodes in the time-series recurrent network unit. In each processing step, the neuron node receives the local detail features and global context features of the current time-series data segment, and combines them with the memory state of the previous time-series data segment stored inside the neuron node to generate the output features of the current time-series data segment and the updated memory state.

[0046] The temporal recurrent network unit processes each feature in the feature sequence sequentially according to temporal order. When processing the features of the current temporal data segment, the input gate of the neuron node controls the inflow of the current feature, determining which information can enter the neuron for processing; the forget gate determines whether to retain the memory state of the previous temporal data segment, forgetting information that is no longer needed; the output gate generates output features based on the current input and memory state, and updates the memory state. For example, when processing the features of temporal data segment t2, the neuron node combines the memory state of temporal data segment t1 and the input features of t2, and after processing by the gating mechanism, generates the output features of t2 and a new memory state, which are used to process the features of the next temporal data segment t3.

[0047] Step S1244: Collect the output features of each time-series data segment generated by the time-series recurrent network unit after processing the complete feature sequence, and form a time-series output feature sequence.

[0048] After a time-series recurrent network unit processes the entire feature sequence with time-series labels, it generates an output feature for each time-series data segment. Collecting these output features in temporal order forms a time-series output feature sequence. This sequence reflects the feature changes at different time points after processing by the time-series recurrent network. Each output feature contains feature information of the corresponding time-series data segment and its association with previous time-series data segments.

[0049] Step S1245: Compare the output features of adjacent time-series data segments in the time-series output feature sequence, and calculate the difference measure between adjacent output features. The difference measure reflects the magnitude of change of the feature from the previous time-series data segment to the next time-series data segment.

[0050] To determine whether features have changed over time, it's necessary to calculate a measure of difference between the output features of adjacent time-series data segments. There are various methods for calculating this measure, such as Euclidean distance and cosine similarity. Taking Euclidean distance as an example, for two adjacent output feature vectors, the square root of the sum of the squares of their corresponding element differences is calculated. A larger Euclidean distance indicates a greater difference between the two features, meaning a larger magnitude of feature change; conversely, a smaller Euclidean distance indicates a smaller magnitude of feature change. By calculating this measure of difference, the degree of feature change can be quantified.

[0051] Step S1246: Based on the difference metric, classify all feature points in the time-series output feature sequence, classify feature points whose difference metric is consistently higher than a preset threshold as continuously changing feature points, and classify feature points whose difference metric is consistently lower than a preset threshold as stable feature points.

[0052] A preset difference measurement threshold is established, determined based on the target's motion characteristics and monitoring requirements in urban patrol monitoring scenarios. For each feature point in the time-series output feature sequence, it is determined whether its difference measurement with the previous feature point exceeds the threshold. If the difference measurement of multiple consecutive time-series data segments exceeds the threshold, the feature point is classified as a continuously changing feature point, indicating that the target corresponding to these features may be in motion. If the difference measurement of multiple consecutive time-series data segments is below the threshold, the feature point is classified as a stable feature point, indicating that the target corresponding to these features may be stationary.

[0053] Step S1247: Group all continuously changing feature points according to their positions in the time-series output feature sequence, aggregate feature points that belong to the same spatial location but continuously change at different time points into a feature group, assign a dynamic feature identifier to each feature group, and integrate all output features and their time-series labels corresponding to the feature group to generate a dynamic feature group.

[0054] Continuously changing feature points typically correspond to dynamic targets, such as moving vehicles or pedestrians. These feature points are grouped according to their spatial location; feature points of the same dynamic target at different time points are aggregated into the same group. For example, the same moving car will generate multiple continuously changing feature points in different time series data segments. These feature points have spatial continuity and are therefore grouped into the same group. Each group is assigned a unique dynamic feature identifier, such as ID1, ID2, etc., and all output features within that group are integrated with their corresponding time series labels to form a dynamic feature group.

[0055] Step S1248: Group all stable feature points according to their positions in the temporal output feature sequence, aggregate feature points that belong to the same spatial location and remain stable at different time points into a feature group, assign a static feature identifier to each feature group, and integrate all output features and their temporal labels corresponding to the feature group to generate a static feature group.

[0056] Stable feature points correspond to static targets, such as buildings, streetlights, and traffic signs. These stable feature points are also grouped according to their spatial location; feature points of the same static target at different time points will be aggregated into the same group. For example, feature points of a building that remain largely in the same spatial location across different time series data segments will be grouped together. Each group is assigned a static feature identifier, and the output features and time series markers within the group are integrated to form a static feature group.

[0057] Step S125: Perform dynamic target trajectory association processing on the dynamic feature group, match and associate the features of the same target in different time series data segments, generate a continuous feature evolution trajectory for each dynamic target, extract the changing parameters and feature attributes in the continuous feature evolution trajectory, and generate dynamic target semantic evolution features.

[0058] Although feature points in a dynamic feature group belong to the same dynamic target, their feature representation may differ in different time-series data segments due to factors such as occlusion during target movement and changes in radar detection angle. The purpose of dynamic target trajectory association processing is to accurately match these feature points to form a continuous trajectory. By analyzing the trajectory, the target's changing parameters (such as velocity and acceleration) and feature attributes (such as size and shape) are extracted, thereby generating dynamic target semantic evolution features that reflect the changes of the dynamic target over time.

[0059] For example, step S1251: Obtain all dynamic feature groups output by the dynamic feature evolution layer. Each dynamic feature group contains a dynamic feature identifier and the sequence information of the feature corresponding to the dynamic feature identifier changing over time.

[0060] The dynamic feature evolution layer outputs multiple dynamic feature groups. Each dynamic feature group has a unique dynamic feature identifier and corresponding feature sequence information. The feature sequence information records the feature changes of the dynamic feature group in different time-series data segments, including feature vectors and corresponding time-series markers.

[0061] Step S1252: Extract the spatial coordinate information and feature description vector corresponding to each time point from the sequence information of the feature changing over time. The spatial coordinate information comes from the distance dimension, orientation dimension, and height dimension information in the four-dimensional imaging enhancement data.

[0062] For each feature sequence of a dynamic feature group, it is necessary to extract the spatial coordinate information and feature description vector at each time point. The spatial coordinate information includes distance, orientation, and altitude, which can be obtained from four-dimensional imaging augmentation data and reflect the target's position in space. The feature description vector is a numerical representation of the target's features, containing information such as the target's shape, size, and reflectivity, and is obtained through multi-scale feature extraction layers and dynamic feature evolution layers.

[0063] Step S1253: For each dynamic feature group, according to its dynamic feature identifier, arrange the spatial coordinate information extracted from the dynamic feature group in different time series data segments in chronological order to form the initial spatial coordinate sequence corresponding to the dynamic feature identifier.

[0064] The spatial coordinate information of the same dynamic feature identifier in different time-series data segments is arranged sequentially according to time to form an initial spatial coordinate sequence. For example, the coordinates of time-series data segment t1 are (x1, y1, z1), the coordinates of t2 are (x2, y2, z2), and so on. In this way, the change of the target's position in space over time can be seen intuitively.

[0065] Step S1254: For each dynamic feature group, based on its dynamic feature identifier, arrange the feature description vectors extracted from the dynamic feature group in different time series data segments in chronological order to form the initial feature description vector sequence corresponding to the dynamic feature identifier.

[0066] Similarly, feature description vectors for the same dynamic feature identifier are arranged in chronological order to form an initial feature description vector sequence. This initial feature description vector sequence reflects how the target's feature description changes over time, and by analyzing this sequence, we can understand how the target's feature attributes change over time.

[0067] Step S1255: Based on the initial spatial coordinate sequence and the initial feature description vector sequence, calculate the similarity measure between the feature sequences corresponding to any two different dynamic feature identifiers. The similarity measure combines the proximity of the spatial coordinate sequences and the similarity of the feature description vector sequences.

[0068] To determine whether two dynamic feature identifiers belong to the same physical target, it is necessary to calculate a similarity metric between their corresponding feature sequences. This similarity metric calculation needs to consider both the proximity of the spatial coordinate sequences and the similarity of the feature description vector sequences. The proximity of the spatial coordinate sequences can be measured by calculating the average spatial distance at corresponding time points within the sequences; the similarity of the feature description vector sequences can be measured by methods such as cosine similarity. Combining these two metrics yields a comprehensive similarity metric, allowing for a more complete assessment of the similarity between the two feature sequences.

[0069] Step S1256: Based on the similarity metric, identify dynamic feature pairs whose similarity metric exceeds the connection threshold. The dynamic feature pairs indicate that two dynamic feature groups may belong to the same physical target in different time periods.

[0070] A connection threshold is set. When the similarity measure of two dynamic feature identifiers exceeds this threshold, their corresponding dynamic feature groups are considered to belong to the same physical target. For example, in urban traffic scenarios, the same car may be divided into two dynamic feature groups during its journey due to turning, occlusion, etc. The similarity measure can identify them as the same target.

[0071] Step S1257: For each identified dynamic feature pair, merge and align the initial spatial coordinate sequences corresponding to the two dynamic feature pairs in time order to generate a merged spatial coordinate sequence.

[0072] The initial spatial coordinate sequences of the two dynamic feature identifiers are merged in chronological order. If the two sequences overlap in time, alignment is required, for example, by using a weighted average method to process the coordinates of the overlapping time points to generate a continuous and smooth merged spatial coordinate sequence that more completely reflects the target's trajectory.

[0073] Step S1258: For each identified dynamic feature identifier pair, merge and align the initial feature description vector sequences corresponding to the two dynamic feature identifiers in time order to generate a merged feature description vector sequence.

[0074] Similarly, the initial feature description vector sequence is merged and aligned. For feature description vectors in the temporally overlapping regions, similar weighted averaging or other fusion methods can be used to generate a merged feature description vector sequence that reflects the continuous changes of the target features over time.

[0075] Step S1259: Bind the merged spatial coordinate sequence and the merged feature description vector sequence, and assign a new unified target identifier to generate a continuous feature evolution trajectory corresponding to the unified target identifier. The continuous feature evolution trajectory includes the complete spatial coordinate sequence and the complete feature description vector sequence of the corresponding target.

[0076] A new, unified target identifier is assigned to the merged spatial coordinate sequence and feature description vector sequence, binding them together to form a continuous feature evolution trajectory. This continuous feature evolution trajectory fully records the target's spatial location and feature information at different time points.

[0077] Step S12510: Calculate the displacement between adjacent time points from the spatial coordinate sequence of the continuous feature evolution trajectory, calculate the velocity change based on the displacement and time interval, and extract the displacement and velocity change as the variation parameters of the continuous feature evolution trajectory.

[0078] For a spatial coordinate sequence of a continuous feature evolution trajectory, the coordinate difference between adjacent time points is calculated to obtain the displacement. The time interval is the time difference between adjacent time series data segments. The velocity change is calculated by dividing the displacement by the time interval. The displacement and velocity change reflect the dynamic characteristics of the target's motion.

[0079] Step S12511: Extract the set of vector dimensions that can reflect the radar reflection characteristics of the target from the feature description vector sequence of the continuous feature evolution trajectory, and use it as the feature attribute of the continuous feature evolution trajectory.

[0080] Different dimensions in the feature description vector sequence represent different characteristics of the target. Dimensions related to radar reflection characteristics, such as reflection intensity and reflection area, are extracted from these dimensions. These dimensions reflect attributes such as the target's material and shape, and are used as feature attributes.

[0081] Step S12512: Combine and encapsulate the unified target identifier, the continuous feature evolution trajectory corresponding to the unified target identifier, the change parameters extracted from the continuous feature evolution trajectory, and the feature attributes extracted from the continuous feature evolution trajectory to generate dynamic target semantic evolution features.

[0082] By combining a unified target identifier, continuous feature evolution trajectory, changing parameters, and feature attributes, a dynamic target semantic evolution feature is formed. This dynamic target semantic evolution feature contains comprehensive information such as the target's identity, motion trajectory, changing parameters, and feature attributes.

[0083] Step S126: Perform spatial topological association processing on the static feature group, analyze the distribution relationship and association pattern of different static features in space, extract the fixed attributes and spatial distribution features in the features, and generate static target semantic fixed features.

[0084] Static feature groups correspond to static targets, such as buildings and streetlights. Spatial topology association processing aims to analyze the spatial relationships between these static targets, such as relative position, distance, and orientation. By analyzing the spatial coordinate information of static feature groups, the distribution relationships and association patterns between different static targets can be determined, such as which buildings are adjacent and the positional relationship between roads and buildings. Simultaneously, fixed attributes of static targets are extracted, such as the height and shape of buildings, the type of streetlights, and their spatial distribution characteristics, such as density and arrangement. Combining the above information generates fixed semantic features for static targets, which reflect the inherent attributes and spatial distribution of static targets.

[0085] Step S127: Input the dynamic target semantic evolution features and the static target semantic fixed features into the semantic association layer. Based on the preset semantic label system, match the corresponding semantic description information for each feature. The semantic description information represents the type attribute and state attribute of the target.

[0086] The semantic association layer pre-defines a semantic labeling system for urban patrol monitoring scenarios. This system includes type and status labels for various targets. Type labels include "vehicle," "pedestrian," "building," "road," and "streetlight," while status labels include "moving," "stationary," "congested," and "normal." Dynamic target semantic evolution features and static target semantic fixed features are matched with these labels. For example, changing parameters and attribute features in dynamic target semantic evolution features might match the "vehicle" type label and the "moving" status label; fixed attributes and spatial distribution features in static target semantic fixed features might match the "building" type label and the "normal" status label. Through matching, each feature is assigned corresponding semantic descriptive information.

[0087] Step S128: Bind the dynamic target semantic evolution features to their corresponding semantic description information to generate dynamic semantic units; bind the static target semantic fixed features to their corresponding semantic description information to generate static semantic units.

[0088] Dynamic semantic units are formed by combining the semantic evolution features of dynamic targets with their corresponding semantic descriptions. Each dynamic semantic unit contains the feature information and semantic description of the dynamic target, clearly representing its identity, state, and characteristics. Similarly, static semantic units are formed by combining the fixed semantic features of static targets with their corresponding semantic descriptions, representing the attributes and states of static targets.

[0089] Step S129: Perform feature deduplication on all dynamic and static semantic units, integrate all deduplicated dynamic and static semantic units, and generate a global semantic feature set containing dynamic target semantic evolution features and static target semantic fixed features.

[0090] During processing, duplicate semantic units may occur, such as the same target being detected multiple times or different dynamic feature groups being incorrectly identified as different targets when they are actually the same target. Feature deduplication is performed by comparing the similarity between semantic units and removing duplicate units. Then, the deduplicated dynamic and static semantic units are integrated to form a global semantic feature set. This global semantic feature set comprehensively reflects the semantic information of all dynamic and static targets within the target area.

[0091] Step S130: Obtain real-time flight status data of the UAV, and perform spatiotemporal bidirectional correlation fusion processing on the real-time flight status data and the global semantic feature set to generate spatiotemporal bidirectional correlation data.

[0092] In urban patrol monitoring, the flight status of drones affects their target detection and perception results. Real-time flight status data of drones, including flight attitude, position, and speed, is acquired and fused with a global semantic feature set. Spatiotemporal bidirectional correlation fusion processing considers the correlation between time and space, aligning and associating the drone's flight status with the target's semantic features in time and space to more accurately understand the relative relationship between the target and the drone, generating spatiotemporal bidirectional correlation data.

[0093] Step S131: Collect flight attitude data of the UAV using the inertial measurement equipment on board the UAV. The flight attitude data reflects the pitch, roll and yaw states of the UAV.

[0094] The inertial measurement unit (IMU) onboard a drone can measure its acceleration and angular velocity. By integrating and processing this measurement data, the drone's pitch, roll, and yaw angles can be obtained. These angular parameters reflect the drone's flight attitude. The pitch angle represents the degree to which the drone's nose tilts up and down, the roll angle represents the degree of rotation of the drone around its longitudinal axis, and the yaw angle represents the horizontal turning angle of the drone's nose. Flight attitude data is typically acquired at a high frequency to reflect changes in the drone's attitude in real time.

[0095] Step S132: Collect real-time coordinate data of the UAV using the satellite positioning equipment mounted on the UAV. The real-time coordinate data reflects the precise position of the UAV in three-dimensional space.

[0096] Satellite positioning equipment (such as GPS) can receive signals from multiple satellites and determine the three-dimensional coordinates of a drone in the Earth's coordinate system, namely longitude, latitude, and altitude, by calculating the signal propagation time. The accuracy of real-time coordinate data is affected by factors such as satellite signal quality and environmental obstruction, and may have some errors in urban environments, but it can still meet the location requirements for drone patrol and monitoring.

[0097] Step S133: Collect flight operation data of the UAV through the flight control module of the UAV. The flight operation data reflects the flight speed adjustment status, heading adjustment status and altitude control status of the UAV.

[0098] The flight control module is the core control unit of the UAV, recording the UAV's flight operation commands. Flight speed adjustment can be reflected by throttle control, heading adjustment can be reflected by rudder deflection, and altitude control can be reflected by elevator deflection or throttle changes. The above data can reflect the UAV's flight operation intentions and current flight status.

[0099] Step S134: Integrate the flight attitude data, real-time coordinate data, and flight operation data to generate real-time flight status data that includes the current flight status and operation intention of the UAV.

[0100] Flight attitude data, real-time coordinate data, and flight operation data are aligned and integrated according to timestamps. Each time point corresponds to a complete set of flight status parameters, including attitude angles, coordinates, speed adjustments, heading adjustments, altitude control parameters, etc. Through integration, real-time flight status data is generated, which comprehensively reflects the UAV's flight status and operational intentions at the current moment.

[0101] Step S135: Extract the time stamp information corresponding to each semantic unit in the global semantic feature set, and determine the collection time node of each semantic feature; extract the time stamp information corresponding to each component in the real-time flight status data, and determine the recording time node of each flight status parameter.

[0102] Each semantic unit in the global semantic feature set carries a timestamp indicating the time of acquisition. By extracting these timestamps, the acquisition time of each semantic feature can be determined. Similarly, each parameter in the real-time flight status data also carries a recording timestamp, allowing us to determine the recording time of each flight status parameter.

[0103] Step S136: Based on the time stamp information, the global semantic feature set and the real-time flight status data are time-axis calibrated so that the semantic features at the same time point correspond to the flight status data.

[0104] Using time as a benchmark, the global semantic feature set and real-time flight status data are aligned according to time nodes. A correspondence is established between semantic features and flight status data at the same time node. If there are minor differences in time stamps, interpolation or alignment algorithms can be used to adjust them, ensuring temporal synchronization. Time axis calibration ensures the accuracy of subsequent spatial association and fusion processing.

[0105] Step S137: Construct a spatial coordinate mapping model to convert the spatial dimension information in the global semantic feature set into a spatial coordinate system consistent with the real-time coordinate data of the UAV. The spatial dimension information includes distance dimension information, orientation dimension information, and height dimension information.

[0106] The spatial dimension information (distance, azimuth, altitude) in the global semantic feature set represents relative coordinates with respect to the UAV, while the UAV's real-time coordinate data represents absolute coordinates in the Earth coordinate system. A spatial coordinate mapping model is constructed to convert relative coordinates to absolute coordinates through coordinate transformation. Specifically, using parameters such as the UAV's real-time coordinates and flight attitude (pitch angle, roll angle, yaw angle), the target's relative distance, azimuth, and altitude are converted to absolute coordinates in the Earth coordinate system. For example, the target's horizontal and vertical offsets relative to the UAV are calculated using trigonometric functions, and then added to the UAV's absolute coordinates to obtain the target's absolute coordinates.

[0107] Step S138: Based on the calibrated time correspondence and the unified spatial coordinate system, construct a spatiotemporal bidirectional correlation model, and bidirectionally correlate the dynamic target semantic evolution features with the UAV's flight attitude data and real-time coordinate data, and bidirectionally correlate the static target semantic fixed features with the UAV's real-time coordinate data and flight operation data.

[0108] The spatiotemporal bidirectional correlation model considers the correlation between both time and space dimensions. For the semantic evolution features of dynamic targets, they are correlated with the UAV's flight attitude data and real-time coordinate data. For example, the relationship between the motion trajectory of dynamic targets and the UAV's attitude changes and position movements is analyzed to understand how the UAV's motion affects the observation of dynamic targets. For the semantic fixed features of static targets, they are correlated with the UAV's real-time coordinate data and flight operation data to analyze the relationship between static targets and the UAV's flight path and operational intentions, such as whether the UAV is approaching or moving away from a static target, and whether flight operations need to be adjusted based on the static target's position.

[0109] Step S139: Deeply fuse the semantic features and flight status data after bidirectional association to generate spatiotemporal bidirectional association data that simultaneously includes semantic attributes, spatial attributes, temporal attributes and UAV status attributes.

[0110] After completing time axis calibration and spatial coordinate unification, the semantic features and flight status data obtained through bidirectional correlation are deeply fused. Fusion methods can include feature concatenation, weighted summation, etc. For example, the changing parameters and feature attributes of the dynamic target's semantic evolution features are concatenated with the corresponding UAV flight attitude and coordinate data to form a comprehensive feature vector containing multifaceted information. Through deep fusion, spatiotemporally correlated data is generated, which simultaneously includes the target's semantic attributes, spatial attributes, temporal attributes, and the UAV's state attributes.

[0111] Step S140: Construct a dynamic environmental semantic feature map based on the spatiotemporal bidirectional correlation data, which includes the target's dynamic evolution relationship, the spatial semantic correlation relationship, and the UAV's own position correlation relationship.

[0112] The spatiotemporal bidirectional correlation data contains multifaceted information about targets and UAVs, and a dynamic environmental semantic feature map is constructed based on this data. This dynamic environmental semantic feature map represents the relationships between targets and between targets and UAVs in a graph structure, including the evolutionary relationships of dynamic targets, semantic relationships in the airspace, and the relationship between the UAV's own position and the target. The dynamic environmental semantic feature map can intuitively reflect the environmental conditions of the target area and the interrelationships between targets.

[0113] Step S141: Perform feature decomposition on the spatiotemporal bidirectional correlation data to separate dynamic target correlation data, static target correlation data and UAV self-correlation data.

[0114] The spatiotemporal bidirectional correlation data includes correlation information between dynamic targets, static targets, and the UAV itself. First, its features are decomposed, dividing the data into three parts: dynamic target correlation data, static target correlation data, and UAV self-correlation data. Dynamic target correlation data includes the semantic evolution features of dynamic targets and their correlation information with the UAV; static target correlation data includes the fixed semantic features of static targets and their correlation information with the UAV; and UAV self-correlation data includes the UAV's real-time flight status data.

[0115] Step S142: Extract the semantic evolution features, motion trajectory information and time series information of the dynamic target association data, and encapsulate each dynamic target as an independent dynamic node. Each dynamic node contains the complete attribute information and state change information of the target.

[0116] Extract dynamic target semantic evolution features, motion trajectory information (continuous feature evolution trajectory), and time series information from dynamic target association data. Each dynamic target is encapsulated as an independent dynamic node, which contains complete attribute information and state change information, including a unified target identifier, feature attributes, change parameters, motion trajectory, time series, and semantic description information.

[0117] Step S143: Extract the fixed semantic features, spatial distribution information and attribute description information of static targets from the static target association data, and encapsulate each static target as an independent static node. Each static node contains the complete attribute information and spatial location information of the target.

[0118] Extract fixed semantic features, spatial distribution information (such as coordinates and relative position with other static targets), and attribute description information of static targets from the associated data. Encapsulate each static target as an independent static node, which contains complete attribute information such as the target's static feature identifier, fixed attributes, spatial distribution features, spatial location information, and semantic description information.

[0119] Step S144: Extract real-time flight status data, position coordinate information and operation intention information from the UAV's own associated data, and encapsulate the UAV itself as an independent main node. The main node contains the UAV's complete status information and behavior information.

[0120] Extract real-time flight status data (flight attitude, speed, coordinates, etc.), position coordinate information, and operational intent information (intent reflected in the flight operation data) from the drone's own associated data. Encapsulate the drone itself as an independent subject node, which contains complete status and behavioral information such as the drone's identity, real-time status parameters, position coordinates, and operational intent.

[0121] Step S145: Based on the motion trajectory information and time series information of dynamic nodes, analyze the possible correlation between the intersection of motion trajectories and state influence between different dynamic nodes, and establish dynamic correlation edges between the correlated dynamic nodes. The correlation edges include correlation type, correlation strength and correlation time series information.

[0122] For dynamic nodes, based on their motion trajectory information and time series information, we analyze whether there is a possibility of intersection between different dynamic nodes. For example, by comparing the motion trajectories of two dynamic nodes, we determine whether they will meet or approach each other at some point in the future. Simultaneously, we analyze their state-influenced relationships, such as whether they are similar targets (e.g., both are vehicles) or whether they will interfere with each other (e.g., traveling in the same lane). For dynamic nodes with potential intersection or state-influenced relationships, we establish dynamic association edges between them. The attributes of these association edges include association type (e.g., intersection association, interference association), association strength (reflecting the closeness of the association), and association time series information (the time range in which the association occurs).

[0123] Step S1451: Extract the motion trajectory information of each dynamic node, wherein the motion trajectory information includes the spatial coordinate sequence and motion state parameters of the dynamic node at different time nodes.

[0124] The motion trajectory information of each dynamic node records the sequence of spatial coordinates (x, y, z) of the node at different time points, as well as motion state parameters (velocity, acceleration, etc.). The above information is the basis for analyzing the relationship between dynamic nodes.

[0125] Step S1452: Synchronize and align the motion trajectory information of all dynamic nodes according to the time series, so that all trajectory data can be analyzed based on the same time reference.

[0126] To accurately compare the motion trajectories of different dynamic nodes, their trajectory information needs to be synchronized and aligned according to the time series. Using a unified time reference (such as the system time of the UAV) as the standard, the trajectory data of each dynamic node is interpolated or adjusted to ensure that there are corresponding spatial coordinates and motion state parameters at the same point in time.

[0127] Step S1453: Compare and analyze the motion trajectory information of any two dynamic nodes one by one, extract the spatial coordinate parameters of the two trajectories at the same time node, and calculate the real-time spatial distance between the two nodes.

[0128] For any two dynamic nodes, extract the spatial coordinate parameters of the same time point from the synchronized time series. Then, calculate the distance between these two spatial coordinates to obtain the real-time spatial distance. By comparing the real-time spatial distances at different time points, we can understand the relative positional changes between the two dynamic nodes.

[0129] Step S1454: Based on the motion state parameters of the two nodes, predict the extension path of the motion trajectory of the two nodes within a subsequent preset time period, and calculate the coordinates of the spatial intersection point and the intersection time node of the extension path. The motion state parameters are velocity dimension information and motion direction information.

[0130] Based on the current motion state parameters (velocity, direction, etc.) of two dynamic nodes, assuming they maintain their current motion state for a subsequent preset time period, predict their trajectory extension paths. By solving for the intersection of the two extension paths, calculate the coordinates of the spatial intersection point and the intersection time node. If the two paths intersect, it indicates that the two dynamic nodes may meet at the intersection time node.

[0131] Step S1455: Analyze whether there are static nodes blocking or other environmental constraints in the spatial region corresponding to the intersection point coordinates, and determine whether the intersection point has the actual intersection conditions.

[0132] Even if the predicted trajectories of two dynamic nodes intersect, actual environmental factors must be considered. It's necessary to analyze whether there are static nodes (such as buildings or obstacles) obstructing the spatial area corresponding to the intersection point's coordinates, or whether there are other environmental constraints (such as road boundaries or restricted areas). If these factors exist, they may prevent the two dynamic nodes from actually intersecting; therefore, it's necessary to determine whether the intersection point meets the actual conditions for intersection.

[0133] Step S1456: Combining the real-time spatial distance change trajectory, the predicted intersection time node, and the intersection condition judgment results, comprehensively evaluate the intersection probability level of the two dynamic node motion trajectories.

[0134] Taking into account the real-time spatial distance trend (whether it gradually decreases), the predicted intersection time (whether it is within a reasonable time range), and the intersection condition judgment result (whether the actual intersection conditions are met), the intersection probability level of the motion trajectories of two dynamic nodes is evaluated. The intersection probability level can be divided into different levels such as high, medium, and low, which are used to represent the likelihood of the two nodes intersecting.

[0135] Step S1457: Analyze the semantic attribute information of the two dynamic nodes to determine whether there is a mutual influence relationship between their target types. The mutual influence relationship is a relationship between similar moving targets or a relationship between motion interference.

[0136] The semantic attribute information of dynamic nodes includes target type (such as vehicles, pedestrians, etc.). Analyzing the target types of two dynamic nodes determines whether there is a mutual influence relationship between them. For example, two vehicle nodes traveling on the same road may have a motion interference relationship; two pedestrian nodes active in the same area may have a similar moving target relationship.

[0137] Step S1458: Based on the interaction between the intersection probability level and the target type, determine the state influence association type between two dynamic nodes, and define the influence method, parameters and duration range.

[0138] Based on the mutual influence between the intersection probability level and the target type, the state influence association type between two dynamic nodes is determined. For example, if two vehicle nodes have a high intersection probability level and there is motion interference, the state influence association type between them may be "collision risk association". The influence method is mutual interference with driving routes, and the influence parameters include distance, relative speed, etc. The duration range is from the current time to the predicted intersection time node.

[0139] Step S1459: For two dynamic nodes that have the potential to intersect or are associated with each other due to state influence, establish dynamic association edges between the corresponding nodes in the feature graph, and label the association edges with attribute information, including the level of intersection potential, the type of association influence, and the duration of influence.

[0140] For two dynamic nodes that are determined to have a potential intersection or state influence relationship after analysis, a dynamic association edge is established between their corresponding nodes in the dynamic environment semantic feature graph. The association edge is then labeled with attributes such as the intersection probability level, influence association type, and influence duration to clearly represent the association relationship between the two dynamic nodes.

[0141] Step S14510: Integrate all established dynamic association edges with their corresponding dynamic nodes to generate a complete dynamic node association network, which reflects the actual motion association and state influence association between dynamic nodes.

[0142] All dynamic nodes and their dynamic connections are integrated to form a dynamic node association network. This network can intuitively demonstrate the actual motion and state influence relationships between dynamic nodes.

[0143] Step S146: Based on the spatial distribution information of static nodes, analyze the spatial distance association, occlusion association and adjacency association between different static nodes, and establish static association edges between static nodes that are associated. The association edges include the association type, spatial distance parameters and positional relationship description.

[0144] The spatial distribution information of static nodes reflects their location and relationships in space. This involves analyzing spatial distance relationships (e.g., proximity), occlusion relationships (e.g., whether one node occludes another), and adjacency relationships (e.g., whether they are closely adjacent) between different static nodes. For static nodes with these relationships, static association edges are established between them, and attribute information such as association type, spatial distance parameters, and positional relationship descriptions are labeled.

[0145] Step S1461: Extract the spatial coordinate information and spatial morphology information of each static node. The spatial morphology information includes the outline range, occupied spatial volume and surface structure features of the static target.

[0146] The spatial coordinate information of each static node is its position in the absolute coordinate system, while the spatial morphology information describes the physical form of the static target, such as the outline range (length, width, height), the volume of space it occupies, and surface structure features (such as whether there are protruding parts, surface flatness, etc.). The above information is used to analyze the spatial relationship between static nodes.

[0147] Step S1462: Convert the spatial coordinate information of all static nodes into coordinate data in a unified three-dimensional spatial coordinate system, calculate the spatial coordinate information of any two static nodes one by one, and obtain the straight-line distance parameter and spatial orientation relationship parameter between the two static nodes.

[0148] Transform the spatial coordinates of all static nodes to the same three-dimensional spatial coordinate system for unified spatial analysis. For any two static nodes, calculate their straight-line distance (Euclidean distance in three-dimensional space) and spatial orientation parameters (such as azimuth and pitch angles) to determine their relative positions.

[0149] Step S1463: Based on the spatial morphology information and spatial orientation parameters of the two nodes, analyze the positional overlap of the two nodes in three-dimensional space, and determine whether there are areas of mutual occlusion and the proportion of occlusion area.

[0150] Based on the spatial morphological information (outline range, volume) and spatial orientation parameters of two static nodes, geometric calculations are used to analyze whether their positions overlap in three-dimensional space. If overlap exists, the occlusion area ratio is further calculated, that is, the proportion of the area of ​​one node occluded by the other node to its total area.

[0151] Step S1464: Divide the spatial distance interval according to the straight-line distance parameter, and determine the occlusion association type between two static nodes in combination with the occlusion area ratio. The occlusion association type is complete occlusion association, partial occlusion association, and no occlusion association.

[0152] The distance between static nodes is divided into different intervals (e.g., close, medium, and long distance) based on the straight-line distance parameter. The occlusion association type is determined by combining this with the occlusion area ratio. For example, if two nodes are close and the occlusion area ratio is greater than a certain threshold (e.g., 80%), it is a complete occlusion association; if the occlusion area ratio is within a certain range (e.g., 20%-80%), it is a partial occlusion association; and if the occlusion area ratio is less than the threshold (e.g., 20%), it is an unoccluded association.

[0153] Step S1465: Based on the spatial orientation relationship parameters and the straight-line distance parameters, determine whether two static nodes are in adjacent spatial regions and determine the adjacent association type, wherein the adjacent association type is direct adjacent association, indirect adjacent association, or non-adjacent association.

[0154] The relative orientation of two static nodes is determined based on spatial orientation parameters, and their straight-line distance is used to determine whether they are in adjacent spatial regions. If two nodes are very close (within a preset adjacent distance threshold) and are directly connected in space, they are directly adjacent; if there are other static nodes between the two nodes, but the distance is still within a certain range, they are indirectly adjacent; if the distance exceeds the adjacent distance threshold, they are not adjacent.

[0155] Step S1466: Integrate spatial distance intervals, occlusion association types, and adjacent association types to generate a set of spatial association features between two static nodes.

[0156] By integrating information such as spatial distance range, occlusion association type, and adjacent association type, a set of spatial association features between two static nodes is formed, which comprehensively describes the spatial association relationship between them.

[0157] Step S1467: For two static nodes with a clear association relationship in the spatial association feature set, establish a static association edge between the corresponding nodes in the feature graph. The clear association relationship is either occlusion association or adjacent association.

[0158] If two static nodes have occlusion associations (complete or partial occlusion) or adjacency associations (direct or indirect adjacency) in their spatial association feature sets, then static association edges are established between their corresponding nodes in the dynamic environment semantic feature graph.

[0159] Step S1468: Label attribute information on the static association edge. The attribute information includes spatial distance parameters, occlusion association type, adjacent association type and association region coordinates, so that the static association edge reflects the spatial association details between static nodes.

[0160] Label the static associated edges with attributes such as spatial distance parameters, occlusion association type, adjacent association type, and coordinates of associated regions (e.g., the coordinate range of the occluded region and the coordinate range of the adjacent region) to describe in detail the spatial association details between static nodes.

[0161] Step S1469: Integrate all static associated edges with their corresponding static nodes to generate a spatial association network between static nodes.

[0162] By integrating all static nodes and the static association edges between them, a spatial association network of static nodes is formed, which reflects the distribution and interrelationships of static targets in space.

[0163] Step S147: Based on the position coordinate information of the main node and the spatial position information of the dynamic and static nodes, analyze the spatial distance relationship, relative motion relationship and safety impact relationship between the main node and each dynamic and static node, and establish the main node association edge between the main node and each target node. The association edge includes the association type, safety impact parameter and relative position parameter.

[0164] There are multiple relationships between the main node (UAV) and dynamic and static nodes. The analysis focuses on the spatial distance relationships (real-time distance), relative motion relationships (relative speed, relative direction), and safety impact relationships (whether there is a collision risk, whether it affects flight safety) between the main node and each target node. For main nodes and target nodes with these relationships, a main node association edge is established between them, and attribute information such as association type, safety impact parameters (e.g., risk level), and relative position parameters (e.g., relative azimuth, distance) are labeled.

[0165] Step S1471: Extract the real-time position coordinates, flight speed and flight direction information of the main node to reflect the current spatial position and motion status of the UAV.

[0166] The real-time position coordinates of the main node represent the position of the UAV in the absolute coordinate system. The flight speed information includes the speed magnitude and direction. The flight direction information can be represented by the heading angle. The above information reflects the current spatial position and motion state of the UAV.

[0167] Step S1472: Extract the real-time position coordinates, motion speed and motion direction information of each dynamic node, and extract the real-time position coordinates and spatial morphology information of each static node.

[0168] For each dynamic node, extract its current real-time position coordinates and motion speed (magnitude and direction); for each static node, extract its real-time position coordinates and spatial morphological information (contour range, volume, etc.).

[0169] Step S1473: Calculate the real-time spatial distance between the main node and each dynamic node, and combine the flight speed information and flight direction information of both to calculate the relative speed parameter and relative motion direction parameter.

[0170] Based on the real-time position coordinates of the main node and the dynamic node, the real-time spatial distance between them is calculated. Combining the flight speed and direction information of both, the relative velocity parameters (relative velocity magnitude and direction) and relative motion direction parameters (such as approaching or moving away) are calculated through vector operations.

[0171] Step S1474: Based on the relative velocity parameters and relative motion direction parameters, predict the trajectory of spatial distance change between the main node and the dynamic node within the subsequent preset time period, and determine whether there is a potential correlation between distance reduction or collision.

[0172] Based on relative velocity and relative motion direction parameters, predict how the spatial distance between the main node and the dynamic node will change over a preset time period. If the predicted distance gradually decreases and may fall below a safe distance threshold at some point, a potential collision is identified.

[0173] Step S1475: Calculate the real-time spatial distance between the main node and each static node, and combine the spatial morphology information of the static nodes to analyze the possibility of spatial overlap between the current flight path of the UAV and the static nodes.

[0174] The real-time spatial distance between the main node and the static node is calculated. Combined with the spatial morphology information of the static node (such as its outline range), it is determined whether the UAV's current flight path will spatially overlap with the static node. For example, if the UAV's flight path passes through the outline range of the static node, there is a possibility of spatial overlap.

[0175] Step S1476: Based on real-time spatial distance, distance change trajectory and possible spatial overlap, divide the safety impact interval between the main node and each target node, and define the risk parameters corresponding to different intervals. The safety impact interval is a near-distance safety impact interval, a medium-distance safety impact interval and a long-distance safety impact interval.

[0176] Based on real-time spatial distance, distance change trajectory (dynamic nodes), or potential spatial overlap (static nodes), the safety impact between the main node and the target node is divided into different intervals. For example, a real-time spatial distance less than a certain threshold (e.g., 50 meters) is considered a short-range safety impact interval, corresponding to higher risk parameters; a distance between threshold 1 and threshold 2 (e.g., 50 meters - 200 meters) is considered a medium-range safety impact interval, corresponding to medium risk parameters; and a distance greater than threshold 2 is considered a long-range safety impact interval, corresponding to lower risk parameters.

[0177] Step S1477: Determine the safety impact association type between the main node and each target node based on the safety impact range. For main nodes and target nodes with safety impact association, establish main association edges between the corresponding nodes in the feature map. Label the association edges with attribute information such as real-time spatial distance, relative motion parameters, safety impact range and risk parameters.

[0178] The type of safety impact association is determined based on the safety impact range, such as "high-risk association", "medium-risk association", and "low-risk association". For subject nodes and target nodes with safety impact associations, subject association edges are established in the dynamic environment semantic feature graph, and attribute information such as real-time spatial distance, relative motion parameters (dynamic nodes), safety impact range, and risk parameters are labeled.

[0179] Step S1478: Add dynamic update trigger conditions for the main associated edges corresponding to dynamic nodes. When the motion state of the dynamic node changes by a preset magnitude, the attribute information of the associated edges is automatically updated.

[0180] The motion state of dynamic nodes may change, so it is necessary to add dynamic update trigger conditions to the associated edges of the corresponding dynamic nodes. For example, when the change in the velocity of a dynamic node exceeds a preset threshold or the change in the angle of its motion direction exceeds a preset threshold, the attribute information of the associated edges should be automatically updated to reflect the latest relative motion state and safety impact.

[0181] Step S1479: For the main associated edges corresponding to static nodes, and in combination with the spatial morphological stability of static nodes, set up a periodic update process to make the associated edge attribute information reflect the latest spatial positional relationship and generate a complete association network between the main node and the target node.

[0182] The spatial form of static nodes is relatively stable, but the main node (drone) is constantly moving. Therefore, the attribute information of the main node's associated edges (such as real-time spatial distance and relative position parameters) will change over time. A periodic update process (such as at regular time intervals) is set up to update this attribute information, ensuring that the main node's associated edges reflect the latest spatial relationships. All main node associated edges are integrated with the main node and target node to generate a complete association network between the main node and the target node.

[0183] Step S148: Arrange all dynamic nodes, static nodes, and main nodes in the feature graph space according to their actual spatial location and temporal relationship, and combine them with dynamic association edges, static association edges, and main association edges to form the basic structure of the dynamic environment semantic feature graph.

[0184] In the dynamic environment semantic feature graph, dynamic nodes, static nodes, and main nodes are arranged according to their actual spatial positions, with the node position coordinates corresponding to their absolute coordinates in three-dimensional space. Simultaneously, temporal relationships are considered, such as the temporal continuity of the movement trajectories of dynamic nodes. Dynamic, static, and main nodes are connected according to the relationships between nodes, forming the basic structure of the dynamic environment semantic feature graph.

[0185] Step S149: Add dynamic update rules to the dynamic environment semantic feature map, and update the node attribute information and associated edge information in real time according to the newly generated spatiotemporal bidirectional correlation data, so that the dynamic environment semantic feature map reflects the real-time state changes of the target area.

[0186] Dynamic environment semantic feature maps need to reflect the state changes of the target area in real time. Therefore, dynamic update rules are added. When new spatiotemporal bidirectional correlation data is generated, the attribute information of nodes in the dynamic environment semantic feature map (such as the position and state of dynamic nodes, and the flight state of main nodes) and the attribute information of associated edges (such as correlation strength and risk parameters) are updated based on the information in the data. Update rules can include triggering conditions (such as data update frequency and the magnitude of node state changes) and update methods (such as direct replacement and incremental update).

[0187] Step S1410: The node attributes and relationships in the dynamic environment semantic feature map are verified and corrected by an iterative optimization algorithm to generate a dynamic environment semantic feature map that includes the target dynamic evolution relationship, the spatial semantic relationship, and the UAV's own position relationship.

[0188] To improve the accuracy of the dynamic environment semantic feature map, an iterative optimization algorithm is used to verify and correct node attributes and relationships. The iterative optimization algorithm, based on methods such as minimizing the energy function of a graph model or probabilistic reasoning, adjusts node attribute values ​​and the weights of associated edges through multiple iterations, enabling the feature map to more accurately reflect the actual situation of the target area. After iterative optimization, the final dynamic environment semantic feature map is generated, which includes the target's dynamic evolution relationship, spatial semantic relationships, and the UAV's own positional relationships.

[0189] Step S150: Perform full-domain scene situation inference and dynamic division of safety boundaries through the dynamic environment semantic feature map, and output full-domain perception decision results covering the distribution of dynamic and static targets in the target area, motion evolution trend and safe flight airspace of UAV.

[0190] The dynamic environment semantic feature map contains rich relationships and state information within the target area. Based on this map, global scenario situation inference and safety boundary delineation are performed. Situation inference predicts the motion evolution trend of the target, and safety boundary delineation determines the safe flight area of ​​the UAV. The final output is a global perception decision result that includes the distribution of dynamic and static targets, motion evolution trends, and safe flight airspace.

[0191] Step S151: Extract the motion trajectory information, semantic attribute information and correlation information of all dynamic nodes from the dynamic environment semantic feature map, and integrate them to form a dynamic target situation dataset.

[0192] Dynamic target situation datasets are used to analyze the overall situation of dynamic targets. Motion trajectory information (continuous feature evolution trajectory), semantic attribute information (type, state, etc.), and correlation information (associations with other dynamic nodes and main nodes) of each dynamic node are extracted from the dynamic environment semantic feature map. This information is then integrated to form a dynamic target situation dataset.

[0193] Step S152: Extract the spatial distribution information, semantic attribute information and correlation information of all static nodes from the dynamic environment semantic feature map, and integrate them to form a static target situation dataset.

[0194] Static target situation datasets are used to analyze the spatial distribution and correlation of static targets. Spatial distribution information (coordinates, shape, etc.), semantic attribute information (type, attributes, etc.), and correlation information (associations with other static nodes and main nodes) of each static node are extracted from the dynamic environment semantic feature map and integrated to form a static target situation dataset.

[0195] Step S153: Based on the dynamic target situation dataset, analyze the motion evolution parameters of each dynamic target through time-series extrapolation algorithm, predict the spatial position change sequence and state change trend of the dynamic target within the subsequent preset time period, and obtain the predicted motion trajectory of the dynamic target.

[0196] Temporal projection algorithms utilize the historical motion trajectory and motion evolution parameters (velocity, acceleration, etc.) of a dynamic target to predict its motion within a predetermined time period. For example, algorithms such as Kalman filtering and particle filtering can be used to predict the future spatial position change sequence and state change trend (such as velocity change and direction change) of the dynamic target based on its current state and motion model, thus obtaining the predicted motion trajectory of the dynamic target.

[0197] Step S154: Based on the static target situation dataset, construct the static spatial topology of the target area, wherein the static spatial topology includes the spatial distribution pattern of static targets and the range of non-flying areas.

[0198] Static spatial topology reflects the distribution and organization of static targets in space. Based on a static target situational dataset, the static spatial topology of the target area is constructed, including the spatial distribution pattern of static targets (such as the arrangement of buildings and the direction of roads) and the extent of non-flying zones (such as the space occupied by buildings and no-fly zones). The extent of non-flying zones can be determined using the spatial morphological information of static targets and preset safety distances.

[0199] Step S155: Overlay analysis is performed on the predicted motion trajectory of the dynamic target and the static spatial topology of the target area to identify the spatial overlap area between the future position of the dynamic target and the area occupied by the static target.

[0200] The predicted trajectory of a dynamic target is overlaid with the non-flying area in the static spatial topology. The analysis then examines whether the dynamic target's position will overlap with the area occupied by the static target within a predetermined future timeframe. If overlap exists, this area is considered a spatial overlap region, indicating that the dynamic target may enter the non-flying area or collide with the static target.

[0201] Step S156: Based on the association edge information between the main node and each target node, and by combining the predicted motion trajectory of the dynamic target, the static spatial topology, and the identified spatial overlapping areas, dynamically divide the safe zone, warning zone, and danger zone of the UAV flight, and define the spatial range and boundary coordinates of the safe zone, warning zone, and danger zone.

[0202] By combining the association edge information between the main node and each target node (such as safety impact intervals and risk parameters), as well as the predicted motion trajectory of dynamic targets, static spatial topology, and spatial overlapping areas, the flight area of ​​the UAV is dynamically divided. The safe zone refers to the area where the UAV can fly safely, the warning zone is the area requiring vigilance, and the danger zone is the area where flight is prohibited. By analyzing the boundaries of each zone, their spatial extent and boundary coordinates are defined.

[0203] For example, step S1561: extract the safety impact interval information, real-time spatial distance information and relative motion parameter information from the main body associated edge, and establish a set of safety assessment indicators.

[0204] Information such as the safety impact range (near distance, medium distance, long distance), real-time spatial distance, and relative motion parameters (relative speed, direction) are extracted from the main body's associated edges. This information is then used as a safety assessment indicator to establish a set of safety assessment indicators.

[0205] Step S1562: Based on the static spatial topology, extract the coordinates of the occupied space range of the static target and define the boundary of the absolutely non-flying spatial region.

[0206] The static spatial topology includes the coordinates of the space occupied by static targets. The area enclosed by these coordinates is defined as the boundary of an absolutely no-fly zone, and drones are strictly prohibited from entering this area.

[0207] Step S1563: Combine the predicted trajectory of the dynamic target with the identified spatial overlap area to extract the coordinates of the spatial coverage area of ​​the dynamic target within a subsequent preset time period, and define the boundary of the dynamic non-flying spatial area.

[0208] Based on the predicted trajectory and spatial overlap area of ​​the dynamic target, determine the coordinates of the spatial coverage area that the dynamic target may occupy within a preset time period. These coordinates are defined as the boundary of a dynamically non-flying spatial area, and drones should avoid entering this area during this time period.

[0209] Step S1564: Based on the set of safety assessment indicators, and combined with the minimum turning radius, climb rate and preset safe interval distance of the UAV, set the criteria for dividing the safe zone, warning zone and dangerous zone.

[0210] Information such as the safety impact range and real-time spatial distance from the safety assessment indicator set, combined with the UAV's performance parameters (minimum turning radius, climb rate) and preset safety interval distances, is used to set standards for dividing safe zones, warning zones, and danger zones. For example, areas where the distance to absolutely no-fly zones and dynamically no-fly zones is greater than the safety interval distance, and where the safety impact range is a long distance, are classified as safe zones; areas where the distance is between the safety interval distance and the warning interval distance, or where the safety impact range is a medium distance, are classified as warning zones; and areas where the distance is less than the safety interval distance, or where the safety impact range is a short distance, are classified as danger zones.

[0211] Step S1565: Overlay the boundary of the absolutely non-flying area with the boundary of the dynamically non-flying area to generate the initial danger zone range.

[0212] The boundaries of the absolutely no-fly zone and the dynamically no-fly zone are superimposed, and the overlapping areas are merged to generate an initial danger zone. This initial danger zone includes all areas where drones are prohibited from entering.

[0213] Step S1566: Taking the current position of the main node as the center, based on the division criteria of safe area, warning area and dangerous area, expand and define the scope of the warning area outward. The warning area surrounds the dangerous area and includes the spatial range that may be affected by dynamic targets.

[0214] Centered on the drone's current location, a warning zone is defined outside the initial danger zone according to the established criteria. The warning zone should encompass the spatial range potentially affected by the dynamic target, i.e., a certain area around the predicted trajectory of the dynamic target, to alert drone operators to potential risks.

[0215] Step S1567: Define the remaining space within the target area, excluding the dangerous area and the warning area, as the safe area. The safe area must meet the requirements of the set of safety assessment indicators in terms of spatial distance from all target nodes.

[0216] The remaining space within the target area, excluding the danger zone and the warning zone, is defined as the safe zone. Within the safe zone, the spatial distance between the drone and all target nodes should meet the safe distance requirements in the safety assessment indicator set to ensure flight safety.

[0217] Step S1568: Calculate the set of boundary coordinate points of the safe area, the warning area and the danger area respectively using the spatial boundary extraction algorithm.

[0218] Spatial boundary extraction algorithms (such as edge detection and contour extraction) are used to extract the boundaries of safe areas, warning areas, and dangerous areas, resulting in a set of boundary coordinate points for each area. These coordinate points are arranged in a certain order to form the boundary contour of the area.

[0219] Step S1569: Smooth the set of boundary coordinate points to generate continuous regional boundary lines. Transform the continuous regional boundary lines according to the three-dimensional spatial coordinate system to generate standardized regional boundary coordinate data, and define the spatial range and boundary location of the safe area, warning area and danger area.

[0220] The set of boundary coordinate points is smoothed to eliminate noise and discontinuities, generating continuous regional boundary lines. Then, these boundary lines are transformed according to a three-dimensional spatial coordinate system to unify them into an absolute coordinate system, generating standardized regional boundary coordinate data. This data accurately defines the spatial extent and boundary locations of safe zones, warning zones, and danger zones.

[0221] Step S157: Based on the division of safe areas, warning areas and dangerous areas, and combined with the real-time flight status data of the UAV, generate flight path suggestions and operation adjustment prompts adapted to the current scenario.

[0222] Based on the designated safe zones, warning zones, and danger zones, and combined with the UAV's real-time flight status data (current position, speed, heading, etc.), a safe and efficient flight path is proposed. The flight path should avoid danger zones and warning zones as much as possible, prioritizing safe zones. Simultaneously, based on the zone division results and the UAV's flight status, operational adjustment prompts are generated, such as suggesting deceleration and heading adjustments when approaching warning zones.

[0223] Step S158: Integrate dynamic target distribution information, static target distribution information, dynamic target motion evolution trend, safe flight airspace boundary coordinates, flight path suggestions and operation adjustment prompts to generate basic data for all-domain perception decision-making.

[0224] By integrating information such as dynamic target distribution information (location and type of dynamic nodes), static target distribution information (location and type of static nodes), dynamic target motion evolution trend (predicted motion trajectory), safe flight airspace boundary coordinates (boundaries of safe areas, warning areas, and dangerous areas), flight path suggestions, and operation adjustment prompts, a comprehensive perception and decision-making foundation data is formed.

[0225] Step S159: The overall perception decision-making base data is structured and organized according to the dimensions of target type, spatial location, time series, and safe zone, etc., to generate initial decision data containing the dynamic target distribution information, static target distribution information, dynamic target motion evolution trend, safe flight airspace boundary coordinates, flight path suggestions and operation adjustment prompts.

[0226] The foundational data for comprehensive perception decision-making is structured and organized according to dimensions such as target type (dynamic and static targets), spatial location (target distribution in different areas), time series (target status and prediction at different time points), and safety zones (safe zones, warning zones, and danger zones). For example, dynamic targets are classified according to types such as vehicles and pedestrians, and safe flight airspace is classified according to the boundary coordinates of different areas. Initial decision data is generated through this classification and organization.

[0227] Step S1510: The initial decision data is formatted according to a preset data structure to generate standardized global perception decision data.

[0228] Initial decision data needs to be formatted according to a pre-defined data structure to ensure a uniform format and standard. For example, JSON or XML data formats can be used to define the name, type, and meaning of each data field. The formatted global perception decision data can then be directly read and used by the UAV's flight control system or ground control station.

[0229] In one exemplary embodiment, a UAV global perception system based on four-dimensional imaging and deep learning is provided. This system can be a terminal, server, etc., and its internal structure diagram can be as follows: Figure 2As shown, this UAV all-domain perception system based on four-dimensional imaging and deep learning includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, near-field communication, or other technologies. When the computer program is executed by the processor, it implements a UAV all-domain perception method based on four-dimensional imaging and deep learning. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device can be a touch layer covering the display screen, or a button, trackball, or touchpad set on the shell of the UAV global perception system based on four-dimensional imaging and deep learning, or an external keyboard, touchpad, or mouse, etc.

[0230] It should be noted that, in order to simplify the description of the present invention and thus help to understand one or more embodiments of the invention, multiple features may sometimes be grouped into one embodiment, drawing or description thereof in the foregoing description of the embodiments of the present invention.

Claims

1. A method for all-domain perception of unmanned aerial vehicles (UAVs) based on four-dimensional imaging and deep learning, characterized in that, The method includes: The original detection signal of the target area is obtained by the four-dimensional imaging millimeter-wave radar carried by the UAV, and the four-dimensional imaging enhanced data containing distance dimension information, azimuth dimension information, height dimension information, and velocity dimension information is generated through multi-modal signal complementary enhancement processing. The four-dimensional imaging enhancement data is input into a deep learning dynamic semantic modeling network. Through feature evolution analysis and dynamic target trajectory association processing, a global semantic feature set containing dynamic target semantic evolution features and static target semantic fixed features is generated. Acquire real-time flight status data of the UAV, and perform spatiotemporal bidirectional correlation and fusion processing on the real-time flight status data and the global semantic feature set to generate spatiotemporal bidirectional correlation data; Based on the aforementioned spatiotemporal bidirectional correlation data, a dynamic environmental semantic feature map is constructed, which includes the target's dynamic evolution relationship, the spatial semantic correlation relationship, and the UAV's own position correlation relationship. The dynamic environment semantic feature map is used to perform full-domain scene situation inference and dynamic division of safety boundaries, and output full-domain perception decision results covering the distribution of dynamic and static targets in the target area, motion evolution trend, and safe flight airspace of UAVs.

2. The UAV global perception method based on four-dimensional imaging and deep learning according to claim 1, characterized in that, The raw detection signal of the target area is acquired by a four-dimensional imaging millimeter-wave radar mounted on a UAV. After multi-modal signal complementary enhancement processing, four-dimensional imaging enhanced data containing distance, azimuth, altitude, and velocity dimension information is generated, including: The drone is controlled to cruise the target area along a full-coverage flight path, and the onboard four-dimensional imaging millimeter-wave radar is simultaneously activated to enter the multi-band detection mode and transmit multi-band millimeter-wave detection signals to the target area. The system receives multi-band echo signals reflected by various objects within the target area and integrates all echo signals to form a raw detection signal set, which contains complete echo data of different frequency bands and different reflection angles. The original detection signal set is subjected to multi-band signal noise reduction processing. The environmental interference component and the effective reflection component in the signal are separated by adaptive signal filtering technology, and the effective signal carrying the target information is retained. The effective signal after noise reduction is fused by multi-view signal complementarity. The effective signals of different frequency bands and different reflection angles are superimposed according to the spatial position correspondence to generate the fused effective signal. The distance-related signal components are extracted from the fused effective signal, and the relative distance parameters between each reflection point and the UAV are calculated through signal propagation delay analysis to generate distance dimension information. The azimuth-related signal components are extracted from the fused effective signal. Based on the phase difference analysis of the radar array antenna, the projection direction parameters of each reflection point on the horizontal plane are determined, and the azimuth dimension information is generated. The height-related signal components in the fused effective signal are extracted, and the actual altitude parameters of each reflection point are calculated by combining the real-time flight altitude data of the UAV with the vertical angle offset information of the signal, thus generating altitude dimension information. The velocity-related signal components in the fused effective signal are extracted, and the motion velocity parameters of each reflection point relative to the UAV are calculated by Doppler frequency offset analysis to generate velocity dimension information. The distance, orientation, height, and velocity dimensions are dynamically compensated. The compensated distance, orientation, height, and velocity dimensions are then correlated one-to-one with spatial coordinates and time nodes to generate four-dimensional imaging enhancement data with spatial and temporal correspondence.

3. The UAV global perception method based on four-dimensional imaging and deep learning according to claim 1, characterized in that, The process involves inputting the four-dimensional imaging enhancement data into a deep learning dynamic semantic modeling network, and through feature evolution analysis and dynamic target trajectory association processing, generating a global semantic feature set containing dynamic target semantic evolution features and static target semantic fixed features, including: The pre-trained deep learning dynamic semantic modeling network is invoked. The deep learning dynamic semantic modeling network includes a multi-scale feature extraction layer, a dynamic feature evolution layer, and a semantic association layer, and is trained using multiple four-dimensional imaging sample data and semantic label data. The four-dimensional imaging enhancement data is divided into continuous time-series data segments according to the time sequence, and each time-series data segment contains complete four-dimensional imaging enhancement data within a preset time length; Each time series data segment is input into the multi-scale feature extraction layer of the deep learning dynamic semantic modeling network. Through convolution operations of multiple layers with different receptive fields, the local detail features and global context features of each time series data segment are extracted. Local detail features and global context features are input into the dynamic feature evolution layer. The features of continuous time-series data segments are analyzed through a time-series recurrent network to track the change trajectory of features over time and filter out dynamic feature groups that have continuous changes and static feature groups that remain stable. Dynamic target trajectory association processing is performed on dynamic feature groups to match and associate features of the same target in different time series data segments, generating continuous feature evolution trajectory of each dynamic target, extracting the changing parameters and feature attributes in the continuous feature evolution trajectory, and generating dynamic target semantic evolution features. Spatial topological association processing is performed on static feature groups to analyze the distribution relationship and association pattern of different static features in space, extract fixed attributes and spatial distribution features from the features, and generate static target semantic fixed features. The dynamic semantic evolution features and static semantic fixed features of the target are input into the semantic association layer. Based on the preset semantic label system, corresponding semantic description information is matched for each feature. The semantic description information represents the type attribute and state attribute of the target. Dynamic semantic evolution features of targets are bound to their corresponding semantic description information to generate dynamic semantic units; static semantic fixed features of targets are bound to their corresponding semantic description information to generate static semantic units. All dynamic and static semantic units are deduplicated, and all deduplicated dynamic and static semantic units are integrated to generate a global semantic feature set containing dynamic target semantic evolution features and static target semantic fixed features.

4. The UAV global perception method based on four-dimensional imaging and deep learning according to claim 1, characterized in that, The process of acquiring real-time flight status data of the UAV and performing spatiotemporal bidirectional correlation fusion processing on the real-time flight status data and the global semantic feature set to generate spatiotemporal bidirectional correlation data includes: Flight attitude data of the UAV is collected by the inertial measurement equipment on board the UAV. The flight attitude data reflects the pitch, roll and yaw states of the UAV. The drone's real-time coordinate data is collected by the satellite positioning equipment carried by the drone, and the real-time coordinate data reflects the drone's precise position in three-dimensional space. The flight control module of the UAV collects flight operation data, which reflects the UAV's flight speed adjustment status, heading adjustment status and altitude control status. By integrating the flight attitude data, real-time coordinate data, and flight operation data, real-time flight status data containing the current flight status and operational intent of the UAV is generated. Extract the time stamp information corresponding to each semantic unit in the global semantic feature set to determine the collection time node of each semantic feature; extract the time stamp information corresponding to each component in the real-time flight status data to determine the recording time node of each flight status parameter; Based on time stamp information, the global semantic feature set and the real-time flight status data are time-axis calibrated so that the semantic features at the same time point correspond to the flight status data. A spatial coordinate mapping model is constructed to convert the spatial dimension information in the global semantic feature set into a spatial coordinate system consistent with the real-time coordinate data of the UAV. The spatial dimension information includes distance dimension information, orientation dimension information, and altitude dimension information. Based on the calibrated time correspondence and the unified spatial coordinate system, a spatiotemporal bidirectional correlation model is constructed to bidirectionally correlate the semantic evolution features of dynamic targets with the flight attitude data and real-time coordinate data of UAVs, and to bidirectionally correlate the semantic fixed features of static targets with the real-time coordinate data and flight operation data of UAVs. The semantic features and flight status data after bidirectional association are deeply fused to generate spatiotemporal bidirectional associated data that simultaneously includes semantic attributes, spatial attributes, temporal attributes and UAV status attributes.

5. The UAV global perception method based on four-dimensional imaging and deep learning according to claim 1, characterized in that, The construction of a dynamic environmental semantic feature map based on the spatiotemporal bidirectional correlation data, which includes the target's dynamic evolution relationship, spatial semantic correlation relationship, and UAV's own position correlation relationship, includes: The spatiotemporal bidirectional correlation data is feature-splitting to separate dynamic target correlation data, static target correlation data, and UAV self-correlation data; Extract the semantic evolution features, motion trajectory information and time series information of dynamic targets from the dynamic target association data, and encapsulate each dynamic target as an independent dynamic node. Each dynamic node contains the complete attribute information and state change information of the target. Extract the fixed semantic features, spatial distribution information and attribute description information of static targets from the static target association data, and encapsulate each static target as an independent static node. Each static node contains the complete attribute information and spatial location information of the target. Extract real-time flight status data, position coordinate information and operation intention information from the drone's own associated data, and encapsulate the drone itself as an independent subject node. The subject node contains the drone's complete status information and behavior information. Based on the motion trajectory information and time series information of dynamic nodes, the intersection of motion trajectories between different dynamic nodes may be related to the influence of state. Dynamic association edges are established between dynamic nodes that are associated, and the association edges include association type, association strength and association time series information. Based on the spatial distribution information of static nodes, we analyze the spatial distance association, occlusion association and adjacency association between different static nodes. Static association edges are established between static nodes that are associated. The association edges include the association type, spatial distance parameters and positional relationship description. Based on the location coordinate information of the main node and the spatial location information of the dynamic and static nodes, we analyze the spatial distance relationship, relative motion relationship and safety impact relationship between the main node and each dynamic and static node. We establish main node association edges between the main node and each target node. The association edges include association type, safety impact parameters and relative position parameters. All dynamic nodes, static nodes, and main nodes are arranged in the feature graph space according to their actual spatial location and temporal relationship, and combined with dynamic association edges, static association edges, and main association edges to form the basic structure of dynamic environment semantic feature graph; Add dynamic update rules to the dynamic environment semantic feature map, and update the node attribute information and associated edge information in real time according to the newly generated spatiotemporal bidirectional correlation data, so that the dynamic environment semantic feature map reflects the real-time state changes of the target area. The node attributes and relationships in the dynamic environment semantic feature map are verified and corrected by an iterative optimization algorithm to generate a dynamic environment semantic feature map that includes the target's dynamic evolution relationship, spatial semantic relationship, and UAV's own position relationship.

6. The UAV global perception method based on four-dimensional imaging and deep learning according to claim 5, characterized in that, The motion trajectory information and time series information based on dynamic nodes are analyzed to determine whether the intersection of motion trajectories between different dynamic nodes may be related to state influences. Dynamic association edges are established between related dynamic nodes, including: Extract the motion trajectory information of each dynamic node, which includes the spatial coordinate sequence and motion state parameters of the dynamic node at different time nodes; Synchronize and align the motion trajectory information of all dynamic nodes according to the time series, so that all trajectory data can be analyzed based on the same time reference; The motion trajectory information of any two dynamic nodes is compared and analyzed one by one, the spatial coordinate parameters of the two trajectories at the same time node are extracted, and the real-time spatial distance between the two nodes is calculated. Based on the motion state parameters of two nodes, predict the extension path of the motion trajectory of the two nodes within a subsequent preset time period, and calculate the spatial intersection point coordinates and intersection time node of the extension path. The motion state parameters are velocity dimension information and motion direction information. Analyze whether there are static nodes blocking or other environmental constraints in the spatial region corresponding to the intersection point coordinates, and determine whether the intersection point has the actual conditions for intersection. By combining the real-time spatial distance change trajectory, the predicted intersection time node, and the judgment results of the intersection conditions, the probability level of intersection of the motion trajectories of the two dynamic nodes is comprehensively evaluated. Analyze the semantic attribute information of two dynamic nodes to determine whether there is a mutual influence relationship between their target types. The mutual influence relationship is a relationship between similar moving targets or a relationship between motion interference. Based on the interaction between the intersection probability level and the target type, the state influence relationship between two dynamic nodes is determined, and the influence method, parameters and duration range are defined. For two dynamic nodes that have the potential to intersect or are associated by state influence, a dynamic association edge is established between the corresponding nodes in the feature graph. The association edge is labeled with attribute information, including the level of intersection potential, the type of influence association, and the duration of influence. All established dynamic association edges are integrated with their corresponding dynamic nodes to generate a complete dynamic node association network, which reflects the actual motion association and state influence association between dynamic nodes.

7. The UAV global perception method based on four-dimensional imaging and deep learning according to claim 5, characterized in that, The spatial distribution information of static nodes is used to analyze the spatial distance association, occlusion association, and adjacency association between different static nodes, and to establish static association edges between static nodes that are associated, including: Extract the spatial coordinate information and spatial morphology information of each static node. The spatial morphology information includes the outline range, occupied spatial volume and surface structure features of the static target. The spatial coordinate information of all static nodes is converted into coordinate data in a unified three-dimensional spatial coordinate system. The spatial coordinate information of any two static nodes is calculated one by one to obtain the straight-line distance parameter and spatial orientation relationship parameter between the two static nodes. Based on the spatial morphological information and spatial orientation parameters of the two nodes, the overlapping situation of their positions in three-dimensional space is analyzed to determine whether there are areas of mutual occlusion and the proportion of occlusion area. The spatial distance interval is divided according to the straight-line distance parameter, and the occlusion association type between two static nodes is determined by combining the occlusion area ratio. The occlusion association type is complete occlusion association, partial occlusion association, and no occlusion association. Based on spatial orientation parameters and straight-line distance parameters, it is determined whether two static nodes are in adjacent spatial regions, and the adjacent association type is determined. The adjacent association type is direct adjacent association, indirect adjacent association, and non-adjacent association. Integrate spatial distance intervals, occlusion association types, and adjacent association types to generate a set of spatial association features between two static nodes; For two static nodes in the spatial association feature set that have a clear association relationship, a static association edge is established between the corresponding nodes in the feature graph. The clear association relationship is either occlusion association or adjacent association. Attribute information is labeled on static association edges. The attribute information includes spatial distance parameters, occlusion association type, adjacent association type and association region coordinates, so that the static association edges reflect the spatial association details between static nodes. Integrate all static associated edges with their corresponding static nodes to generate a spatial association network between static nodes.

8. The UAV global perception method based on four-dimensional imaging and deep learning according to claim 5, characterized in that, Based on the position coordinate information of the main node and the spatial position information of dynamic and static nodes, the spatial distance relationship, relative motion relationship, and safety impact relationship between the main node and each dynamic and static node are analyzed. Main node association edges are established between the main node and each target node, including: Extract the real-time position coordinates, flight speed, and flight direction information of the main node to reflect the current spatial position and motion status of the UAV; The real-time position coordinates, motion speed and motion direction information of each dynamic node are extracted, and the real-time position coordinates and spatial morphology information of each static node are extracted. Calculate the real-time spatial distance between the main node and each dynamic node, and combine the flight speed and flight direction information of both to calculate the relative speed parameters and relative motion direction parameters; Based on relative velocity parameters and relative motion direction parameters, predict the trajectory of spatial distance change between the main node and the dynamic node within a preset time period, and determine whether there is a potential correlation between distance reduction or collision. Calculate the real-time spatial distance between the main node and each static node, and combine the spatial morphology information of the static nodes to analyze the possibility of spatial overlap between the current flight path of the UAV and the static nodes; Based on real-time spatial distance, distance change trajectory and possible spatial overlap, the safety impact interval between the main node and each target node is divided, and the risk parameters corresponding to different intervals are defined. The safety impact interval is defined as the near-distance safety impact interval, the medium-distance safety impact interval and the long-distance safety impact interval. Based on the safety impact range, determine the safety impact association type between the main node and each target node. For main nodes and target nodes with safety impact association, establish main association edges between corresponding nodes in the feature map. The association edges are labeled with attribute information such as real-time spatial distance, relative motion parameters, safety impact range and risk parameters. For the main associated edges corresponding to dynamic nodes, add dynamic update trigger conditions. When the movement state of the dynamic node changes by a preset magnitude, the attribute information of the associated edges will be automatically updated. For the main associated edges corresponding to static nodes, and in combination with the spatial morphological stability of static nodes, a periodic update process is set up to ensure that the attribute information of the associated edges reflects the latest spatial positional relationship, thereby generating a complete association network between the main node and the target node.

9. The UAV global perception method based on four-dimensional imaging and deep learning according to claim 1, characterized in that, The process involves performing full-domain scene situational estimation and dynamic boundary delineation using the dynamic environment semantic feature map, outputting a full-domain perception and decision-making result covering the distribution of dynamic and static targets in the target area, their motion evolution trends, and the safe flight airspace for UAVs. This includes: The motion trajectory information, semantic attribute information and correlation information of all dynamic nodes are extracted from the dynamic environment semantic feature map and integrated to form a dynamic target situation dataset. The spatial distribution information, semantic attribute information and correlation information of all static nodes are extracted from the dynamic environment semantic feature map and integrated to form a static target situation dataset. Based on the dynamic target situation dataset, the motion evolution parameters of each dynamic target are analyzed by time-series extrapolation algorithm to predict the spatial position change sequence and state change trend of the dynamic target in the subsequent preset time period, and the predicted motion trajectory of the target is obtained. Based on the static target situation dataset, a static spatial topology structure of the target area is constructed, which includes the spatial distribution pattern of static targets and the range of non-flying areas. By overlaying the predicted motion trajectory of a dynamic target with the static spatial topology of the target area, we can identify the spatial overlap between the future position of the dynamic target and the area occupied by the static target. Based on the association edge information between the main node and each target node, and by combining the predicted motion trajectory of the dynamic target, the static spatial topology, and the identified spatial overlapping areas, the safe zone, warning zone, and danger zone of the UAV flight are dynamically divided, and the spatial range and boundary coordinates of the safe zone, warning zone, and danger zone are defined. Based on the division of safe zones, warning zones, and danger zones, and combined with the real-time flight status data of the drone, flight path suggestions and operation adjustment prompts adapted to the current scenario are generated. Integrate dynamic target distribution information, static target distribution information, dynamic target motion evolution trend, safe flight airspace boundary coordinates, flight path suggestions and operation adjustment prompts to generate comprehensive perception decision-making basis data; The overall perception decision-making base data is structured and organized according to dimensions such as target type, spatial location, time series, and safe zone to generate initial decision data containing dynamic target distribution information, static target distribution information, dynamic target motion evolution trend, safe flight airspace boundary coordinates, flight path suggestions and operation adjustment prompts; The initial decision data is formatted according to a preset data structure to generate standardized global perception decision data.

10. A UAV global perception system based on four-dimensional imaging and deep learning, characterized in that, include: processor; A machine-readable storage medium for storing machine-executable instructions of the processor; The processor is configured to execute the UAV global perception method based on four-dimensional imaging and deep learning according to any one of claims 1 to 9 by executing the machine-executable instructions.

Images