Sparse interaction beam prediction method fusing perception and communication

Abstract

The invention discloses a sparse interaction beam prediction method fusing perception and communication. The method comprises the following steps: acquiring and preprocessing multi-modal sensing data; feature extraction is carried out on the preprocessed multi-modal sensing data; introducing a learnable potential vector by adopting a Perceeiver-Transform structure, and carrying out cross attention interaction on the learnable potential vector and the spliced sequence, so as to obtain a multi-modal fusion feature; and mapping the multi-modal fusion features to a millimeter wave beam space through a beam domain projection mechanism, generating probability distribution of the beam space, and outputting a predicted optimal beam according to the probability distribution. According to the method, the learnable potential vectors are adopted, sparse interaction and efficient fusion of the multi-modal fusion features are achieved, the sensing features are mapped to the millimeter wave beam space through a beam domain projection mechanism, the calculation amount is reduced, the robustness and prediction precision are improved, and the method is suitable for large-scale popularization and application. And the method is suitable for millimeter wave communication systems in low-altitude unmanned aerial vehicles, Internet of Vehicles and high-dynamic environments.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of wireless communication, in particular to a sparse interactive beam prediction method combining perception and communication. BACKGROUND

[0002] In the UAV low-altitude communication network, especially in the UAV-to-UAV (U2U) communication in the millimeter wave frequency band, the system faces complex dynamic factors such as high-speed movement, attitude change and environmental obstruction, and the rapid prediction and alignment of the beam become the core link to ensure the reliability of communication. Because the millimeter wave signal has strong directivity and is extremely sensitive to beam error, the traditional feedback-based beam training method often brings a large amount of communication overhead and training delay, which is difficult to meet the real-time communication needs of the dynamic airspace.

[0003] With the development of multi-modal perception technology, researchers have begun to try to use heterogeneous perception data such as vision, radar, LiDAR, GPS, etc. to assist beam prediction, aiming to realize the forward reasoning of the beam through environmental perception information, thereby reducing the overhead of traditional beam training. The introduction of multi-modal perception provides a new idea for UAV communication networks, but also brings new technical challenges, especially in the efficient fusion of perception data and the effective docking of beam prediction tasks, there is still a lack of unified and efficient solutions.

[0004] Therefore, it is necessary to propose a scheme to improve one or more problems in the above-mentioned related technical solutions.

[0005] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The purpose of the present disclosure is to provide a sparse interactive beam prediction method combining perception and communication, thereby at least partially overcoming one or more problems caused by the limitations and defects of the related art.

[0007] A sparse interactive beam prediction method combining perception and communication according to an embodiment of the present disclosure comprises the following steps: Collecting multi-modal perception data of the UAV, the multi-modal perception data comprising vision image data, LiDAR point cloud data, millimeter wave radar echo data and GPS position information data; Preprocessing the multi-modal perception data, wherein the dynamic time warping method is used to align different modal time series, the low sampling rate modal data is resampled by cubic spline interpolation, and the sampling frequencies of each modal are unified; The pre-processed multi-modal perception data are respectively subjected to feature extraction, wherein visual features are extracted by a visual encoder, LiDAR features are extracted by a LiDAR encoder, radar time series features are extracted by a radar encoder, and GPS position features are extracted by a GPS processing module; The visual features, LiDAR features, radar time series features and GPS position features are spliced to form a spliced sequence, a Perceiver-Transformer structure is adopted to introduce a learnable latent vector to cross attention interact with the spliced sequence, and multi-modal fusion features are obtained. The multi-modal fusion features are mapped to a millimeter wave beam space through a beam domain projection mechanism to generate a probability distribution of the beam space, and a predicted optimal beam is output according to the probability distribution.

[0008] In an example embodiment of the present application, the multi-modal perception data are pre-processed, wherein the alignment of different modal time series is realized by a dynamic time warping method, low sampling rate modal data are resampled by cubic spline interpolation, and the steps of unifying the sampling frequencies of each modal include: Let the time series of two modalities be and , respectively, wherein represents the feature vector of modality one at the th time step, represents the feature vector of modality two at the th time step, and represent the total number of time steps of modality one and modality two, respectively, a dynamic time warping method is introduced to find the optimal path , and the optimization objective is wherein represents the alignment path, represents a matching point in the path, is the Euclidean norm, and the path needs to satisfy the following constraint conditions: wherein is the total number of steps of the alignment path, and the constraint condition specifies that the movement of the path is in the right, upward or diagonal direction; Cubic spline interpolation is performed on low sampling rate modal data, let the time stamps of adjacent known sampling points be and , and the interpolation signal value of the target time point in the interval is denoted as , and its expression is represents the interpolation value at time , is the interpolation coefficient, and These are the timestamps of adjacent known sampling points. The target time point to be calculated.

[0009] In an exemplary embodiment of this application, the extraction of visual features by the visual encoder specifically includes: The visual image data is processed by a ResNet-18 network to output a feature map. The ASPP module employs multiple parallel dilated convolutions with different dilation rates to capture contextual information at different spatial scales. The feature maps output by the ResNet-18 network are then processed, concatenated by a Concat operation, and processed by 1×1 convolution and batch normalization. The SE attention module performs spatial dimension global average pooling on the feature map processed by the ASPP module to generate channel descriptors. The channel descriptors are then input into two fully connected layers and activated sequentially by ReLU and Sigmoid to generate channel attention weights. The channel attention weights are then multiplied with the feature map processed by the ASPP module to obtain the visual features.

[0010] In an exemplary embodiment of this application, the output of the ASPP module can be formally represented as: in, This represents a 1×1 convolution operation. This represents a 3×3 dilated convolution operation with an inflation rate r; GlobalAvgPool(·) represents global average pooling over the spatial dimension (H,W); This indicates that the pooled features are upsampled to the original spatial size; This indicates that the outputs of the parallel branches are concatenated along the channel dimension, and F represents the input feature map.

[0011] The expression for the channel descriptor is: in, Indicates the first The global average pooling result of each channel is used to measure the average activation intensity of that channel in the spatial dimension; Indicates the input feature map at the th Position in each channel The pixel values, where Indexed in the vertical direction. For horizontal indexing, and These represent the height and width of the feature map, respectively. This represents the total number of channels.

[0012] In an example embodiment of the present application, the extracting LiDAR features by the LiDAR encoder specifically includes: projecting the LiDAR point cloud data into a BEV image, filtering the BEV image to suppress static background noise; inputting the filtered BEV image into a CNN network, the CNN network integrating an ASPP module and an SE attention module, the structure of the ASPP module and the SE attention module being consistent with the ASPP module and the SE attention module of the visual encoder, and outputting the LiDAR features, the spatial structure of the LiDAR features being aligned with the visual features.

[0013] In an example embodiment of the present application, the extracting radar time sequence features by the radar encoder specifically includes: splitting the millimeter wave radar echo data into multiple frames of radar echoes, and extracting single-frame spatial features for each frame of radar echoes by two-dimensional convolution operation; stacking the single-frame spatial features along the time axis to form a radar time sequence feature sequence; projecting the radar time sequence feature sequence into a query matrix Q, a key matrix K, and a value matrix V, and modeling the time dependence of the radar time sequence feature sequence by a multi-head attention mechanism to obtain time-optimized features; mapping the time-optimized features to a preset shared embedding space by linear projection to obtain the radar time sequence features.

[0014] In an example embodiment of the present application, the extracting GPS position features by the GPS processing module specifically includes: obtaining GPS original latitude and longitude data of the main unmanned aerial vehicle and the target unmanned aerial vehicle, and converting the latitude and longitude data into local Cartesian coordinates; calculating the relative offset vector of the main unmanned aerial vehicle and the target unmanned aerial vehicle according to the local Cartesian coordinates of the two; normalizing the relative offset vector using the empirical mean and standard deviation of the training set; inputting the normalized relative offset vector into an MLP, projecting it to a preset shared embedding space by the MLP, and obtaining the GPS position features.

[0015] In an example embodiment of the present application, the visual features, LiDAR features, radar time sequence features, and GPS position features are spliced to form a spliced sequence, and the steps of introducing a learnable latent vector into the spliced sequence by a Perceiver-Transformer structure and performing cross-attention interaction include: The visual features, LiDAR features, radar time-series features, and GPS location features are spliced ​​together into a spliced ​​sequence. Using the learnable latent vector as the query and the concatenated sequence as the key and value, cross-attention interaction is performed to obtain the cross-attention output; The cross-attention output is subjected to Transformer encoder layer self-attention optimization, and then residual connection is performed with the multimodal feature sequence through linear projection to obtain multimodal fusion features.

[0016] In an exemplary embodiment of this application, the step of mapping the multimodal fusion features to a millimeter-wave beam space through a beam domain projection mechanism to generate a probability distribution of the beam space, and outputting the predicted optimal beam based on the probability distribution, includes: Global average pooling is performed on the multimodal fusion features to obtain a one-dimensional compact feature vector; The compact feature vector is then input into a multilayer perceptron classifier. The output layer of the multilayer perceptron classifier uses the Softmax function to generate the probability distribution for each beam direction; The beam direction with the highest probability is selected as the prediction result.

[0017] In an exemplary embodiment of this application, the probability of each beam direction is calculated using the following formula: in, The first in the softmax classifier The probability of each beam direction. This represents the input feature vector; It is the first Weight vectors for each category; It is the corresponding bias term. It is a linear discriminant function. It is an exponential function. The first The weight vectors and biases for each category, For category index variables, It is the total number of predefined beam directions. and They represent A 3D real vector space and a set of real numbers.

[0018] The application provides a sparse interaction beam prediction method fusing perception and communication. On the one hand, special encoders are configured for visual images, LiDAR, millimeter wave radar and GPS multi-modal perception data and preprocessed, so that the features of the heterogeneous data of various modes can be accurately extracted and the dimensional differences are eliminated, thereby laying a high-quality data foundation for subsequent fusion. On the other hand, a Perceiver-Transformer structure is adopted to introduce a learnable latent vector, so that the multi-modal feature fusion process is replaced by sparse attention interaction instead of traditional full connection attention, so that the calculation complexity of the multi-modal feature fusion process is significantly reduced, and the deployment requirements of resource-limited platforms such as unmanned aerial vehicles are adapted. On the other hand, the multi-modal fusion features are mapped to the millimeter wave beam space through a beam domain projection mechanism, so that the multi-modal perception features can be directly connected with the beam selection task, the efficiency and accuracy of beam prediction are improved, and the real-time requirements of low-altitude unmanned aerial vehicle dynamic communication scenarios are met. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application. It is apparent that the accompanying drawings are only some embodiments of the present application, and other drawings can be obtained from the accompanying drawings without creative labor for those skilled in the art.

[0020] Figure 1 A step schematic diagram of the sparse interaction beam prediction method fusing perception and communication in the exemplary embodiment of the application is shown. Figure 2 A network model framework schematic diagram in the exemplary embodiment of the application is shown. Figure 3 A loss function convergence curve schematic diagram in the simulation experiment of the application is shown. Figure 4 A beam prediction accuracy comparison curve schematic diagram in the simulation experiment of the application is shown. DETAILED DESCRIPTION

[0021] Example implementations will now be described more fully with reference to the accompanying drawings. Example implementations, however, can be implemented in many different forms and should not be construed as limited to the examples set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of example implementations to those skilled in the art. The described features, structures, or characteristics can be combined in one or more implementations.

[0022] Furthermore, the accompanying drawings are included to provide a further understanding of the present application, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the present application and, together with the description, serve to explain the principles of the present application. In the drawings:

[0023] The present example embodiment provides a sparse interaction beam prediction method fusing perception and communication, as shown in the following steps: Figures 1-2 Step S101: Collecting unmanned aerial vehicle multi-modal perception data, the multi-modal perception data including visual image data, LiDAR point cloud data, millimeter wave radar echo data and GPS position information data; Step S102: Preprocessing the multi-modal perception data, wherein the alignment of different modal time series is realized by using a dynamic time warping method, the low sampling rate modal data is resampled by using a cubic spline interpolation, and the sampling frequencies of each modal are unified; Step S103: Performing feature extraction on the preprocessed multi-modal perception data, respectively, wherein the visual features are extracted by a visual encoder, the LiDAR features are extracted by a LiDAR encoder, the radar time series features are extracted by a radar encoder, and the GPS position features are extracted by a GPS processing module; Step S104: Splicing the visual features, LiDAR features, radar time series features and GPS position features to form a spliced sequence, introducing a learnable latent vector into the spliced sequence by using a Perceiver-Transformer structure to perform cross-attention interaction, and obtaining multi-modal fusion features; Step S105: Mapping the multi-modal fusion features to a millimeter wave beam space by a beam domain projection mechanism, generating a probability distribution of the beam space, and outputting a predicted optimal beam according to the probability distribution.

[0024] ​The sparse interaction beam prediction method fusing perception and communication proposed in the embodiment of the application, on the one hand, configures a special encoder for visual images, LiDAR, millimeter wave radar, GPS multi-modal perception data and performs preprocessing, so that the features of the heterogeneous data of each mode can be accurately extracted and the dimensional differences are eliminated, laying a high-quality data foundation for subsequent fusion; on the other hand, the Perceiver-Transformer structure is adopted to introduce a learnable latent vector, so that the traditional full-connection attention is replaced by sparse attention interaction, so that the calculation complexity of the multi-modal feature fusion process is significantly reduced, and the deployment requirements of resource-limited platforms such as unmanned aerial vehicles are adapted; on the other hand, the multi-modal fusion features are mapped to the millimeter wave beam space through the beam domain projection mechanism, so that the multi-modal perception features can be directly connected with the beam selection task, improving the efficiency and accuracy of beam prediction and meeting the real-time requirements of low-altitude unmanned aerial vehicle dynamic communication scenarios.

[0025] As shown in the following Figures 1-4 A sparse interaction beam prediction method fusing perception and communication will be described in more detail.

[0026] It should be understood that the existing multi-modal perception assisted beam prediction scheme usually adopts a unified fusion framework based on Transformer, extracts features from multi-modal data such as vision, radar, LiDAR, GPS, etc. through convolution or encoder, directly inputs them into a multi-layer self-attention network, performs full-modal feature interaction, and finally outputs the classification result of the beam category. This method improves the accuracy of beam prediction by combining general perception networks and classification heads, and alleviates the problem of large training overhead of traditional methods.

[0027] However, the existing scheme mostly adopts the standard Transformer architecture, which has significant computational redundancy problems, especially in the multi-modal perception scene, where all feature interactions cause the model parameter size and computational complexity to rapidly expand. In addition, the existing method only models beam prediction as a classification task, without considering the beam domain characteristics of millimeter wave communication, and the perception features and beam space lack direct mapping, resulting in insufficient model generalization ability and inability to fully adapt to complex dynamic airspace actual communication environments.

[0028] Step S101: Collecting multi-modal perception data of the unmanned aerial vehicle, the multi-modal perception data including visual image data, LiDAR point cloud data, millimeter wave radar echo data and GPS position information data.

[0029] Specifically, the visual image data is derived from the forward-looking RGB camera carried by the unmanned aerial vehicle, and the collected content is a real-time environment image sequence (such as city daytime / nighttime scene, aerial target, building occlusion, etc.) during the flight process of the unmanned aerial vehicle.

[0030] LiDAR point cloud data is collected by a LiDAR sensor mounted on a UAV, and the collected content is three-dimensional point cloud data (including spatial coordinates of target objects, reflection intensity, etc.), which can accurately represent the geometric shape of dynamic targets in the air (such as other UAVs) and static obstacles (such as building roofs).

[0031] Millimeter wave radar time series echo data is collected by a millimeter wave radar sensor mounted on a UAV, and the collected content is a series of time series echo frames, forming a three-dimensional tensor in the "range-velocity-time domain" (i.e. each frame of data corresponds to a range-velocity two-dimensional heat map at a specific time, and multiple frames of data are stacked along the time axis to form a time series dimension), which is used to capture the dynamic motion characteristics (such as speed and relative distance change) of targets in the air.

[0032] GPS position data is collected by a high-precision global navigation satellite system (GNSS) module mounted on a UAV, and the collected content is the real-time latitude and longitude and attitude information of the main UAV and the target UAV, which is the core data for calculating the relative spatial position of the two.

[0033] Step S102: Preprocessing the multi-modal perception data, wherein the dynamic time warping method is used to align the time series of different modalities, the low sampling rate modality data is resampled by cubic spline interpolation, and the sampling frequencies of each modality are unified.

[0034] In multi-modal perception data processing, let the time series of two modalities be and , respectively, where represents the feature vector of modality one at the th time step, represents the feature vector of modality two at the th time step, and represent the total number of time steps of modality one and modality two, respectively. To achieve time series alignment, the dynamic time warping (DTW) method is introduced to find the optimal path , and the optimization objective is: where represents the alignment path, which describes the mapping relationship between modality one and modality two in time, represents a matching point in the path, representing the correspondence between the th time step and the th time step, is the Euclidean norm, which is used to measure the difference between feature vectors. The path needs to satisfy the following constraint conditions: where To align the total number of steps of the path, the constraint specifies that the path can only move in the right, up or diagonal direction, thus ensuring the monotonicity and continuity of the time alignment.

[0035] After completing the timing alignment, in order to unify the sampling frequency of different modalities on the time axis, the low sampling rate modality data needs to be interpolated three times. Let the timestamps of adjacent known sampling points be and , in the interval , the interpolated signal value of the target time point is denoted as , and its expression is: wherein represents the interpolation value at time , is the interpolation coefficient, which is determined by ensuring the continuity of the interpolation function and its first and second derivatives, and are the timestamps of adjacent known sampling points to be calculated. The target time point. This method can ensure that the resampled multi-modal data strictly corresponds at each time step, thereby maintaining the consistency of cross-modal timing.

[0036] Step S103: performing feature extraction on the preprocessed multi-modal perception data respectively, wherein visual features are extracted through a visual encoder, LiDAR features are extracted through a LiDAR encoder, radar temporal features are extracted through a radar encoder, and GPS position features are extracted through a GPS processing module.

[0037] In the feature extraction and encoding stage, each modality data is independently processed and time-aligned by a dedicated encoder.

[0038] The visual modality adopts ResNet-18 combined with ASPP module and SE module to obtain image features enhanced by multi-scale context and channel attention, i.e., visual features; The LiDAR modality projects the point cloud into BEV representation and adopts the same CNN (Convolutional Neural Network), ASPP module and SE module as the visual modality to extract sparse spatial features, i.e., LiDAR features; The radar modality models dynamic features through convolution and multi-head attention, and projects them into a shared embedding space to obtain radar temporal features; The GPS modality obtains relative position vectors through coordinate conversion and standardization, and maps them into geometric priors through MLP to obtain GPS position features; The four types of features are kept in the same scale, providing uniform representation for subsequent cross-modal fusion.

[0039] Specifically, the step of extracting visual features through the visual encoder specifically includes steps S3101-S3103.

[0040] The image encoder adopts an enhanced Convolutional Neural Network (CNN) architecture to extract representative features from the captured camera images.

[0041] Step S3101: outputting a feature map of the visual image data through a ResNet-18 network; using a pre-trained 18-Layer residual Network (ResNet-18) model as a backbone network to obtain a preliminary visual representation. On the basis of the feature map generated by the ResNet-18, an Atrous Spatial Pyramid Pooling (ASPP) module is integrated to enhance multi-scale feature extraction.

[0042] Step S3102: processing the feature map output by the ResNet-18 network through the ASPP module using multiple groups of parallel atrous convolutions with different dilation rates to capture context information of different spatial scales, concatenating and processing through 1x1 convolution and batch normalization; The ASPP module uses multiple groups of parallel atrous convolutions to capture context information of different spatial scales by setting different dilation rates (Dilation Rate). For a given input feature map The output of the ASPP module can be formally represented as: wherein, represents a 1x1 convolution operation, represents a 3x3 atrous convolution operation with dilation rate r; GlobalAvgPool(·) represents global average pooling on the spatial dimensions (H, W); represents upsampling the pooled features to the original spatial dimensions; represents concatenating the outputs of the parallel branches in the channel dimension, and F represents the input feature map obtained from the previous convolutional neural network.

[0043] After the above operations, the combined feature map Further processing is performed through a 1x1 convolution and batch normalization (Batch Normalization) to generate a feature representation with context enhancement. The encoder's perceptual ability for targets with different scale ranges and geometric complexity is improved.

[0044] Step S3103: The SE attention module is used to perform spatial dimension global average pooling on the feature map processed by the ASPP module to generate a channel descriptor. The channel descriptor is input into two fully connected layers and sequentially activated by ReLU and Sigmoid to generate a channel attention weight. The channel attention weight is multiplied with the feature map processed by the ASPP module in the channel level to obtain the visual feature.

[0045] To further refine the feature information in the channel dimension, an SE (Squeeze-and-Excitation, attention mechanism) module is introduced to adaptively recalibrate the feature response. The SE module first performs global average pooling in the spatial dimension to generate a channel descriptor: wherein, represents the global average pooling result of the i-th channel, which is used to measure the average activation intensity of the channel in the spatial dimension; represents the pixel value of the input feature map at the i-th channel and position (h, w), wherein h and w are the vertical and horizontal direction indexes, respectively, and H and W represent the height and width of the feature map, respectively, and C is the total number of channels. Next, the channel descriptor vector is input into two fully connected layers, and is activated by ReLU and Sigmoid to finally generate a channel attention weight: wherein, wherein, and are the weight matrices of the two fully connected layers (dimension reduction and dimension expansion are performed with a compression ratio of 1 / 16), represents the weight matrix of the fully connected layer for dimension reduction, which is used to compress the input channel number from C to C / 16; and represents the dimension-reduced channel attention intermediate representation, which retains the main features between channels and is used for subsequent recovery of the original channel dimension, ​​​​​​​​​​​denotes the element-wise Sigmoid function. Finally, the calibrated feature map The original ASPP output features are re-weighted by the learned channel attention weights: wherein, denotes the channel-wise multiplication operation (i.e. each channel of the feature map and the corresponding scalar are multiplied element-wise). This mechanism enables the network to emphasize useful channel features and suppress irrelevant or redundant ones, thus enhancing the discriminative ability of the encoder.

[0046] Finally, the output of the image encoder is a high-dimensional feature map as the visual feature, denoted as wherein, , , denote the number of channels, height and width of the feature map, respectively.

[0047] The LiDAR feature extraction by the LiDAR encoder specifically includes steps S3201-S3202.

[0048] Step S3201: projecting the LiDAR point cloud data into a BEV image, and filtering the BEV image to suppress static background noise; The LiDAR point cloud is first projected into a Bird's-Eye View (BEV) representation, and then input into a CNN for feature encoding. In the unmanned aerial vehicle scene, the BEV projection can consider the aerial perspective and combine with the filtering algorithm to suppress static background noise such as the ground, building roof, and other non-navigation related areas, and focus on the air space area that can be passed and other dynamic aerial targets.

[0049] Step S3202: inputting the filtered BEV image into a CNN network, wherein the CNN network integrates a dilated spatial ASPP module and an SE attention module, the structures of the ASPP module and the SE attention module are consistent with those of the ASPP module and the SE attention module of the visual encoder, and the LiDAR feature is output, and the spatial structure of the LiDAR feature is aligned with that of the visual feature.

[0050] The preprocessed BEV image is input into a CNN backbone network, and the process is similar to that of the image encoder, while an ASPP module is introduced to extract multi-scale spatial features. Then, an SE attention module is used to adaptively calibrate the channel-level feature response, and the reflection features of dynamic targets are emphasized. The ASPP and SE modules used here are consistent with the structures in the visual encoder, ensuring consistency in multi-scale feature extraction and channel calibration during the multi-modal encoding process.

[0051] Finally, the LiDAR encoder outputs a high-dimensional feature map as LiDAR features, denoted as , whose spatial structure is aligned with image features, facilitating subsequent multi-modal seamless fusion.

[0052] The radar time-series feature extraction by the radar encoder specifically includes steps S3301-S3304.

[0053] The millimeter wave radar sensor mounted on the unmanned aerial vehicle outputs a series of time-series echo frames, which can form a three-dimensional tensor defined in the range-velocity-time domain. In order to effectively model the spatio-temporal characteristics of the aerial sensing data, the radar encoder designs a special feature encoding network.

[0054] Step S3301: split the millimeter wave radar echo data into multiple radar echo frames, and extract single-frame spatial features for each radar echo frame using two-dimensional convolution operation; For each radar echo frame , a two-dimensional convolution operation is used to extract spatial features: wherein, represents the radar heat map of the th frame, with a height and a width ; represents the feature vector extracted from the th frame, with channels, represents the total number of radar frames.

[0055] Step S3302: stack the multiple single-frame spatial features along the time axis to form a radar time-series feature sequence; These frame-by-frame features are then stacked along the time axis to form the radar time-series feature representation: wherein, represents the total number of radar frames, represents the entire radar feature sequence covering all frames in the time dimension.

[0056] Step S3303: project the radar time-series feature sequence into a query matrix Q, a key matrix K, and a value matrix V, and model the time dependence of the radar time-series feature sequence through a multi-head attention mechanism to obtain time-optimized features; In order to model the time dependence between radar frames, a multi-head attention mechanism is used. The feature sequence is projected into a query (Q), a key (K), and a value (V) matrix, in the form of: wherein, denotes the dimension of query and key vectors in each attention head. Subsequently, the radar time-series features based on the temporal attention mechanism are calculated as follows: wherein, denotes the radar feature sequence optimized by the attention mechanism, denotes the output feature dimension corresponding to each time frame after multi-head attention aggregation.

[0057] Step S3304: mapping the time-series optimized features to a preset shared embedding space through linear projection to obtain the radar time-series features.

[0058] The radar features are projected into a shared embedding space through linear transformation to achieve alignment with other modal features. The process is defined as follows: wherein, and are the learnable weight and bias parameters of the projection layer, respectively. After projection, the obtained radar feature embedding is consistent in time with other modalities and aligned in dimension with multi-modal features, facilitating subsequent fusion operations. In this way, the radar features are mapped to a feature space consistent with other modal features, thereby promoting joint learning and fusion of multi-modalities.

[0059] Extracting GPS position features through the GPS processing module specifically includes steps S3401-S3404.

[0060] In U2U communication, accurate relative positioning between aerial platforms is crucial for achieving rapid beam alignment. Typically, both the master drone and the target drone are equipped with high-precision Global Navigation Satellite System (GNSS) modules. In order to effectively extract the spatial geometric relationship between the two drones, the original GPS latitude and longitude data need to be processed as follows.

[0061] Step S3401: obtaining the GPS original latitude and longitude data of the master drone and the target drone, and converting the latitude and longitude data into local Cartesian coordinates.

[0062] Converting geographic coordinate system data into local Cartesian coordinates . Assuming that the Earth is a sphere with a radius of , the conversion relationship is as follows: wherein, and denote latitude and longitude in radian, respectively, denotes the constant radius of the earth.

[0063] Step S3402: According to the local Cartesian coordinates of the main UAV and the target UAV, the relative offset vector between the two is calculated.

[0064] The main UAV (ego UAV) calculates the relative offset vector between itself and the target UAV according to its own position and the position of the target UAV , as follows: Step S3403: The empirical mean and standard deviation of the training set are used to normalize the relative offset vector.

[0065] In view of the scale change of the relative motion between different sampling frames, in order to improve the stability of the training, the relative offset vector is normalized, and the mathematical expression is as follows: First, the mean and standard deviation are used to normalize the relative offset vector wherein, denotes the normalized position information vector, denotes the original relative offset of the target UAV and the main UAV in the two-dimensional Cartesian coordinate system. denotes the empirical mean calculated in the training set.

[0066] wherein, denotes the standard deviation, and the calculation formula is as follows: In the formula, denotes the expected operation on all training samples. The parameters and are obtained through offline calculation before training, ensuring the stability and consistency of the normalization operation in the training process. This normalization step effectively alleviates the problem of too large dynamic range in the relative position information, enabling the model to learn more robust geometric representations.

[0067] Step S3404: The normalized relative offset vector is input into the MLP, and is projected into the preset shared embedding space through the MLP, to obtain the GPS position feature.

[0068] ​Normalized relative position vector are treated as feature vectors and input into a dedicated positional encoding branch. To align with the feature dimensionality of other modalities, a Multilayer Perceptron (MLP) is adopted to project them into a unified embedding space. This positional embedding is then integrated into the multi-modal fusion network to provide spatial prior information for beam prediction.

[0069] Step S104: The visual features, LiDAR features, radar time-series features, and GPS position features are spliced to form a spliced sequence. A Perceiver-Transformer structure is used to introduce a learnable latent vector to cross-attention interact with the spliced sequence to obtain multi-modal fusion features. This includes steps S4001-S4003.

[0070] Step S4001: The visual features, LiDAR features, radar time-series features, and GPS position features are spliced into a spliced sequence.

[0071] The visual features and LiDAR features are first expanded into sequences and . Then, these two features are spliced with the radar frame-by-frame feature sequence and the GPS feature vector to form a unified input sequence .

[0072] Step S4002: Cross-attention interaction is performed with the learnable latent vector as the query and the spliced sequence as the key and value to obtain cross-attention output.

[0073] The self-attention mechanism of the Transformer can calculate the pairwise similarity between all elements in the input sequence, thereby achieving adaptive fusion of multi-modal information in a global range. Let the query, key, and value matrices be The formula for single-head attention operation is as follows: where denotes the dimension of the query and key vectors in each attention head, and the Softmax function is used to normalize the attention weights. By using a multi-head attention mechanism, the network can simultaneously utilize multiple attention heads to extract modality-specific correlations from different subspaces, thereby further enhancing the representation ability of cross-modal features. The multi-head attention mechanism enables the model to focus on important corresponding relationships between modalities from multiple perspectives, significantly improving the expressiveness and robustness of the fusion process.

[0074] The PerceiverIO structure is adopted as the core of the fusion module. PerceiverIO introduces a set of learnable latent vectors that interact with input features as queries. In the cross-attention layer, the latent vectors form a query matrix while the input sequence provides keys and values . Through the interaction of latent vectors with multi-modal inputs, a unified cross-modal representation is constructed. Through the cross-modal attention mechanism, multiple latent vectors simultaneously focus on different parts of the multi-modal input, enabling the network to capture comprehensive joint representations across modalities. After the cross-modal attention stage, the latent vectors are processed through multiple Transformer encoder layers with self-attention, further refining the latent space and extracting deeper feature associations.

[0075] Step S4003: Perform Transformer encoder layer self-attention optimization on the cross-attention output, and then perform residual connection with the multi-modal feature sequence through linear projection to obtain multi-modal fusion features.

[0076] Project the optimized latent representation back to the original feature space through linear mapping, and perform residual connection with the input sequence to obtain the final fusion output . The overall fusion process can be formally represented as follows: These variables include represent the initial set of latent vectors; represent the latent representation after self-attention optimization; is the linear projection matrix. The residual connection mechanism ensures the preservation of input information while improving the stability of the model training process. The fusion features integrate key information from all modalities as input for the subsequent beam prediction module.

[0077] Step S105: Map the multi-modal fusion features to the millimeter wave beam space through the beam domain projection mechanism to generate a probability distribution in the beam space, and output the predicted optimal beam according to the probability distribution. Including steps S5001-S5004.

[0078] Step S5001: Perform global average pooling on the multi-modal fusion features to obtain a one-dimensional compact feature vector.

[0079] After obtaining the fusion features Next, an MLP was designed for beam prediction. Perform global pooling to obtain a compact feature vector. .

[0080] Step S5002: Input the compact feature vector into the multilayer perceptron classifier.

[0081] This vector is then fed into a multi-layer fully connected network, and the output layer uses the Softmax function to generate a probability distribution for the beam direction. , It is the total number of predefined beam directions.

[0082] Step S5003: The output layer of the multilayer perceptron classifier uses the Softmax function to generate the probability distribution for each beam direction.

[0083] For each candidate beam direction, its probability of being selected is calculated using a Softmax classifier, for the _th ... The prediction probability of each beam is defined as follows: This formula defines the first step in the softmax classifier. Predicted probabilities for each category .in, This represents the input feature vector, extracted and contained by the neural network. One real-valued feature; It is the first Weight vectors for each category; These are the corresponding bias terms. Together they form the linear discriminant function. This reflects the raw score of the category, calculated using an exponential function. Mapped to positive Value. Denominator term It is for all Scores for each category were aggregated. One, where each is the first The weight vectors and biases for each category, This is a category index variable, representing the total number of predefined beam directions. Normalization ensures that all... The sum equals 1, thus forming an effective probability distribution. and They represent A dimensional real vector space and a set of real numbers are used to indicate the data type of features and parameters.

[0084] Step S5004: Select the beam direction with the highest probability as the prediction result.

[0085] Output a normalized probability distribution for all beam directions. Predicted beam number The class selected as the most probable: This mechanism realizes the direct mapping of the fusion feature representation and the beam index space, thereby realizing fast and accurate beam selection.

[0086] It needs to be understood that in the multi-modal data preprocessing stage, the system receives raw sensing data from vision (RGB image), LiDAR, millimeter wave radar (Radar), and GPS. All data are first subjected to normalization and standardization preprocessing, unifying the data scale and eliminating the dimensional differences between modalities, in preparation for subsequent feature extraction and fusion. In the hierarchical feature extraction and sparse fusion stage, the entire model adopts a multi-level feature extraction and fusion mechanism, and a three-layer (Layer-1, Layer-2, Layer-3) phased processing procedure is designed: Layer-1: Initial perception feature extraction, each modality data is extracted by a dedicated encoder to extract bottom-level features. Images and LiDAR use a shared structure of ResNet-18, ASPP (Atrous Spatial Pyramid Pooling) module and compression and SE (excitation attention mechanism) to obtain multi-scale spatial features; millimeter wave radar is extracted by time convolution and multi-head attention to extract dynamic time sequence features; the normalized GPS position information is input into the MLP to map to the embedding space.

[0087] Layer-2: Middle-level feature fusion and sparse interaction, the multi-modal sparse interaction network is input into the multi-modal sparse interaction network, through sparse attention mechanism and interpolation operation, to eliminate redundant features, retain cross-modal key associations, and reduce computational complexity. This layer realizes the middle-scale complementarity of modality information through cross-modal perception alignment mechanism.

[0088] Layer-3: Deep feature interaction and high-level expression, the multi-modal sparse interaction network is continued in the deep layer, combined with cross-modal relationship and global attention mechanism, to further extract high-order feature associations, and perform per-modality feature residual connection to enhance the robustness and information retention of the model.

[0089] Feature splicing and beam classification decision stage, the high-dimensional features output by all layers are subjected to unified interpolation and alignment, spliced into a fusion feature vector. This vector is input into the MLP classifier, which outputs the probability distribution of the candidate beam. The classifier uses the Softmax function to generate Top-1 and Top-3 beam prediction results, and finally selects the beam direction with the highest probability as the output.

[0090] In the output stage, the system directly outputs the "predicted beam", realizes an end-to-end closed-loop process from multi-modal perception to millimeter wave beam selection, does not need traditional feedback training, realizes fast, efficient and low-power beam prediction, and is suitable for dynamic unmanned aerial vehicle network environment.

[0091] In order to verify the effectiveness of the technical scheme of the present application, a large number of experiments were carried out based on simulation data sets in the experimental environment and data sets. The data set is derived from the unmanned aerial vehicle U2U (unmanned aerial vehicle to unmanned aerial vehicle) millimeter wave communication simulation platform, covering a variety of typical flight scenarios, including daytime flight, night flight and different trajectory modes in urban environment. In each scenario, the multi-modal synchronous data of the master and slave unmanned aerial vehicles is collected: the vision mode provides the front-view RGB camera image sequence, the point cloud mode provides the laser radar scanning data, the radar mode provides the millimeter wave radar altimeter reading, and the GPS mode provides the high-precision position and attitude information. In the simulation data set, the two unmanned aerial vehicles are equipped with an antenna beamforming codebook containing 64 beams, and the communication frequency band works in the millimeter wave frequency band. The optimal beam number of the communication link is calculated offline for each frame of data as the label for training and testing. The entire data set covers a large number of samples under all-weather, different bit rate requirements and channel conditions, providing sufficient data support for model training. In the training process, the AdamW optimizer is used with a learning rate of 1×10^(-3) starting and cosine annealing adjustment with the training process, and the exponential moving average (EMA) is used to improve the training stability. Under the unified software and hardware environment (Ubuntu system, NVIDIA A6000 GPU), the model converges after 20 epochs of training. The experiment also designs multiple comparison and ablation experiments, including different feature extraction depths, different fusion module layers, and single modal input comparison, to comprehensively evaluate the performance advantages of the method of the present application.

[0092] In terms of results, in order to verify the performance advantages and training convergence of the method of the present application, multiple comparison simulation experiments were carried out on the model under the unified experimental environment, focusing on the influence of different network layers on the model training stability and beam prediction accuracy.

[0093] a) loss function convergence analysis, Figure 3The cross-entropy loss of the training set and the validation set with the number of iteration rounds is shown under different model depths. The convergence performance of 2-layer, 3-layer and 4-layer network structures is tested respectively. It can be observed that the loss function of all models converges rapidly within 20 epochs, indicating that the training process is stable. When the depth of the multi-modal perception network is small (such as 2 layers), the convergence speed of the model is faster, but there is a slight shock in the loss of the validation set in the later training, and there is a certain risk of underfitting. The 3-layer network shows the best convergence balance on the training set and the validation set, and the loss curve of the validation set tends to be stable, and has good generalization ability. Although the 4-layer network has a lower training set loss, the validation set loss has a tendency of overfitting, and the model complexity is higher.

[0094] b) beam prediction accuracy comparison, Figure 4 The accuracy curves of different models in the beam prediction task are shown, and the Top-1 and Top-3 accuracy indicators are plotted respectively, reflecting the actual inference performance of the model. The experimental results show that in terms of Top-3 accuracy, the method of the present application performs best under the 3-layer model setting, and the accuracy after training can reach more than 97%, which is significantly improved compared with the 2-layer model; in terms of Top-1 accuracy, the 3-layer model is still better than the 2-layer and 4-layer models, indicating that the structure also has strong discrimination ability in the single beam prediction scene. The accuracy of the 2-layer model is low, indicating that the model capacity is insufficient; while the 4-layer model is close in Top-3 index, but there is a certain overfitting, and the training stability is slightly worse. Through the experiment, it is known that the 3-layer fusion structure realizes the optimal balance in accuracy and training complexity.

[0095] In addition, the terms "first", "second", "third", etc. are used only for descriptive purposes and are not to be construed as indicating or implying relative importance or an ordered ranking of the indicated technical features. Thus, a feature defined with "first", "second" or "third" can include, explicitly or implicitly, one or more of the features. In the description of the embodiments of the present application, the term "a plurality" means two or more, unless otherwise expressly specified.

[0096] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms is not necessarily for the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined and combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in the present application.

[0097] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

Claims

1. A sparse interactive beam prediction method integrating sensing and communication, characterized in that, Includes the following steps: Collect multimodal perception data from the UAV, including visual image data, LiDAR point cloud data, millimeter-wave radar echo data, and GPS location information data; The multimodal sensing data is preprocessed, wherein dynamic time warping is used to align the time series of different modalities, and cubic spline interpolation is used to resample the low sampling rate modal data to unify the sampling frequency of each modality. Feature extraction is performed on the preprocessed multimodal sensing data. Specifically, visual features are extracted by a visual encoder, LiDAR features are extracted by a LiDAR encoder, radar time-series features are extracted by a radar encoder, and GPS location features are extracted by a GPS processing module. The visual features, LiDAR features, radar temporal features, and GPS location features are concatenated to form a concatenated sequence. A Perceiver-Transformer structure is used to introduce learnable latent vectors and interact with the concatenated sequence through cross-attention to obtain multimodal fusion features. The multimodal fusion features are mapped to the millimeter-wave beam space through a beam domain projection mechanism to generate a probability distribution of the beam space, and the predicted optimal beam is output based on the probability distribution.

2. The sparse interactive beam prediction method integrating sensing and communication according to claim 1, characterized in that, The preprocessing of the multimodal sensing data includes the following steps: aligning different modal time series using dynamic time warping; resampling low-sampling-rate modal data using cubic spline interpolation; and unifying the sampling frequencies of each modality. Let the time series of the two modes be respectively and ,in Mode 1 indicates the first time. Feature vectors at each time step Mode 2 indicates the second mode in the first... Feature vectors at each time step and Let these represent the total number of time steps for mode 1 and mode 2, respectively. A dynamic time warping method is introduced to find the optimal path. Its optimization objective is in, Indicates the alignment path. This represents a matching point in the path. For a path to be Euclidean norm, it must satisfy the following constraints: in, To align the total number of steps on the path, the constraints specify that the path can move in a right, up, or diagonal direction. Perform cubic spline interpolation on low-sampling-rate modal data, assuming the timestamps of adjacent known sampling points are... and In the interval Within, for the target time point The interpolated signal value is denoted as Its expression is Indicates time interpolated values, These are the interpolation coefficients. and These are the timestamps of adjacent known sampling points. The target time point to be calculated.

3. The sparse interactive beam prediction method integrating sensing and communication according to claim 1, characterized in that, The extraction of visual features via a visual encoder specifically includes: The visual image data is processed by a ResNet-18 network to output a feature map. The ASPP module employs multiple parallel dilated convolutions with different dilation rates to capture contextual information at different spatial scales. The feature maps output by the ResNet-18 network are then processed, concatenated by a Concat operation, and processed by 1×1 convolution and batch normalization. The SE attention module performs spatial dimension global average pooling on the feature map processed by the ASPP module to generate channel descriptors. The channel descriptors are then input into two fully connected layers and activated sequentially by ReLU and Sigmoid to generate channel attention weights. The channel attention weights are then multiplied with the feature map processed by the ASPP module to obtain the visual features.

4. The sparse interactive beam prediction method integrating sensing and communication according to claim 3, characterized in that, The output of the ASPP module can be formally represented as: in, This represents a 1×1 convolution operation. This represents a 3×3 dilated convolution operation with an inflation rate r; GlobalAvgPool(·) represents global average pooling over the spatial dimension (H,W); This indicates that the pooled features are upsampled to the original spatial size; This indicates that the outputs of the parallel branches are concatenated along the channel dimension, and F represents the input feature map. The expression for the channel descriptor is: in, Indicates the first The global average pooling result of each channel is used to measure the average activation intensity of that channel in the spatial dimension; Indicates the input feature map at the th Position in each channel The pixel values, where Indexed in the vertical direction. For horizontal indexing, and These represent the height and width of the feature map, respectively. This represents the total number of channels.

5. The sparse interactive beam prediction method for fusion sensing and communication according to claim 4, characterized in that, The extraction of LiDAR features via the LiDAR encoder specifically includes: The LiDAR point cloud data is projected into a BEV image, and the BEV image is filtered to suppress static background noise. The filtered BEV image is input into a CNN network, which integrates a holed spatial ASPP module and a SE attention module. The structure of the ASPP module and the SE attention module is consistent with that of the visual encoder. The network outputs the LiDAR features, and the spatial structure of the LiDAR features is aligned with the visual features.

6. The sparse interactive beam prediction method for fusion sensing and communication according to claim 1, characterized in that, The extraction of radar time-series features via a radar encoder specifically includes: The millimeter-wave radar echo data is split into multiple radar echo frames, and a two-dimensional convolution operation is used to extract the spatial features of each radar echo frame. The spatial features of a single frame from multiple frames are stacked along the time axis to form a radar temporal feature sequence. The radar time-series feature sequence is projected into a query matrix Q, a key matrix K, and a value matrix V. The time-dependent modeling of the radar time-series feature sequence is performed through a multi-head attention mechanism to obtain time-optimized features. The time-series optimized features are mapped to a preset shared embedding space through linear projection to obtain the radar time-series features.

7. The sparse interactive beam prediction method integrating sensing and communication according to claim 1, characterized in that, The extraction of GPS location features via the GPS processing module specifically includes: Obtain the raw GPS latitude and longitude data of the main UAV and the target UAV, and convert the latitude and longitude data into local Cartesian coordinates; Calculate the relative offset vector between the main UAV and the target UAV based on their local Cartesian coordinates. The relative offset vector is normalized using the empirical mean and standard deviation of the training set; The normalized relative offset vector is input into the MLP and projected onto a preset shared embedding space to obtain the GPS location features.

8. The sparse interactive beam prediction method for fusion sensing and communication according to claim 1, characterized in that, The steps of concatenating the visual features, LiDAR features, radar temporal features, and GPS location features to form a concatenated sequence, and then using a Perceiver-Transformer structure to introduce learnable latent vectors to interact with the concatenated sequence through cross-attention to obtain multimodal fusion features include: The visual features, LiDAR features, radar time-series features, and GPS location features are spliced ​​together into a spliced ​​sequence. Using the learnable latent vector as the query and the concatenated sequence as the key and value, cross-attention interaction is performed to obtain the cross-attention output; The cross-attention output is subjected to Transformer encoder layer self-attention optimization, and then residual connection is performed with the multimodal feature sequence through linear projection to obtain multimodal fusion features.

9. The sparse interactive beam prediction method for fusion sensing and communication according to claim 1, characterized in that, The steps of mapping the multimodal fusion features to the millimeter-wave beam space through a beam domain projection mechanism to generate a probability distribution of the beam space, and outputting the predicted optimal beam based on the probability distribution, include: Global average pooling is performed on the multimodal fusion features to obtain a one-dimensional compact feature vector; The compact feature vector is then input into a multilayer perceptron classifier. The output layer of the multilayer perceptron classifier uses the Softmax function to generate the probability distribution for each beam direction; The beam direction with the highest probability is selected as the prediction result.

10. The sparse interactive beam prediction method for fusion sensing and communication according to claim 9, characterized in that, The formula for calculating the probability of each beam direction is: in, The first in the softmax classifier The probability of each beam direction. This represents the input feature vector; It is the first Weight vectors for each category; It is the corresponding bias term. It is a linear discriminant function. It is an exponential function. The first The weight vectors and biases for each category, For category index variables, It is the total number of predefined beam directions. and They represent A 3D real vector space and a set of real numbers.

Images