Perceptual auxiliary beam prediction method based on multi-modal space-time fusion

Abstract

The invention discloses a perception auxiliary beam prediction method based on multi-modal space-time fusion, and the method comprises the steps: carrying out the millimeter wave communication beam forming prediction through employing an RGB image, a laser radar point cloud and a millimeter wave radar signal as multi-modal perception input, and integrating the complementary information of multi-modal perception in the beam forming prediction through cross-modal space fusion. Dynamic perception of historical time sequence information on a communication target is extracted through time sequence fusion coding, and accurate beam forming prediction of fast moving targets such as vehicles can be achieved. According to the cross-modal fusion and time sequence fusion part, a sequence coding technology based on a selection state space is adopted to replace a sequence coding technology based on an attention mechanism, and during long sequence coding of modal space fusion and time sequence fusion, the calculation overhead is smaller, so that the high accuracy of beam forming prediction is ensured, and meanwhile, the time sequence coding efficiency is improved. And the model training and reasoning speed is accelerated.

Description

Technical Field

[0001] This invention relates to the field of communication beam prediction, and more particularly to a sensing-assisted beam prediction method based on multimodal spatiotemporal fusion. Background Technology

[0002] In millimeter-wave communication, to overcome the problems of high signal propagation loss and small coverage area in the high-frequency band, communication systems rely on highly directional narrow beams for data transmission. Traditional methods for determining the optimal beam in an antenna array involve testing the echo signals of candidate beams one by one to find the optimal beamforming path, which is typically time-consuming and requires frequent beam switching for moving targets, leading to communication delays. Sensing-assisted communication methods, however, can directly predict the communication beam by identifying the user's location and motion status using sensors, enabling a much faster response to moving targets.

[0003] Multimodal fusion sensing technology can leverage the complementary advantages of heterogeneous modes, helping sensing-assisted beam prediction cope with more complex weather and environmental contexts. However, multimodal inputs increase the computational burden of beam prediction, making it difficult to achieve a balance between beam prediction accuracy and inference efficiency. To address this challenge, designing efficient multimodal fusion mechanisms in sensing-assisted beam prediction is crucial. Summary of the Invention

[0004] This invention proposes a multimodal spatiotemporal fusion sensing method to assist in communication beam prediction based on multimodal sensing data. The method includes preprocessing and preliminary feature extraction of sensing data, a cross-modal spatial fusion module, and a temporal fusion module. It employs low-computational-complexity sequence coding based on selected state space to achieve fast and accurate communication beam prediction.

[0005] The objective of this invention is achieved through the following technical solution: a sensing-assisted beam prediction method based on multimodal spatiotemporal fusion, comprising the following steps:

[0006] Acquire multimodal sensing data, including RGB images of the communication path, LiDAR point clouds, and millimeter-wave signals;

[0007] Data preprocessing and preliminary feature extraction are performed on the sensing data of different modalities;

[0008] Within the same time frame, feature sequences of heterogeneous modalities are fused across modal spaces: channel switching mechanism is used to enhance cross-modal interaction, bidirectional rotation sequence coding based on selected state space is adopted, and multimodal complementary information is fused.

[0009] Temporal fusion is performed on the feature sequences after cross-modal fusion: dynamic correlation between time steps is established by using sequence encoding, and a pooling-based attention mechanism is used to perform weighted summation of temporal features to obtain the temporal fusion result;

[0010] Using a beam prediction classifier and inputting spatiotemporal fusion features, the optimal communication beam between the base station and the user is predicted in the next time step.

[0011] Furthermore, the multimodal sensing data specifically includes:

[0012] Millimeter-wave communication base stations acquire multimodal sensing data related to transmission through cameras, lidar, and millimeter-wave radar, including RGB images, lidar point clouds, and millimeter-wave signals.

[0013] Furthermore, the specific steps of preprocessing the sensing data of different modalities and extracting preliminary features are as follows:

[0014] For RGB images, a convolutional network is used to encode the RGB image in the front view coordinates;

[0015] For the point cloud data acquired by LiDAR, it is first projected onto the bird's-eye view and structured into a two-dimensional grid. In each grid cell, the maximum height, intensity and density of the points in that cell are used as feature attributes. Then, a convolutional network is used for preliminary feature extraction.

[0016] For radar signals, Fourier transform is used to convert them into range-Doppler maps, and then a convolutional network is used for preliminary feature extraction.

[0017] By extracting features through convolutional networks, feature maps of each modality of sensing data at different scales are obtained. Shallow, large-scale feature maps contain spatial detail information, while deep, small-scale feature maps contain semantic information.

[0018] Furthermore, the cross-modal space fusion specifically refers to:

[0019] Different perceptual features that have been initially encoded at the same scale are concatenated through channels to obtain multimodal features with the same feature size as the perceptual features. The multimodal features are then stretched into a feature sequence form. Then, channel swapping is performed on each feature node in the feature sequence. The multimodal feature sequence is then normalized, linearly mapped, and activated by an activation function in the feature channel dimension to perform channel encoding, so that the multimodal features are fused in the channels. The fused multimodal feature sequence is then bidirectionally rotated using sequence encoding based on selected state space.

[0020] Furthermore, the adoption of bidirectional rotation sequence encoding based on the selected state space specifically involves:

[0021] The order of the feature sequence is rearranged based on the spatial coordinates of the feature nodes in the two-dimensional feature map; in forward rotation encoding, the order is based on the spatial coordinates of the feature nodes in the feature map. Starting from the position, traverse all nodes clockwise from the outside to the inside and add the corresponding position code to obtain the forward rotation sequence. Perform one sequence encoding to obtain the forward encoded feature. In the reverse rotation encoding, the sequence is reversed based on the forward rotation sequence and a second sequence encoding is performed to obtain the reverse encoded feature.

[0022] The forward and backward encoded features are compressed through a linear layer to reduce the channel dimension to one dimension. The channel dimension is then converted into weights using the ReLU activation function. Finally, the forward and backward encoded features are aggregated through their own information density weights to achieve spatial fusion of multimodal features.

[0023] Furthermore, the establishment of dynamic correlations between time steps using sequence encoding specifically involves:

[0024] The feature sets from multiple time steps are concatenated end-to-end in chronological order to obtain a long sequence. This long sequence is then simultaneously input into an encoding path and a gating path. The encoding path contains a linear layer, a one-dimensional convolution, and a selected state space encoding module. The gating path contains a linear layer and a Silu activation function. The outputs of the two paths are multiplied to obtain the temporal coding result.

[0025] Furthermore, the use of a pooling-based attention mechanism for weighted summation of temporal features specifically includes:

[0026] The temporal coding results are passed through a linear layer, and then the max pooling and average pooling results are calculated separately. The sum of the two results is then passed through another linear layer, and the attention weights are obtained by using the sigmoid activation function. The attention weights are then used to aggregate the temporal coding results of different nodes.

[0027] Furthermore, the cross-modal spatial fusion, temporal fusion, and network in the classifier adopt an end-to-end model training method, and the loss function includes constraints on the predicted intensity of all beams and constraints on the best beam classification result.

[0028] The constraint on the predicted intensity of all beams normalizes the true value of the beam intensity and calculates the loss with the confidence of the predicted output; for the constraint on the best beam classification result, Focal Loss is used to alleviate the imbalance between positive and negative samples in beam classification.

[0029] On the other hand, this application also provides a sensing-assisted beam prediction device based on multimodal spatiotemporal fusion, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the sensing-assisted beam prediction method based on multimodal spatiotemporal fusion.

[0030] On the other hand, this application also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the aforementioned perception-assisted beam prediction method based on multimodal spatiotemporal fusion.

[0031] The beneficial effects of this invention are as follows: Based on multimodal sensing data, this invention proposes a multimodal spatiotemporal fusion sensing method to assist in communication beam prediction. This method employs cross-modal spatial fusion design and temporal fusion design to achieve efficient fusion of complementary features from sensing modes, and establishes a strong correlation between sensing features and the communication beam prediction task. This solves the problem of limited application scenarios and difficulty in coping with complex and changing environments inherent in single-modal sensing-assisted communication, achieving accurate and robust communication beam prediction and providing support for establishing stable millimeter-wave communication paths. Furthermore, this method uses sequence coding based on selected state space as the main coding method, which is more computationally efficient than Transformer-based attention mechanism coding methods. This solves the problem of inference delay caused by high computational complexity during coding, achieving high real-time communication beam prediction and providing support for communication users who need to frequently switch communication beams during rapid movement. Attached Figure Description

[0032] Figure 1 This is a flowchart of the sensing-assisted beam prediction method based on multimodal spatiotemporal fusion implemented in this invention;

[0033] Figure 2 This is a structural diagram of the cross-modal spatial fusion module described in this invention;

[0034] Figure 3 This is a structural diagram of the timing fusion module described in this invention;

[0035] Figure 4 This is a structural diagram of the beam prediction classifier described in this invention;

[0036] Figure 5 This is a schematic diagram of the sensing-assisted beam prediction device based on multimodal spatiotemporal fusion implemented in this invention. Detailed Implementation

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Based on multimodal sensing data, the present invention proposes a multimodal spatiotemporal fusion sensing method to assist in communication beam prediction. This method includes preprocessing and preliminary feature extraction of sensing data, a cross-modal spatial fusion module, and a temporal fusion module, thereby achieving fast and accurate communication beam prediction.

[0038] like Figure 1 The flowchart shown below illustrates the sensing-assisted beam prediction method based on multimodal spatiotemporal fusion implemented in this invention, which includes the following steps:

[0039] S1. Millimeter-wave communication base stations set up along urban roads need to communicate with pedestrians and vehicles on the road, establishing communication paths based on beam selection. These base stations are equipped with various sensors, including cameras, LiDAR, and millimeter-wave radar, to acquire multimodal sensing data. All sensors are oriented towards the road area, collecting image data, LiDAR point cloud data, and Radar signal data for the same road region.

[0040] S1. Millimeter-wave communication base stations acquire multimodal sensing data, including image data, LiDAR point cloud data, and Radar signal data, through cameras, LiDAR, and millimeter-wave radar.

[0041] S2. Preprocessing and extracting preliminary features from different types of sensory data. For images acquired by the camera, a ResNet34 convolutional network is used to encode the image in the front-view coordinates. For point cloud data acquired by LiDAR, it is first projected onto a bird's-eye view and structured into a two-dimensional grid. In each grid cell, the maximum height, intensity, and point density of points within that cell are used as feature attributes, and then a ResNet18 convolutional network is used for preliminary feature extraction. For radar signals, they are converted into range-Doppler maps using Fourier transform, and then a ResNet18 convolutional network is used for preliminary feature extraction. Through feature extraction by the convolutional network, feature maps of the sensory data at different scales can be obtained. Shallow, large-scale feature maps contain spatial detail information, while deep, small-scale feature maps contain semantic information.

[0042] For the fusion encoding of multimodal features, the core module is selected state-space encoding, which describes the evolution of the dynamic system state over time through a state-space model, including two parts: state equations and output equations, as follows:

[0043]

[0044] in, It is a state variable. It is an input variable. It is an output variable. It is a state matrix. It is a control matrix. It is the output matrix. It is the instruction matrix. Based on the state-space model, a selection mechanism is introduced to transform the control matrix... and output matrix It is dynamically dependent on the input, increasing the flexibility of modeling. It can adjust the state space parameters for different input conditions, has stronger sequence encoding capabilities, and has only linear computational complexity, which is more efficient than the attention encoding of Transformer.

[0045] S3. Within the same time frame, the cross-modal space fusion module is used to perform cross-modal space fusion on the feature sequences of heterogeneous modalities. A channel switching mechanism is adopted to enhance cross-modal interaction, and multimodal complementary information is fused based on bidirectional sequence coding of the selected state space. For example... Figure 2 As shown, perceptual features at the same scale , , Multimodal features are obtained by channel splicing. The feature size is the same as the perceptual feature, and the number of feature channels is the sum of the number of channels of the three perceptual features. Multimodal features The process involves stretching the data to transform it into a feature sequence, and then processing each feature node in the feature sequence. Channel swapping is performed. Specifically, the order of feature channels within a feature node is shuffled and rearranged to achieve cross-modal channel interaction and feature representation diversity. Next, the multimodal feature sequence undergoes channel encoding through normalization, linear mapping, and activation functions along the feature channel dimension, resulting in channel-wise fusion of multimodal features. Then, bidirectional rotational sequence encoding is performed on the multimodal feature sequence using selection-state-space-based sequence encoding. Specifically, the order of the feature sequence is rearranged according to the spatial coordinates of the feature nodes in the two-dimensional feature map; in the forward rotational encoding, the sequence is rotated based on the spatial coordinates of the feature nodes in the feature map. Starting from the given position, traverse all nodes clockwise from the outside in, adding the corresponding position code to obtain a forward rotation sequence. Perform sequence encoding once to get the output. In reverse rotation encoding, the sequence is reversed based on the forward rotation sequence, and then a second sequence encoding is performed to obtain the output. Finally, the bidirectional rotation sequence encoded features are aggregated together through their own information density weights to achieve spatial fusion of multimodal features. Specifically, and Information compression is performed using a linear layer, reducing the channel dimension to one dimension, and then converting it into weights using the ReLU activation function. and Thus, through aggregation calculation, multimodal spatial fusion is obtained. As shown below:

[0046]

[0047] S4. For spatial fusion features across multiple time frames, temporal fusion is performed using a temporal fusion module. Sequence encoding is employed to establish dynamic correlations between time steps, and a weighted summation of temporal features is performed based on a pooling attention mechanism to obtain the temporal fusion result. For example... Figure 3 As shown, for the feature set of N time steps First, they are concatenated in chronological order, end to end, to form a long sequence. Then, a selection state-space sequence encoding pair with a gating mechanism is used. Perform timing coding. Specifically, The code passes through an encoding path containing a linear layer, a one-dimensional convolution, and a selected state-space coding module; it also passes through a gated path containing a linear layer and a Silu activation function. The outputs of the two paths are multiplied to obtain the temporal coding result. This establishes dynamic relationships between time steps. Finally, a pooling-based attention mechanism is employed to perform max pooling and average pooling on the encoded output sequence to calculate attention weights, achieving temporal aggregation. Specifically, First, the sample passes through a linear layer. Then, the results of max pooling and average pooling are calculated separately. The sum of these two results is then passed through another linear layer, and the attention weights are obtained by applying a sigmoid activation function. Pooling-based attention weights give greater weight to the features of important nodes during aggregation, making them more effective for the final beam prediction. Temporal aggregation will... The beamforming prediction features of spatiotemporal fusion attributes are obtained by weighted summation of the features of different nodes. The calculation is as follows:

[0048]

[0049] S5. Finally, using a beam prediction classifier and the beamforming prediction features of spatiotemporal fusion attributes, the optimal communication beam between the base station and the user is predicted for the next moment, ensuring high-quality communication. For example... Figure 4 As shown, the beam prediction classifier consists of a multi-layer sensing network. The input spatiotemporal fusion attribute features include fusion at multiple feature scales, and the prediction output is the confidence score of all set communication beams. For a millimeter-wave communication base station with a 64-antenna array, the beam prediction classifier needs to output a 64-dimensional confidence score and select the channel with the highest confidence score as the optimal communication beam.

[0050] The entire cross-modal space fusion module, temporal fusion module, and classifier network adopt an end-to-end model training approach, and the loss function is shown below:

[0051]

[0052] Specifically, the loss function It contains two parts. This represents a constraint on the predicted beam strength for all beams. The true beam strength is normalized and the loss is calculated by comparing it with the confidence level of the predicted output. This represents the constraint on the optimal beam classification result. Focal Loss is used to alleviate the imbalance between positive and negative samples in beam classification. and This is the loss weight. It is set to... , Table 1 compares the performance of the attention-based method and the method of this invention. The method of this invention has better beam prediction accuracy, fewer model parameters, faster inference speed, and lower computational resource memory usage, demonstrating the advantages of the proposed method.

[0053] Table 1 Performance Comparison of Attention-Based and Two-Stage Generation Methods

[0054] Beam prediction method Beam prediction accuracy Model parameters Reasoning speed Video memory usage Attention-based methods 87.89% 78.42 MB 6.32 fps 61.73 GB Method of the present invention 91.62% 66.91 MB 13.50 fps 41.16 GB

[0055] Corresponding to the aforementioned embodiment of a sensing-assisted beam prediction method based on multimodal spatiotemporal fusion, the present invention also provides an embodiment of a sensing-assisted beam prediction device based on multimodal spatiotemporal fusion.

[0056] See Figure 5 The present invention provides a sensing-assisted beam prediction device based on multimodal spatiotemporal fusion, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a sensing-assisted beam prediction method based on multimodal spatiotemporal fusion as described in the above embodiment.

[0057] The embodiment of the sensing-assisted beam prediction device based on multimodal spatiotemporal fusion provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 5The diagram shown is a hardware structure diagram of any device with data processing capabilities, including the sensing-assisted beam prediction device based on multimodal spatiotemporal fusion provided by this invention. (Except for...) Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0058] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0059] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0060] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements a sensing-assisted beam prediction method based on multimodal spatiotemporal fusion as described in the above embodiments.

[0061] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0062] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the aforementioned sensing-assisted beam prediction method based on multimodal spatiotemporal fusion.

[0063] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0064] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A sensing-assisted beam prediction method based on multimodal spatiotemporal fusion, characterized in that, Includes the following steps: Acquire multimodal sensing data, including RGB images of the communication path, LiDAR point clouds, and millimeter-wave signals; Data preprocessing and preliminary feature extraction are performed on the sensing data of different modalities; Within the same time frame, feature sequences of heterogeneous modalities are fused across modal spaces: channel switching mechanism is used to enhance cross-modal interaction, bidirectional rotation sequence coding based on selected state space is adopted, and multimodal complementary information is fused. Temporal fusion is performed on the feature sequences after cross-modal fusion: dynamic correlation between time steps is established by using sequence encoding, and a pooling-based attention mechanism is used to perform weighted summation of temporal features to obtain the temporal fusion result; Using a beam prediction classifier and inputting spatiotemporal fusion features, the optimal communication beam between the base station and the user is predicted in the next time step.

2. The sensing-assisted beam prediction method based on multimodal spatiotemporal fusion according to claim 1, characterized in that, The multimodal sensing data specifically includes: Millimeter-wave communication base stations acquire multimodal sensing data related to the same area through cameras, lidar, and millimeter-wave radar, including RGB images, lidar point clouds, and millimeter-wave signals.

3. The sensing-assisted beam prediction method based on multimodal spatiotemporal fusion according to claim 1, characterized in that, The specific steps for preprocessing and extracting preliminary features from the sensing data of different modalities are as follows: For RGB images, a convolutional network is used to encode the RGB image in the front view coordinates; For the point cloud data acquired by LiDAR, it is first projected onto the bird's-eye view and structured into a two-dimensional grid. In each grid cell, the maximum height, intensity and density of the points in that cell are used as feature attributes. Then, a convolutional network is used for preliminary feature extraction. For radar signals, Fourier transform is used to convert them into range-Doppler maps, and then a convolutional network is used for preliminary feature extraction. By extracting features through convolutional networks, feature maps of each modality of sensing data at different scales are obtained. Shallow, large-scale feature maps contain spatial detail information, while deep, small-scale feature maps contain semantic information.

4. The sensing-assisted beam prediction method based on multimodal spatiotemporal fusion according to claim 1, characterized in that, The cross-modal space fusion specifically refers to: Different perceptual features that have been initially encoded at the same scale are concatenated through channels to obtain multimodal features with the same feature size as the perceptual features. The multimodal features are then stretched into a feature sequence form. Then, channel swapping is performed on each feature node in the feature sequence. The multimodal feature sequence is then normalized, linearly mapped, and activated by an activation function in the feature channel dimension to perform channel encoding, so that the multimodal features are fused in the channels. The fused multimodal feature sequence is then bidirectionally rotated using sequence encoding based on selected state space.

5. The sensing-assisted beam prediction method based on multimodal spatiotemporal fusion according to claim 1, characterized in that, The specific method of employing bidirectional rotation sequence encoding based on selected state space is as follows: The order of the feature sequence is rearranged based on the spatial coordinates of the feature nodes in the two-dimensional feature map; in forward rotation encoding, the order is based on the spatial coordinates of the feature nodes in the feature map. Starting from the position, traverse all nodes clockwise from the outside to the inside and add the corresponding position code to obtain the forward rotation sequence. Perform a sequence encoding to obtain the forward encoded feature. In reverse rotation coding, the sequence is reversed based on the forward rotation sequence, and secondary sequence coding is performed to obtain the reverse encoded features; The forward and backward encoded features are compressed through a linear layer to reduce the channel dimension to one dimension. The channel dimension is then converted into weights using the ReLU activation function. Finally, the forward and backward encoded features are aggregated through their own information density weights to achieve spatial fusion of multimodal features.

6. The sensing-assisted beam prediction method based on multimodal spatiotemporal fusion according to claim 1, characterized in that, The specific steps for establishing dynamic correlations between time steps using sequence encoding are as follows: The feature sets from multiple time steps are concatenated end-to-end in chronological order to obtain a long sequence. This long sequence is then simultaneously input into an encoding path and a gating path. The encoding path contains a linear layer, a one-dimensional convolution, and a selected state-space encoding module. The gating path contains a linear layer and a Silu activation function. The outputs of the two paths are multiplied to obtain the temporal encoding result. The selected state-space encoding module uses a state-space model to describe the evolution of the dynamic system's state over time.

7. The sensing-assisted beam prediction method based on multimodal spatiotemporal fusion according to claim 1, characterized in that, The specific steps of using a pooling-based attention mechanism for weighted summation of temporal features include: The temporal coding results are passed through a linear layer, and then the max pooling and average pooling results are calculated separately. The sum of the two results is then passed through another linear layer, and the attention weights are obtained by using the sigmoid activation function. The attention weights are then used to aggregate the temporal coding results of different nodes.

8. The sensing-assisted beam prediction method based on multimodal spatiotemporal fusion according to claim 1, characterized in that, The cross-modal fusion, temporal fusion, and network in the classifier adopt an end-to-end model training method, and the loss function includes constraints on the predicted intensity of all beams and constraints on the best beam classification result. The constraint on the predicted intensity of all beams normalizes the true value of the beam intensity and calculates the loss with the confidence of the predicted output; for the constraint on the best beam classification result, Focal Loss is used to alleviate the imbalance between positive and negative samples in beam classification.

9. A sensing-assisted beam prediction device based on multimodal spatiotemporal fusion, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a sensing-assisted beam prediction method based on multimodal spatiotemporal fusion as described in any one of claims 1-8.

10. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a sensing-assisted beam prediction method based on multimodal spatiotemporal fusion as described in any one of claims 1-8.

Images