Multi-mode millimeter wave beam prediction method based on multi-task learning
By employing cross-modal gating fusion and multi-task learning methods, adaptive fusion of multimodal sensing information and correction using physical constraints solve the robustness and accuracy problems of beam prediction in millimeter-wave communication, achieving efficient beam prediction in dynamic and complex environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-26
- Publication Date
- 2026-05-05
AI Technical Summary
Existing millimeter-wave communication methods suffer from limitations in single-mode sensing and lack of physical constraints in dynamic and complex environments, resulting in insufficient robustness and low prediction accuracy in beam prediction, making them difficult to adapt to complex communication scenarios.
A cross-modal gating fusion and multi-task collaborative training strategy is adopted to adaptively fuse multimodal perception information and use physical constraint information to correct prediction results. The main task and auxiliary task are processed collaboratively through a multi-task learning framework to achieve high-precision beam prediction.
It improves the robustness and accuracy of beam prediction, enhances the interpretability of prediction, reduces training overhead and communication latency, and is suitable for dynamic and complex communication scenarios.
Smart Images
Figure CN121984552A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of millimeter-wave communication technology and relates to a multimodal millimeter-wave beam prediction method based on multi-task learning. Background Technology
[0002] Millimeter-wave communication, with its abundant spectrum resources and high transmission rate, has become a key technology for achieving ultra-wideband and low-latency transmission in 5G and 6G mobile communication systems. However, millimeter-wave signals have extremely short wavelengths and are highly sensitive to dynamic environmental changes. Especially in complex communication scenarios involving high-speed mobile terminals and dynamic obstacles, the relative motion between transceivers and random obstructions in the environment can easily cause line-of-sight link interruptions and severe signal attenuation. To overcome this challenge, utilizing environmental perception information such as vision, lidar, and location to assist millimeter-wave beam prediction has become a research hotspot in the field of communications. Existing methods have introduced deep learning algorithms to predict the optimal beam direction by mining the mapping relationship between environmental perception information and wireless communication channel characteristics. This enables rapid beam alignment without the need for cumbersome signaling interactions, providing a highly promising solution for addressing link stability issues in highly dynamic scenarios.
[0003] Despite significant progress in environment-aware-assisted beam prediction methods, they still face considerable challenges in practical deployments in dynamic and complex environments. First, in terms of multimodal fusion, existing solutions, whether sequence modeling or contrastive learning, largely rely on single-modal data. However, single sensors have inherent limitations in specific environments; for example, cameras are prone to failure in low light or inclement weather, while LiDAR, although containing geometric information, lacks semantic texture, resulting in insufficient robustness of the model in complex environments. Second, although some methods attempt to fuse multimodal data, existing fusion strategies mostly employ simple feature concatenation and static weighting, lacking the ability to adaptively adjust the contribution weights of each modality based on dynamic scene changes. When the environment changes dynamically, such as moving from open ground to an occluded area, existing multimodal solutions cannot automatically adjust the modal weights and may even reduce prediction accuracy due to the introduction of noise from failed modes. Finally, most existing prediction models are purely data-driven black-box networks that ignore the physical constraints of wireless signal propagation, such as signal reflection intensity, diffraction loss, and blockage probability. They are difficult to effectively cope with channel mutations and limit the interpretability and generalization ability of the models in actual communication systems.
[0004] Therefore, how to design a beam prediction method that can adaptively fuse multimodal sensing information and effectively incorporate communication physical constraints to achieve high-precision prediction in dynamic and complex scenarios has become a technical challenge that urgently needs to be solved in this field. Summary of the Invention
[0005] In view of this, the purpose of this invention is to provide a multimodal millimeter-wave beam prediction method based on multi-task learning. To address the problems of limited single-modal perception and lack of physical constraints in dynamic communication scenarios, this invention adopts a cross-modal gating fusion and multi-task collaborative training strategy. While achieving adaptive fusion of multimodal features, it uses physical constraint information to correct the prediction results, thereby providing high-precision and robust beam prediction in complex communication environments.
[0006] To achieve the above objectives, the present invention provides the following technical solution:
[0007] A multimodal millimeter-wave beam prediction method based on multi-task learning is applied to a millimeter-wave communication system, which includes a base station and user equipment. The base station is equipped with a camera, a 3D point cloud sensor, and a positioning device, and employs a predefined beamforming codebook. The method includes: Based on the geometric channel model in dynamic communication scenarios, the beam prediction task is modeled as a deep learning optimization task. Multimodal sensing data S={I,P,M} is acquired, where I is image data, P is 3D point cloud data, and M is motion information. The image data, 3D point cloud data, and motion information are preprocessed and features are extracted to obtain visual feature vectors. Point cloud geometric feature vectors and motion feature vectors ; The prediction results of candidate beams are output based on the deep learning model, and the optimal beam index is determined from the beamforming codebook. This also includes: The , , Projecting them separately onto the common feature space, we obtain , , ; Normalized gating weights are calculated based on the projected features using a cross-modal gating fusion network. And based on the gate weights, multimodal fusion features are generated: , Based on the multimodal fusion features Construct a multi-task learning framework for parallel output: Candidate beam probability distribution for beam prediction primary task The blocking probability vector of the blocking prediction auxiliary task Global reflection intensity of the reflection intensity prediction auxiliary task and directional reflection intensity vector ; Based on the blocking probability vector and reflection intensity prediction results, physical constraint correction is performed on the candidate beam probability distribution, first according to the blocking probability threshold. and reflection intensity threshold The candidate beams are filtered to obtain the corrected intermediate probability. Based on the reflection gain coefficient After enhancement, we get: , According to the above The index with the highest probability is selected as the optimal beam index.
[0008] This method is performed according to the following steps: S1, based on the geometric channel model in dynamic communication scenarios, transforms the beam prediction task into a deep learning optimization task. S2, collect multimodal data S={I,P,M}, where I is image data, P is 3D point cloud data, and M is motion information; preprocess the data and extract features to obtain visual feature vectors, point cloud geometric feature vectors, and motion feature vectors; the preprocessing includes extracting regions of interest from the image data, downsampling the 3D point cloud data, and transforming the motion information into spatial coordinates; S3, input the preprocessed multimodal data into the cross-modal gated fusion module; the cross-modal gated fusion module extracts each modal feature, calculates the adaptive weight of each modal feature using a gating mechanism, and performs weighted fusion of each modal feature based on the adaptive weight to obtain multimodal fusion features; S4, Construct a multi-task learning framework, and input the multimodal fusion features into the multi-task learning framework to collaboratively process the main task and auxiliary tasks; wherein, the main task is beam prediction, which is used to output the probability distribution of candidate beams, and the auxiliary tasks include blocking prediction and reflection intensity prediction, which are used to output physical constraint information. S5. Design a physical constraint correction mechanism, which uses the physical constraint information output by the auxiliary task to correct the candidate beam probability distribution output by the main task, and selects the index with the highest probability as the optimal beam index based on the corrected probability distribution.
[0009] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) Improved robustness of beam prediction. Through cross-modal gating fusion mechanism, the contribution of each modal feature is adaptively adjusted according to scene changes, which effectively reduces the impact of single mode failure or noise on the prediction results, making beam prediction more stable in complex dynamic scenes.
[0010] (2) Introduce physical constraints to enhance the interpretability of predictions. By using multi-task learning to predict the blocking probability and reflection intensity, and using them as physical constraints to correct the beam prediction results, the prediction process not only depends on data-driven results, but also conforms to the physical laws of wireless propagation, thereby improving the rationality and interpretability of the prediction results.
[0011] (3) Improve prediction accuracy and reduce training overhead. By physically constraining the probability distribution of candidate beams, the probability of unreasonable beams being selected is reduced, the hit rate of optimal beam prediction is improved, thereby reducing the frequency of beam training and realignment, and reducing communication latency and system overhead.
[0012] (4) Applicable to dynamic and complex communication scenarios. This invention can comprehensively utilize environmental perception information and motion information to jointly model factors such as occlusion changes, multipath reflection and user movement, and is applicable to vehicle communication, industrial wireless and other dynamic and complex millimeter wave communication scenarios.
[0013] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description
[0014] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the dynamic millimeter-wave communication system model described in this invention; Figure 2 This is a schematic diagram of the multimodal millimeter-wave beam prediction neural network model based on multi-task learning proposed in this invention; Figure 3 This is a flowchart of the multimodal millimeter-wave beam prediction process based on multi-task learning in this invention. Detailed Implementation
[0015] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0016] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.
[0017] Please see Figures 1-3 This invention provides a multimodal millimeter-wave beam prediction method based on multi-task learning, applicable to, for example... Figure 1 The dynamic millimeter-wave communication system shown.
[0018] Example 1 provides a multimodal millimeter-wave beam prediction method based on multi-task learning, including steps S1 to S5.
[0019] S1: Based on the geometric channel model in dynamic communication scenarios, the beam prediction task is transformed into a deep learning optimization task.
[0020] S11: Establish a millimeter-wave communication system model under dynamic communication scenarios. The base station is equipped with a camera, LiDAR sensor, and GPS receiver, as well as... A uniform linear array of antennas serves a single mobile user equipment (User Equipment). Each User Equipment is equipped with a single antenna and a GPS receiver. The system employs an orthogonal frequency division multiplexing (OFDM) transmission scheme, and the base station uses a predefined beamforming codebook. .in, This represents the total number of beamforming vectors in the codebook.
[0021] S12: At each time step, the base station uses the beamforming vector from the codebook. Send a signal to the user equipment, the user equipment in the... The received signal on each subcarrier is represented as follows: , in, This is the channel vector for the downlink; For beamforming vectors; To transmit a signal and satisfy , This represents the average transmit power. It is additive noise that follows a complex Gaussian distribution.
[0022] S13: The downlink channel is characterized using a three-dimensional geometric channel model, and the channel vector... Represented as: , in, Indicates the total number of transmission paths. For the first The complex attenuation coefficient of the path, and These are the azimuth and elevation angles of arrival, respectively. and These are the departure azimuth and departure elevation angles, respectively. The array steering vector for base stations and user equipment; for a uniform linear array, its form is determined by the antenna spacing. and signal wavelength Decide.
[0023] S14: Define the objective of the beam prediction task as starting from the codebook. Select an optimal beam vector To maximize the received signal power, the beam selection problem can be expressed as: , in, For the true optimal beam index, This is a modulo operation.
[0024] S15: Constructing Deep Learning Prediction Functions Utilizing multimodal sensing data collected by base stations Predicting the optimal beam index; multimodal sensing data , containing image data 3D point cloud data and sports information Predicted optimal beam index This can be obtained by selecting the category with the highest predicted probability: , in, The first output of the model The probability of each beam index.
[0025] S16: Prediction Function From the set of learnable parameters The parameterization process involves training a deep learning model to maximize the total success probability of correctly predicting the optimal beam; mathematically, this optimization objective is expressed as: , in, For the expected operation, Indicates that given input data and parameters Predicting beam index under the condition With the true optimal beam index Consistent conditional probabilities.
[0026] S2: Collect multimodal data S={I,P,M}, where I is image data, P is 3D point cloud data, and M is motion information; preprocess the data and extract features to obtain visual feature vectors, point cloud geometric feature vectors, and motion feature vectors; the preprocessing includes extracting regions of interest from the image data, downsampling the 3D point cloud data, and transforming the motion information into spatial coordinates.
[0027] S21: Extract the region of interest from the image data.
[0028] The target user and potential occluders in images acquired by the base station are identified using an object detection algorithm; the region of interest is cropped according to the target bounding box and adjusted to a uniform size; the processed image is then input into a deep residual network to extract visual feature vectors. .
[0029] S22: Perform point cloud downsampling on the 3D point cloud data.
[0030] The 3D spatial coordinates of the point cloud are standardized using standard deviation, and voxelized meshes are used for downsampling with reflection intensity normalized. The processed 3D point cloud is then input into a hierarchical point cloud feature extraction network to extract the geometric feature vectors of the point cloud. .
[0031] S23: Perform spatial coordinate transformation on motion information.
[0032] The system acquires user device location data and transforms it from a global geodetic coordinate system to a station-centered horizontal coordinate system with the base station as the origin. It calculates the relative displacement difference and concatenates it with the instantaneous velocity to form a motion vector. This motion vector is then input into a multilayer perceptron to extract high-dimensional motion feature vectors. .
[0033] Based on the above, the preprocessing methods for image data, 3D point cloud data, and motion information are not limited to the specific implementation forms mentioned above.
[0034] For example, the target detection algorithm used for image data preprocessing can be replaced by any target detection or segmentation model that can output the spatial location information of the user or occlusion; the downsampling method used for 3D point cloud data preprocessing can be replaced by random downsampling, farthest point sampling or other point cloud simplification methods; and the spatial coordinate transformation method used for motion information processing can also be adjusted according to the actual deployment scenario.
[0035] When replacing or modifying the above preprocessing method, the supervision labels used to train the multi-task learning model are generated or aligned based on spatial references and geometric relationships consistent with the preprocessing method, so as to ensure that the input features are consistent with the corresponding beam labels, blocking labels and reflection intensity labels, thereby ensuring the effectiveness of the model training and prediction process.
[0036] S3: Input the preprocessed multimodal data into the cross-modal gating fusion module; the cross-modal gating fusion module extracts the features of each modality, calculates the adaptive weights of each modality feature using the gating mechanism, and performs weighted fusion of each modality feature based on the adaptive weights to obtain the multimodal fusion features.
[0037] S31: Feature projection and extraction.
[0038] The visual feature vector extracted from S2 is obtained using a fully connected layer. Point cloud geometric feature vectors and motion feature vectors The features are projected onto a common space, and regularization is introduced to prevent overfitting. The projected features are represented as follows: , in, and These are the projection matrix and bias vector corresponding to each mode, respectively; S32: Gating weight calculation.
[0039] The projected modal features are concatenated along the feature dimension to obtain the concatenated vector. The concatenated feature vectors are input into the gating network, and the normalized gating weights for each mode are calculated using the normalized exponential activation function. , in, This indicates a splicing operation. and These are the learnable parameters of the gated network. Represents the normalized exponential activation function. The generated gated weight vector satisfies And all elements are non-negative; S33: Adaptive weighted fusion.
[0040] Based on the calculated gating weight vector, the projected modalities are weighted and summed to obtain the final multimodal fusion feature. : , This multimodal fusion feature It is then mapped through a fully connected layer and used as input for subsequent multi-task learning frameworks.
[0041] The gating network can be a single-layer or multi-layer fully connected network; the softmax remains unchanged to ensure weight normalization.
[0042] If a modality is missing, the processing method used for the missing modality during implementation is consistent with that used during the model training phase. This ensures that the model has learned the feature distribution and gating weight adjustment method for the missing modality during training, thereby guaranteeing the stability of the inference phase and the reliability of the prediction results.
[0043] S4: Construct a multi-task learning framework and input the multimodal fusion features into the multi-task learning framework to collaboratively process the main task and auxiliary tasks; wherein, the main task is beam prediction, which is used to output the probability distribution of candidate beams, and the auxiliary tasks include blocking prediction and reflection intensity prediction, which are used to output physical constraint information.
[0044] S41: Beam prediction main task.
[0045] Multimodal fusion features Input a fully connected network, map the high-dimensional features to the dimensions of all candidate beams in the predefined codebook, and output a beam probability distribution vector representing the probability of each candidate beam being selected. Each element in the vector represents the confidence that the corresponding beam index is the optimal beam.
[0046] S42: Blocking prediction auxiliary task.
[0047] Multimodal fusion features Input a fully connected network to predict whether there is line-of-sight link obstruction in each candidate beam direction, and output a blocking probability vector consistent with the number of candidate beams. Each element in the vector represents a binary classification probability value of blocking in the corresponding beam direction.
[0048] S43: Auxiliary task for predicting reflection intensity.
[0049] Multimodal fusion features The input is a fully connected network, and a dual-output structure is used to predict the reflection intensity. The first output is a global reflection intensity scalar of the overall reflection capability of the current communication environment, and the second output is a directional reflection intensity vector of the strength of the multipath reflection component of each candidate beam.
[0050] Based on the above, the network branches in the multi-task learning framework used to perform the main task of beam prediction, the auxiliary task of blocking prediction, and the auxiliary task of reflection intensity prediction can adopt fully connected network structures with different numbers of layers or different hidden dimensions to adapt to the learning difficulty and output feature dimensions of different tasks.
[0051] When the network structure is modified or replaced, the multimodal fusion features are shared as the input of each task branch, and the multi-task joint loss function and its dynamic weighting strategy are combined to ensure that the model maintains stable convergence during training, thereby ensuring that effective prediction of the optimal beam can still be achieved under different network structure configurations.
[0052] S5: Design a physical constraint correction mechanism to correct the candidate beam probability distribution output by the main task using the physical constraint information from the auxiliary task output, and select the index with the highest probability as the optimal beam index based on the corrected probability distribution. This step includes correction during the inference phase and loss function design during the training phase.
[0053] S51: Two-stage physical constraint correction (inference / post-processing).
[0054] The first phase involves blocking filtering, setting a blocking probability threshold. and reflection intensity threshold A candidate beam index Corresponding blocking probability Greater than And its directional reflection intensity Less than If the probability of prediction for that beam is zero, then the prediction probability is forced to zero; otherwise, the original probability remains unchanged. The formula is as follows: .
[0055] The processed probability distribution Normalize.
[0056] The second stage performs reflection enhancement, utilizing global reflection intensity. and directional reflection intensity An enhancement factor is constructed to amplify the probability of the unfiltered beam; the enhanced probability distribution is shown below. The calculation is as follows: , in, For the reflection gain coefficient, finally based on The index with the highest probability is selected as the optimal beam index.
[0057] Based on the above, the specific values of the blocking probability threshold, reflection intensity threshold, and reflection gain coefficient used for physical constraint correction can be set or adjusted according to the characteristics of the communication scenario, deployment environment, or system performance requirements.
[0058] Under different parameter configurations, by combining the blocking prediction results and the reflection intensity prediction results to correct the probability distribution of candidate beams, it is possible to suppress unreasonable beams and enhance favorable beams, thereby improving the reliability of beam prediction.
[0059] S52: Multi-task joint loss function and dynamically weighted training (training phase) A multi-task joint loss function is constructed and a dynamic weighting strategy is adopted for model training.
[0060] Define the total loss function Cross-entropy loss from beam prediction task Binary cross-entropy loss for blocking prediction tasks and the mean square error loss of the reflection intensity prediction task The composition and calculation are as follows: , In the formula, and To follow the training rounds To prioritize the convergence of the main task in the early stages of training and to strengthen the role of physical constraints in the later stages, the dynamic weight coefficients of the auxiliary task are dynamically adjusted using a linear growth function. , in, and These are the initial weight values. and This is the growth rate coefficient. This refers to the total number of training rounds.
[0061] This embodiment achieves modal reliability adaptation through gating fusion; obtains physical constraints on obstruction and reflection through multi-task learning; and suppresses occluded and weak reflection directions and enhances strong reflection directions through two-stage correction, thereby improving prediction robustness and reducing training or alignment overhead in dynamic scenes.
[0062] Example 2 provides a beam prediction system for a millimeter-wave communication base station, comprising: The data acquisition module, including cameras, LiDAR, and GPS / positioning modules, collects I, P, and M data. The preprocessing and feature extraction module executes steps S21 to S23; The cross-modal gating fusion module executes S31~S33 and outputs... ; The multi-task inference module outputs the probability distribution of the main task. , , ; Physical constraint correction module, calculated according to S5 , And output the final beam index; The beamforming control module selects the codebook beam based on the final index. Proceed with the downlink launch.
[0063] Some modules can be moved to edge computing units or dedicated AI accelerators; training is performed on offline servers and inference is executed in real time on the base station side.
[0064] In this embodiment, the functional modules of the system cooperate with each other and are configured to execute the multimodal millimeter-wave beam prediction method based on multi-task learning described in the foregoing embodiments, thereby realizing the prediction and selection of the optimal beam.
[0065] Figure 2 This invention constructs a multimodal millimeter-wave beam prediction neural network model based on multi-task learning.
[0066] The model mainly consists of three cascaded parts: a multimodal feature extraction module, a cross-modal gating fusion module, and a multi-task joint prediction module.
[0067] First, the multimodal feature extraction module preprocesses the input raw data. This module uses a visual branch that includes object detection and a deep residual network to extract the region of interest features in the image, uses a hierarchical point cloud feature extraction network to capture the geometric structure features of the 3D point cloud, and uses a multilayer perceptron network to extract the motion state features of the user device.
[0068] Subsequently, the extracted modal features enter the cross-modal gating fusion module. This module introduces an adaptive gating mechanism, which can automatically calculate the reliability weight of each modal feature according to the dynamic changes of the communication scenario, and perform weighted fusion of the features based on the weight to generate a multimodal fusion feature vector.
[0069] Finally, the multimodal fusion feature vector is input into the multi-task joint prediction module, which is structurally divided into a main task branch and an auxiliary task branch in parallel. The main task branch is used to output the initial probability distribution of candidate beams, while the auxiliary task branch is used to output physical constraint information such as blocking probability and reflection intensity. The outputs of all branches are merged into the physical constraint correction mechanism, which uses the physical constraint information to filter and enhance the initial probability distribution, and selects the optimal beam index based on the correction results.
[0070] Figure 3 This is a flowchart of the multimodal millimeter-wave beam prediction process based on multi-task learning according to the present invention. The process specifically includes the following steps: P1~P3: First, acquire image data, 3D point cloud data collected by the base station and motion information from the user terminal. Construct a beam prediction problem model based on the geometric channel model in the dynamic scene, and transform the beam prediction task into an optimization task based on deep learning. P4~P7: The acquired multimodal data is preprocessed, including image region of interest extraction, point cloud downsampling, and spatial coordinate transformation. The processed data enters the cross-modal gating fusion module to calculate adaptive weights and generate multimodal fusion features containing rich environmental semantics. Subsequently, the fusion features are input into a multi-task learning framework to collaboratively process the main task of beam prediction and the auxiliary tasks of blocking prediction and reflection intensity prediction. Finally, the initial probability distribution and physical constraint information of the candidate beams are output. P8~P9: Construct a multi-task joint loss function based on the main task and auxiliary tasks, and adopt a dynamic weighting strategy to balance the weights of each task for model training. Determine whether the model has reached the training termination condition. If not, return to the data processing stage to continue training. If it has, output the trained model parameters and enter the subsequent correction stage. P10~P12: The probability distribution of candidate beams is corrected using the physical constraint information output by the auxiliary task. Finally, based on the corrected probability distribution, the index with the highest probability is selected as the optimal beam index to realize the beam prediction task.
[0071] In summary, the multimodal millimeter-wave beam prediction method based on multi-task learning provided by this invention acquires various key parameters in a dynamic millimeter-wave communication system based on a geometric channel model, and obtains multimodal perception data such as images, 3D point clouds, and motion data from the base station and user terminal. A model for millimeter-wave communication beam prediction in dynamic scenarios is constructed, transforming the beam prediction problem into a deep learning-based optimization problem. After preprocessing, the raw multimodal data is input into a cross-modal gating fusion module to extract environmental features and calculate adaptive weights for weighted fusion. Then, a collaborative prediction framework based on multi-task learning is constructed. This framework introduces a physical constraint correction mechanism that uses the physical information output by auxiliary tasks to correct the main task. Finally, the index with the highest probability is selected based on the corrected probability distribution to achieve optimal beam index prediction. This significantly reduces the signaling overhead and communication latency of beam training while ensuring the reliability of millimeter-wave communication.
[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A multimodal millimeter-wave beam prediction method based on multi-task learning, applied to a millimeter-wave communication system, wherein the millimeter-wave communication system includes a base station and user equipment, the base station is equipped with a camera, a 3D point cloud sensor and a positioning device, and employs a predefined beamforming codebook. Its characteristics are, The method includes: Based on the geometric channel model in dynamic communication scenarios, the beam prediction task is modeled as a deep learning optimization task. Multimodal sensing data S={I,P,M} is acquired, where I is image data, P is 3D point cloud data, and M is motion information. The image data, 3D point cloud data, and motion information are preprocessed and features are extracted to obtain visual feature vectors. Point cloud geometric feature vectors and motion feature vectors ; The prediction results of candidate beams are output based on the deep learning model, and the optimal beam index is determined from the beamforming codebook. This also includes: The , , Projecting them separately onto the common feature space, we obtain , , ; Normalized gating weights are calculated based on the projected features using a cross-modal gating fusion network. And based on the gate weights, multimodal fusion features are generated: ， Based on the multimodal fusion features Construct a multi-task learning framework for parallel output: Candidate beam probability distribution for beam prediction primary task The blocking probability vector of the blocking prediction auxiliary task Global reflection intensity of the reflection intensity prediction auxiliary task and directional reflection intensity vector ; Based on the blocking probability vector and reflection intensity prediction results, physical constraint correction is performed on the candidate beam probability distribution, first according to the blocking probability threshold. and reflection intensity threshold The candidate beams are filtered to obtain the corrected intermediate probability. Based on the reflection gain coefficient After enhancement, we get: ， According to the above The index with the highest probability is selected as the optimal beam index.
2. The multimodal millimeter-wave beam prediction method based on multi-task learning according to claim 1, characterized in that, The method includes at least the following steps: S1, based on the geometric channel model in dynamic communication scenarios, transforms the beam prediction task into a deep learning optimization task. S2, collect multimodal data S={I,P,M}, where I is image data, P is 3D point cloud data, and M is motion information; preprocess the data and extract features to obtain visual feature vectors, point cloud geometric feature vectors, and motion feature vectors; the preprocessing includes extracting regions of interest from the image data, downsampling the 3D point cloud data, and transforming the motion information into spatial coordinates; S3, input the preprocessed multimodal data into the cross-modal gating fusion module; The cross-modal gating fusion module extracts features of each modality, calculates adaptive weights of each modality feature using a gating mechanism, and performs weighted fusion of each modality feature based on the adaptive weights to obtain multimodal fusion features; S4, Construct a multi-task learning framework, and input the multimodal fusion features into the multi-task learning framework to collaboratively process the main task and auxiliary tasks; wherein, the main task is beam prediction, which is used to output the probability distribution of candidate beams, and the auxiliary tasks include blocking prediction and reflection intensity prediction, which are used to output physical constraint information. S5. Design a physical constraint correction mechanism, which uses the physical constraint information output by the auxiliary task to correct the candidate beam probability distribution output by the main task, and selects the index with the highest probability as the optimal beam index based on the corrected probability distribution.
3. The multimodal millimeter-wave beam prediction method based on multi-task learning according to claim 1, characterized in that, In the geometric channel model, the received signal of the user equipment on the k-th subcarrier satisfies: ， in, This is the channel vector for the downlink; For beamforming vectors; To transmit a signal and satisfy , This represents the average transmit power. It is additive noise that follows a complex Gaussian distribution.
4. The multimodal millimeter-wave beam prediction method based on multi-task learning according to claim 3, characterized in that, The downlink channel vector satisfies: ， in, Indicates the total number of transmission paths. For the first The complex attenuation coefficient of the path, and These are the azimuth and elevation angles of arrival, respectively. and These are the departure azimuth and departure elevation angles, respectively. The array steering vector for base stations and user equipment; for a uniform linear array, its form is determined by the antenna spacing. and signal wavelength Decide.
5. The multimodal millimeter-wave beam prediction method based on multi-task learning according to claim 1, characterized in that, The preprocessing of the image data includes: using an object detection algorithm to identify the target user and potential occluders, cropping the region of interest according to the bounding box and adjusting it to a uniform size, and then inputting it into a depth residual network to obtain the visual feature vector.
6. The multimodal millimeter-wave beam prediction method based on multi-task learning according to claim 1, characterized in that, The preprocessing of the three-dimensional point cloud data includes: standardizing the three-dimensional spatial coordinates according to the standard deviation, downsampling using a voxelized grid and normalizing the reflection intensity, and then inputting it into a hierarchical point cloud feature extraction network to obtain the point cloud geometric feature vector.
7. The multimodal millimeter-wave beam prediction method based on multi-task learning according to claim 1, characterized in that, The preprocessing of the motion information includes: converting the positioning data from the world geodetic coordinate system to the station-centered horizontal coordinate system with the base station as the origin, calculating the relative displacement difference and concatenating it with the instantaneous velocity to form a motion vector, and then inputting the motion vector into a multilayer perceptron to obtain the motion feature vector.
8. The multimodal millimeter-wave beam prediction method based on multi-task learning according to claim 1, characterized in that, The gating weight is obtained by the following formula: ， in, This indicates a splicing operation. and These are the learnable parameters of the gated network. Let g represent the normalized exponential activation function, satisfying And all elements are non-negative.
9. The multimodal millimeter-wave beam prediction method based on multi-task learning according to claim 1, characterized in that, The blocking filter satisfies: 。 10. The multimodal millimeter-wave beam prediction method based on multi-task learning according to claim 1, characterized in that, Training is performed using a multi-task joint loss function, which satisfies the following: ， ， in, This is the total loss function; This is the cross-entropy loss function, used for beam prediction tasks; This is a binary cross-entropy loss function used for blocking prediction tasks; This is the mean squared error loss function, used for reflection intensity prediction tasks; and To follow the training rounds Changing dynamic weighting coefficients and These are the initial weight values. and This is the growth rate coefficient. This refers to the total number of training rounds.