Multi-task joint training method and apparatus for beam prediction models
By employing a multi-task joint training method, utilizing the expert hybrid model MoE gating concept and a temporal modeling network, the problems of modal weight collapse and poor consistency in millimeter-wave vehicle-road cooperative communication were solved, improving the accuracy and reliability of beam prediction and ensuring the stability of high-speed communication.
Patent Information
- Application Number
- CN202611593348.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-15
- Publication Date
- 2026-11-17
AI Technical Summary
In millimeter-wave vehicle-to-infrastructure (V2I) vehicle-road cooperative communication scenarios, existing technologies suffer from problems such as excessive reliance on a single mode for mode weights, poor model generalization and fault tolerance, and poor consistency between vehicle position and beam angle prediction, resulting in high beam scanning overhead and low communication reliability.
A multi-task joint training method is adopted, which uses the expert hybrid model MoE gating concept for dynamic feature fusion. Combined with temporal modeling and multi-dimensional loss constraints, it realizes adaptive fusion of multimodal data and multi-task collaborative optimization, including dynamic feature fusion, temporal modeling, and backpropagation of joint loss function.
It significantly improves the prediction accuracy, stability, and environmental generalization ability of the model in complex scenarios, suppresses modal weight collapse and coordinate trajectory distortion, and ensures the stability and reliability of high-speed vehicle communication links.
Smart Images

Figure CN123115802A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of millimeter-wave vehicle-road cooperative communication technology, and more specifically, to a multi-task joint training method and apparatus for beam prediction models. Background Technology
[0002] In millimeter-wave vehicle-to-infrastructure (V2I) cooperative communication scenarios, roadside equipment relies on multimodal sensing data to predict vehicle position and beam direction, achieving millimeter-wave beam pre-alignment. This reduces beam scanning overhead and ensures the continuous and stable communication link for high-speed moving vehicles. The training effect of the beam prediction model directly determines the beam alignment accuracy and communication reliability under complex road conditions.
[0003] However, training based on multimodal feature fusion lacks constraints on modal weights, making it prone to weight collapse due to over-reliance on a single modality, resulting in poor model generalization tolerance. Furthermore, traditional training only performs single-task regression optimization on the vehicle's horizontal and vertical coordinates, easily leading to coordinate drift and trajectory distortion. Moreover, relying solely on coordinates to calculate beam angles fails to provide consistent alignment constraints for coordinate-derived and directly predicted angles, potentially causing physical-logical contradictions between the model's internal coordinate output and beam angle output, making it difficult to meet the training requirements for high-reliability, high-precision beam prediction in vehicle-road cooperative millimeter-wave communication. Summary of the Invention
[0004] The main objective of this application is to provide a multi-task joint training method and apparatus for beam prediction models, which solves the technical problems of single-task supervision constraints, uncontrollable modal fusion weights, and poor consistency between position and beam angle prediction in existing technologies. It achieves the technical effects of multi-modal dynamic adaptive fusion and multi-task collaborative optimization, which can effectively suppress modal weight collapse, coordinate trajectory distortion and beam angle mismatch, and significantly improve the prediction accuracy, stability and environmental generalization ability of the model in complex scenarios.
[0005] To achieve the above objectives, the first aspect of this application proposes a multi-task joint training method for a beam prediction model, comprising: acquiring a training dataset, wherein the training samples of the training dataset include multimodal data at a first time step, multimodal data at a second time step, and the actual two-dimensional coordinates of the vehicle at a third time step, wherein the multimodal data includes wireless channel data, visual image data, and depth image data collected at the same time step, and the first time step, the second time step, and the third time step are adjacent time steps; performing feature concatenation on the wireless channel data, visual image data, and depth image data at each sampling time step, inputting them into a dynamic feature fusion model based on the expert hybrid model MoE gated idea to obtain dynamic fusion weights corresponding to each modality, and generating multimodal fusion features for the corresponding time step according to the dynamic fusion weights; and using a temporal modeling network to train the multimodal data at the first time step in chronological order. The dynamic feature fusion feature and the multimodal fusion feature at the second time step are used for temporal modeling to obtain temporal features. These temporal features are then concatenated with the multimodal fusion feature at the second time step to obtain joint spatial and temporal features. The joint features are input into the beam prediction model for forward inference to obtain the predicted position coordinates, lateral classification results, and predicted angle vectors of the vehicle at the third time step. The predicted beam direction is calculated based on the predicted position coordinates and roadside unit coordinates. The model parameters of the beam prediction model, the dynamic feature fusion model, and the temporal modeling network are updated through backpropagation using a joint total loss function. The joint total loss function includes coordinate regression loss, lateral classification auxiliary loss, angle joint constraint loss, gated weight balancing loss, and gated entropy regularization term. The angle joint constraint loss includes coordinate derived angle loss, angle vector supervision loss, and angle consistency loss.
[0006] According to the multi-task joint training method of the beam prediction model provided in this application, the beam prediction model includes: a joint position prediction head, a lateral classification auxiliary head, and an angle vector auxiliary head; the step of inputting the joint features into the beam prediction model for forward inference to obtain the predicted position coordinates, lateral classification result, and predicted angle vector of the vehicle at the third time step includes: predicting the predicted position coordinates of the vehicle through the joint position prediction head, predicting the lateral classification result through the lateral classification auxiliary head, and predicting the predicted angle vector through the angle vector auxiliary head.
[0007] According to the multi-task joint training method for a beam prediction model provided in this application, after updating the model parameters of the beam prediction model through backpropagation using a joint total loss function, the method further includes: obtaining a first multimodal fusion feature at the current time and a second multimodal fusion feature at the previous time, wherein both the first and second multimodal fusion features are spatial perception information fused from wireless channel data, visual image data, and depth image data; performing temporal modeling on the first and second multimodal fusion features in chronological order through a temporal modeling network to obtain temporal features, wherein the temporal features represent the historical movement trend of the vehicle; concatenating the temporal features with the first multimodal fusion features to obtain a joint feature of spatial and temporal characteristics; inputting the joint feature into the beam prediction model to obtain the predicted position coordinates of the vehicle at the next time moment, and calculating the predicted beam direction of the vehicle at the next time moment by combining the roadside unit coordinates; wherein the predicted position coordinates and the predicted beam direction are used to enable the roadside unit to perform millimeter-wave beam alignment in advance to achieve vehicle-road cooperative communication.
[0008] According to the multi-task joint training method for beam prediction model provided in this application, the dynamic feature fusion model includes a MoE gated network and a normalization module. After feature stitching of wireless channel data, visual image data, and depth image data at each sampling time, the data is input into the dynamic feature fusion model based on the expert hybrid model MoE gated idea to obtain the dynamic fusion weights corresponding to each mode. This includes adding preset random noise to the original weight allocation coefficients output by the MoE gated network, and smoothing the original weight allocation coefficients using a normalization module with a temperature coefficient before outputting the dynamic fusion weights corresponding to the three modes.
[0009] According to the multi-task joint training method for a beam prediction model provided in this application, after obtaining the training dataset, the method further includes: performing modal degradation enhancement processing on the training samples in the training dataset to simulate a single-modal perception failure scenario; wherein, the modal degradation enhancement processing includes reducing the brightness of visual image data, blacking out the entire image, discarding visual image data, adding noise to wireless channel data, and adding noise to depth image data.
[0010] According to the multi-task joint training method for beam prediction model provided in this application, the method involves concatenating features from wireless channel data, visual image data, and depth image data at each sampling time, inputting them into a dynamic feature fusion model based on the expert hybrid model MoE gated idea, obtaining dynamic fusion weights corresponding to each modality, and generating multimodal fusion features corresponding to each time based on the dynamic fusion weights. The method includes: acquiring multimodal data at each time, the multimodal data including wireless channel data, visual image data, and depth image data; extracting corresponding original modal features for each single modality data, mapping the original modal features corresponding to each modality to a feature space of a unified dimension to obtain dimension-aligned standardized modal features; concatenating all the standardized modal features to obtain global context features, adaptively generating dynamic fusion weights corresponding to each modality based on the global context features; weighting each standardized modal feature based on the dynamic fusion weights to obtain dynamically weighted fusion features, performing an equal-weighted average on all the standardized modal features to obtain residual average features; and fusing the dynamically weighted fusion features and the residual average features to obtain multimodal fusion features.
[0011] This application also provides a multi-task joint training device for a beam prediction model, comprising the following modules: an acquisition module and a processing module; the acquisition module is used to acquire a training dataset, wherein the training samples of the training dataset include multimodal data at a first time step, multimodal data at a second time step, and the actual two-dimensional coordinates of the vehicle at a third time step, wherein the multimodal data includes wireless channel data, visual image data, and depth image data collected at the same time step, and the first time step, the second time step, and the third time step are adjacent time steps; the processing module is used to perform feature concatenation on the wireless channel data, visual image data, and depth image data at each sampling time step, input the data into a dynamic feature fusion model based on the expert hybrid model MoE gated idea, obtain the dynamic fusion weights corresponding to each modality, and generate multimodal fusion features corresponding to the time step based on the dynamic fusion weights; and through a temporal modeling network, the features are processed sequentially according to time order. The system performs temporal modeling on the multimodal fusion features at the first and second time points to obtain temporal features. These temporal features are then concatenated with the multimodal fusion features at the second time point to obtain joint spatial and temporal features. These joint features are input into a beam prediction model for forward inference to obtain the predicted vehicle position coordinates, lateral classification results, and predicted angle vectors at the third time point. The predicted beam direction is calculated based on the predicted position coordinates and roadside unit coordinates. The model parameters of the beam prediction model, the dynamic feature fusion model, and the temporal modeling network are updated through backpropagation using a joint total loss function. The joint total loss function includes coordinate regression loss, lateral classification auxiliary loss, angle joint constraint loss, gated weight balancing loss, and gated entropy regularization. The angle joint constraint loss includes coordinate derived angle loss, angle vector supervision loss, and angle consistency loss.
[0012] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a multi-task joint training method for the beam prediction model as described above.
[0013] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a multi-task joint training method for beam prediction models as described above.
[0014] This application also provides a computer program product, including a computer program that, when executed by a processor, implements a multi-task joint training method for beam prediction models as described above.
[0015] The technical solutions provided by the embodiments of this application may include the following beneficial effects: Because the dynamic feature fusion model based on the expert hybrid model MoE gating idea is used to adaptively allocate and fuse multimodal features of wireless channel, visual image, and depth image, it can dynamically adjust the contribution ratio of each modality feature according to the real-time scene conditions. Therefore, it can effectively avoid the problem that the traditional fixed weight fusion method cannot adapt to complex interference scenarios, and significantly suppress feature pollution caused by single modality anomalies. At the same time, with the gating weight balance loss and gating entropy regularization term in the training process, regular constraints can be applied to the modality weight distribution, effectively solving the weight collapse problem in the training process, and significantly improving the model's environmental generalization ability and perceptual fault tolerance. Because coordinate regression and lateral classification multi-task supervision are performed simultaneously during training, the accuracy of vehicle 2D position prediction is optimized by coordinate regression loss, and coarse-grained position constraints are applied by introducing lateral classification auxiliary loss. Therefore, the coordinate drift and trajectory distortion problems caused by single-task coordinate regression can be effectively suppressed, and the stability and rationality of vehicle position prediction can be improved. By setting up multi-angle joint constraint loss, the coordinate-derived beam angle is accurately fitted through coordinate-derived angle loss, the independent angle prediction branch is accurately supervised through angle vector supervision loss, and the output results of different angles are aligned with each other through angle consistency loss. Therefore, a deep physical relationship between position coordinates and beam angle can be established, which solves the defects of logical contradiction and poor self-consistency between coordinates and beam angle prediction under traditional training methods. In summary, this application achieves comprehensive collaborative training of modal fusion optimization, vehicle position constraints, accurate beam angle fitting, and angle consistency alignment through a multimodal dynamic adaptive fusion mechanism and a multidimensional joint multi-task loss constraint system. This overcomes the problems of poor modal tolerance, easy trajectory distortion, and low beam matching accuracy in existing training schemes, and can significantly improve the accuracy and reliability of millimeter-wave beam prediction in complex vehicle-road cooperative scenarios, ensuring the stability of millimeter-wave communication links for high-speed vehicles. Attached Figure Description
[0016] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application. In the drawings: Figure 1 One of the flowcharts illustrating the multi-task joint training method for the beam prediction model provided in this application; Figure 2 The second flowchart illustrates the multi-task joint training method for the beam prediction model provided in this application. Figure 3 This is a schematic diagram of the structure of the multi-task joint training device for the beam prediction model provided in this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation
[0017] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present application.
[0018] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0019] In this application, the terms "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing this application and its embodiments, and are not intended to limit the indicated device, element, or component to having a specific orientation, or to be constructed and operated in a specific orientation.
[0020] Furthermore, in addition to indicating location or positional relationship, some of the aforementioned terms may also have other meanings. For example, the term "above" may also be used in some cases to indicate a certain dependency or connection relationship. Those skilled in the art can understand the specific meaning of these terms in this application based on the specific circumstances.
[0021] Furthermore, the terms "installation," "setup," "equipped with," "connection," "linked," and "socketing" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral structure; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or an internal connection between two devices, components, or parts. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.
[0022] This application describes some exemplary embodiments for illustrative purposes. It should be understood that this application may be implemented in other ways not specifically shown in the accompanying drawings.
[0023] like Figure 1 As shown, this application provides a multi-task joint training method for a beam prediction model, which can be applied to a multi-task joint training device for beam prediction models. The multi-task joint training method for beam prediction models may include steps S101-S105: S101, the multi-task joint training device for beam prediction models acquires the training dataset.
[0024] The training dataset includes multimodal data at a first time step, multimodal data at a second time step, and the actual two-dimensional coordinates of the vehicle at a third time step. The multimodal data includes wireless channel data, visual image data, and depth image data collected simultaneously at the same time step, with the first, second, and third time steps being adjacent. The wireless channel data can be used to characterize the wireless propagation state between the vehicle and the roadside unit; this data can be Sub-6 GHz Channel State Information (CSI). The visual image data can be used to characterize the vehicle's actual appearance and the surrounding environment's occlusion and illumination conditions. The depth image data can be used to characterize the three-dimensional spatial distance information between the vehicle and the roadside unit.
[0025] Specifically, in real-world millimeter-wave V2I vehicle-to-infrastructure (V2I) scenarios, the multi-task joint training device can achieve synchronous sampling through various sensing devices mounted on roadside units. A unified clock synchronization mechanism triggers acquisition commands at the first and second moments, ensuring complete alignment of data timestamps and spatial field of view. Specifically, the roadside wireless radio frequency module collects wireless channel data corresponding to vehicle communication, including channel state information, multipath delay, signal attenuation, and other channel characteristic data; the roadside high-definition vision camera collects visual image data of road vehicles, covering different lighting, occlusion, vehicle speed, and road condition scenarios; and the roadside depth camera synchronously collects depth image data of the corresponding scenarios, recording the spatial distance and depth position information between the vehicle and the roadside unit.
[0026] Simultaneously, the multi-task joint training device can acquire the actual two-dimensional coordinates of the vehicle at the third moment in real time using high-precision positioning equipment, as the true value for position training; it can also further combine the roadside unit calibration coordinates with the actual two-dimensional coordinates of the vehicle, and solve the actual beam angle pointing from the roadside to the vehicle through geometric calculations, as the true value for angle training. Among them, the first moment, the second moment, and the third moment are adjacent moments arranged in chronological order.
[0027] Subsequently, the multi-task joint training device can clean, denoise, and match the collected raw data to remove invalid samples with perceptual anomalies, missing data, or spatiotemporal misalignment, ultimately obtaining a massive, scenario-rich, and accurately labeled multimodal training dataset, providing high-quality and reliable data support for the subsequent multi-task joint training of beam prediction models.
[0028] S102. The multi-task joint training device for the beam prediction model performs feature stitching on the wireless channel data, visual image data, and depth image data at each sampling time, and inputs them into a dynamic feature fusion model based on the expert hybrid model MoE gating idea to obtain the dynamic fusion weights corresponding to each mode, and generates multi-modal fusion features at the corresponding time according to the dynamic fusion weights.
[0029] Optionally, the step of performing feature concatenation on the wireless channel data, visual image data, and depth image data at each sampling time, inputting them into a dynamic feature fusion model based on the expert hybrid model MoE gated idea, obtaining dynamic fusion weights corresponding to each modality, and generating multimodal fusion features corresponding to the time based on the dynamic fusion weights, includes: acquiring multimodal data at each time, the multimodal data including wireless channel data, visual image data, and depth image data; extracting the corresponding original modal features for each single modality data, mapping the original modal features corresponding to each modality to a feature space of a unified dimension, obtaining dimension-aligned standardized modal features; concatenating all the standardized modal features to obtain global context features, adaptively generating dynamic fusion weights corresponding to each modality based on the global context features; weighting each standardized modal feature based on the dynamic fusion weights to obtain dynamically weighted fusion features, performing equal-weighted averaging on all the standardized modal features to obtain residual average features; and fusing the dynamically weighted fusion features and the residual average features to obtain multimodal fusion features.
[0030] Specifically, the multi-task joint training device can perform branch-path feature processing on three types of heterogeneous multimodal data of the training samples. It performs dedicated feature extraction operations on wireless channel data, visual image data, and depth image data respectively, removing redundant noise information from the original data and outputting original modal features adapted to their own modal characteristics. Since the three types of original modal features have problems such as inconsistent dimensions, large differences in units, and inconsistent feature distributions, they cannot be directly fused across modalities. Therefore, the multi-task joint training device can perform dimensional transformation and spatial mapping on each original modal feature through an independent projection mapping layer, unifying all modal features to a common feature space of the same dimension, obtaining standardized modal features with consistent dimensions that can be fused with each other, and completely eliminating the feature interference problem caused by heterogeneous modal fusion.
[0031] Subsequently, the multi-task joint training device can stitch together all standardized modal features along the feature channel dimension, integrating global information such as wireless channel propagation characteristics, visual appearance characteristics, and depth space characteristics to construct a global context feature that can fully characterize the perception quality of each modality in the current scene. Then, this global context feature is input into a dynamic feature fusion model based on the MoE gating concept, which adaptively learns and generates dynamic fusion weights for each modality that are adapted to the current training sample conditions based on the global information of the scene, thereby realizing an adaptive allocation mechanism that enhances effective modal weights and suppresses ineffective modal weights.
[0032] Building upon this foundation, the multi-task joint training device can construct a dual-branch feature fusion structure in parallel. On one hand, it utilizes dynamic fusion weights to weight and sum the features of each standardized modality, generating dynamically weighted fusion features that align with the perceptual reliability of the current scenario, accurately adapting to the modal differentiation requirements of complex dynamic road conditions. On the other hand, it performs an equal-weighted average operation on all standardized modal features, generating residual average features that retain all basic modal information and ensure feature integrity, serving as a fallback constraint to prevent feature loss caused by extreme shifts in dynamic weights. Finally, the dynamically weighted fusion features and the residual average features are superimposed and fused to obtain the final multimodal fusion features used for model training.
[0033] It should be noted that the introduction of the MoE dynamic weight learning mechanism during the model training phase can continuously optimize the weight allocation strategy of each modality during the iteration process, suppressing the problems of single modality weight collapse and poor modality adaptability from the training source. This enables the learned multimodal fusion features to have both scene adaptability and feature integrity, providing a highly robust basic feature support for subsequent multi-task joint training, and significantly improving the model's generalization ability under complex conditions such as occlusion, interference, and low light.
[0034] Optionally, the dynamic feature fusion model includes a MoE-gated network and a normalization module; after feature stitching of the wireless channel data, visual image data, and depth image data at each sampling time, the data is input into the dynamic feature fusion model based on the expert hybrid model MoE-gated idea to obtain the dynamic fusion weights corresponding to each mode, including: adding preset random noise to the original weight allocation coefficients output by the MoE-gated network, and using a normalization module with a temperature coefficient to smooth the original weight allocation coefficients before outputting the dynamic fusion weights corresponding to the three modes.
[0035] Specifically, during model training iterations, the MoE-gated network can initially infer the original weight allocation coefficients corresponding to the three modalities—wireless channel, visual image, and depth image—based on global context features. To break the tendency of weight solidification during training and improve the modal weight search space, the multi-task joint training device can add Gaussian random noise of a preset amplitude to the original weight allocation coefficients output by the MoE-gated network, slightly perturbing the original weight distribution. This prevents the model from directly converging to a local optimum in early iterations, thus preventing problems such as training rigidity and singular modal weight allocation.
[0036] The weight coefficients after noise superposition are then fed into a normalization module with a temperature coefficient. The temperature coefficient is used to smoothly regulate the weight distribution, weaken the preemptive effect of extreme weights, and suppress the weight collapse phenomenon where the weight of a single mode approaches 1 infinitely while the weights of other modes approach 0. At the same time, the normalization operation constrains the weight coefficients of the three modes to the legal probability range and ensures that the sum of the three weights is 1, outputting a smooth, balanced final dynamic fusion weight that is adapted to the current scenario.
[0037] It should be noted that adding preset random noise to the original weight allocation coefficients output by the MoE gated network can enhance the exploration capability of the weight parameters and improve the model's learning ability for different scene modal combinations. By smoothing the original weight allocation coefficients through a normalization module with a temperature coefficient, the defects of modal weight collapse and poor generalization that are prone to occur in traditional MoE fusion training can be solved from the structural level, making the multimodal weight allocation strategy obtained by training more robust and reasonable, and further improving the adaptive fusion capability and prediction stability of the beam prediction model in complex interference scenarios.
[0038] Optionally, after acquiring the training dataset, the multi-task joint training device can perform modal degradation enhancement processing on the training samples in the training dataset to simulate a single-modal perception failure scenario; wherein, the modal degradation enhancement processing includes reducing the brightness of visual image data, blacking out the entire image, discarding visual image data, adding noise to wireless channel data, and adding noise to depth image data.
[0039] Specifically, in order to improve the anti-interference capability and modal fault tolerance performance of the beam prediction model under complex and harsh perception conditions, and to avoid the model's over-reliance on a single modal feature leading to insufficient generalization ability, the multi-task joint training device can perform modal degradation enhancement preprocessing on the original training samples during the offline training phase of the model, and actively simulate common perception failure and data degradation problems in actual V2I vehicle-road cooperative scenarios.
[0040] For visual image data, the multi-task joint training device can simulate low-light imaging degradation scenarios such as nighttime, backlight, and cloudy days by randomly lowering the overall brightness of the image and weakening the contrast. It can also simulate real-world conditions where vehicles are obscured by roadside obstacles, pedestrians, and greenery by randomly generating occlusion masks to cover local pixel areas of the image. At the same time, it can randomly discard all visual image data from some samples to simulate scenarios where the visual modality completely fails due to visual camera failure, black screen, or extreme occlusion.
[0041] For wireless channel data, the multi-task joint training device can randomly select a portion of training samples as channel degradation samples, determine the noise amplitude based on the pre-set normalized mean square error (NMSE), and superimpose Gaussian distributed random noise into the channel state information of such samples to approximate degradation conditions such as channel measurement disturbances and wireless transmission distortion.
[0042] For depth image data, the multi-task joint training device can superimpose Gaussian noise onto the entire depth pixel matrix to simulate depth numerical deviations caused by depth camera ranging jitter and decreased ranging accuracy under adverse weather conditions. This type of data augmentation method expands the fault condition samples and improves the model's adaptive fusion capability for single-modal perception of abnormal scenes.
[0043] It should be noted that by employing a modal degradation strategy with random combination and random intensity, enhanced training samples with multimodal degradation and single-modal failure can be generated in batches. This enriches the scene coverage dimension of the training set, enabling the MoE dynamic fusion model to actively learn the weight compensation mechanism of effective modes when some modes fail or data is distorted, thus weakening the interference of failed mode features. This addresses the problem of traditional training samples having a single scene and model weights tending to favor a single mode from the data source, significantly improving the model's adaptive fusion capability and beam prediction robustness under real complex road conditions.
[0044] S103. The multi-task joint training device for the beam prediction model uses a temporal modeling network to perform temporal modeling on the multimodal fusion features at the first time step and the multimodal fusion features at the second time step in chronological order to obtain temporal features. The temporal features are then spliced with the multimodal fusion features at the second time step to obtain joint features of spatial and temporal characteristics.
[0045] Specifically, the multi-task joint training device can fuse multimodal features at the first moment. Multimodal fusion features at the second time step Arranged chronologically, they form a characteristic time sequence of length 2. ; this characteristic time series sequence The input time-series modeling network, in this embodiment, employs a gated recurrent unit (GRU) network. The GRU network receives multimodal fusion features input sequentially step-by-step. Utilizing internal update and reset gates, it selectively memorizes and filters historical feature information, mining motion evolution patterns such as displacement changes, driving deviations, and speed fluctuations within continuous sampling intervals, and outputs time-series features representing the vehicle's historical motion trends. The time-series features The feature dimensions are consistent with the feature dimensions of single-time multimodal fusion.
[0046] Subsequently, temporal features were analyzed along the feature channel dimension. Multimodal fusion features at the second time step By splicing the information together, the temporal information representing historical movement trends and the spatial perception information representing the current environmental perception are coupled together to obtain joint spatial and temporal features. The combined spatial and temporal characteristics can simultaneously carry the historical motion time sequence of the vehicle as well as the multimodal spatial perception information of the current wireless channel, vision, and depth.
[0047] S104. The multi-task joint training device of the beam prediction model inputs the joint features into the beam prediction model for forward inference to obtain the vehicle's predicted position coordinates, lateral classification results, and predicted angle vector, and calculates the predicted beam direction based on the predicted position coordinates and roadside unit coordinates.
[0048] Optionally, the beam prediction model includes: a joint position prediction head, a lateral classification auxiliary head, and an angle vector auxiliary head; the step of inputting the joint features into the beam prediction model for forward inference to obtain the predicted position coordinates, lateral classification result, and predicted angle vector of the vehicle at the third time step includes: predicting the predicted position coordinates of the vehicle using the joint position prediction head, predicting the lateral classification result using the lateral classification auxiliary head, and predicting the predicted angle vector using the angle vector auxiliary head.
[0049] Specifically, the multi-task joint training device can combine spatial and temporal features. The input beam prediction model relies on a three-branch parallel inference structure to simultaneously complete the main task prediction and dual-auxiliary task supervised inference. Each prediction head shares the same multimodal fusion feature input. The joint position prediction head, as the main prediction branch, employs a multi-layer fully connected network structure to perform deep nonlinear mapping and feature refinement of the multimodal fusion features. It fully learns the mapping relationship between vehicle spatial position, multimodal perception information, and temporal motion patterns, outputting the two-dimensional predicted position coordinates of the vehicle at the current moment, including horizontal and vertical coordinates. The lateral classification auxiliary head, as the position constraint branch, uses the lateral distribution interval of the lane as the classification basis, dividing the road lateral space into multiple continuous discrete categories. Based on the multimodal fusion features, it outputs the classification probability result and final classification label of the lateral interval to which the vehicle belongs. The angle vector auxiliary head, as an independent angle prediction branch, can mine the implicit patterns of millimeter-wave beam alignment from the multimodal fusion features, outputting a normalized two-dimensional predicted angle vector.
[0050] Optionally, the step of calculating the predicted beam direction based on the predicted position coordinates and roadside unit coordinates includes: after completing the multi-task forward inference output, the multi-task joint training device can retrieve the fixed calibration coordinates of the roadside unit, perform a difference operation between the vehicle's two-dimensional position coordinates predicted by the model and the roadside unit coordinates to obtain the lateral offset and the longitudinal offset; then, using the lateral offset and the longitudinal offset as input parameters, the predicted beam direction of the vehicle at the next moment is calculated through the four-quadrant arctan2 function.
[0051] S105. The multi-task joint training device for the beam prediction model updates the model parameters of the beam prediction model, the dynamic feature fusion model, and the temporal modeling network through backpropagation using the joint total loss function.
[0052] The joint total loss function includes coordinate regression loss, lateral classification auxiliary loss, angle joint constraint loss, gated weight balancing loss, and gated entropy regularization term; the angle joint constraint loss includes coordinate derived angle loss, angle vector supervision loss, and angle consistency loss.
[0053] Specifically, the aforementioned joint total loss function It can be represented as: ; ; in, This represents the x-axis regression loss in coordinate regression. This represents the ordinate regression loss in coordinate regression. This represents the lateral classification auxiliary loss. Indicates the joint constraint loss of angles. This indicates the loss of angle derived from the coordinates. Represents the angle vector supervision loss. Indicates the loss of angular consistency. This represents the loss due to gating weight balancing. This represents the gated entropy regularization term. , , , , , , , These are the weighting coefficients.
[0054] It should be noted that the horizontal axis regression loss Regression loss with ordinate As the main task loss, we fit and optimize the errors between the predicted horizontal and vertical coordinates of the vehicle and the actual two-dimensional coordinates to minimize the position prediction deviation and ensure the positioning prediction accuracy of the main task of the model. This is the core constraint for accurate vehicle position prediction. Lateral classification auxiliary loss To address the position-assisted constraint loss, the classification error is calculated between the lane interval classification results output by the lateral classification auxiliary head and the actual lateral interval labels. Through coarse-grained spatial classification supervision, road space boundary constraints are applied to the coordinate regression process, which can effectively suppress problems such as position drift, trajectory distortion, and boundary crossing anomalies caused by simple coordinate regression, thereby improving the rationality and continuity of vehicle driving position prediction. Angular joint constraint loss For multi-dimensional supervision loss, including coordinate-derived angle loss. Angle vector supervision loss and loss of angular consistency , This is used to constrain the beam direction derived from the predicted coordinate geometry to fit and align with the actual beam angle, thus ensuring the accuracy of the coordinate derived angle. The direct predicted angle vector used to constrain the angle vector of the auxiliary head output converges with the true angle vector, establishing an independent mapping relationship from features to beam angles; It is used to constrain the numerical consistency between the beam angle calculated by coordinate inverse calculation and the angle vector directly predicted by the model, eliminate the prediction deviation of the dual-angle branch, ensure the physical and logical consistency between the position output and the beam angle output, and solve the problem of the disconnect between coordinates and beam angle in traditional schemes. For example, using the actual beam angle at a future time as... The predicted beam angle obtained by back-calculation from the position regression head is: The direct predicted angle vector output by the angle vector auxiliary head is: For example, let's first set the actual beam angle to... With the predicted beam angle as Convert each into an L2-normalized two-dimensional angle vector: ; ; Then, calculate the coordinate-derived angle loss respectively. Angle vector supervision loss and loss of angular consistency : ; ; .
[0055] Gated weight equalization loss It is used to constrain the weight distribution of the fusion of three modalities: wireless channel, vision, and depth, so as to avoid the weight collapse problem during training, where the model relies too much on a single modality and the weights of other modalities approach zero, and improve the collaborative utilization rate of multimodal fusion. Gated entropy regularization term It is used to constrain the distribution entropy of the dynamic fusion weights of each modality, prevent the gating network from producing overly sharp weight allocation results during training, reduce the model's premature dependence on a single modality, and thus enhance the stability of the multimodal fusion process and the generalization ability in complex scenarios.
[0056] It should be noted that the gated weight balancing loss and the gated entropy regularization term are auxiliary regularization losses, calculated solely based on the dynamic fusion weights of each modality output by the dynamic feature fusion model. The dynamic fusion weights are output by the MoE gated network through a normalization module with a temperature coefficient, possessing differentiable properties. The gradient of the auxiliary regularization loss is only propagated back to the MoE gated network parameters within the dynamic feature fusion model to constrain the distribution of each modality weight and suppress modality weight collapse during training. It does not directly impose gradient constraints on the temporal modeling network or the beam prediction model.
[0057] The multi-task joint training device inputs the calculated joint total loss into the backpropagation algorithm to complete the gradient calculation as a whole. Using a gradient descent-type optimizer, it synchronously updates all model parameters of the beam prediction model, dynamic feature fusion model, and temporal modeling network based on the obtained gradients. During training iterations, the main task loss continuously reduces the prediction errors of vehicle position, lateral classification, and beam angle; the auxiliary regularization loss continuously constrains the output of the MoE gating network to a reasonable modal weight distribution, enabling the model to learn multimodal adaptive fusion strategies under complex conditions and improving the model's prediction robustness in modal degradation scenarios.
[0058] In this embodiment, a dynamic feature fusion model based on the expert hybrid model MoE gating concept is used to adaptively allocate and fuse multimodal features of wireless channels, visual images, and depth images. This model can dynamically adjust the contribution ratio of each modality feature according to the real-time scene conditions, thus effectively avoiding the problem that traditional fixed-weight fusion methods cannot adapt to complex interference scenarios and significantly suppressing feature pollution caused by single-modality anomalies. Simultaneously, the gating weight balancing loss and gating entropy regularization term during training can impose regularization constraints on the modality weight distribution, effectively solving the weight collapse problem during training and significantly improving the model's environmental generalization ability and perception fault tolerance. Furthermore, since coordinate regression and lateral classification multi-task supervision are performed simultaneously during training, the accuracy of vehicle 2D position prediction is optimized through coordinate regression loss, while lateral classification auxiliary loss is introduced to apply coarse-grained position constraints. This effectively suppresses coordinate drift and trajectory distortion caused by single-task coordinate regression. The problem is to improve the stability and rationality of vehicle position prediction. By setting up multi-angle joint constraint loss, the coordinate-derived angle loss is used to achieve accurate fitting of the coordinate-derived beam angle, and the angle vector supervision loss is used to achieve accurate supervision of the independent angle prediction branches. At the same time, the angle consistency loss is used to constrain the output results of different angles to be aligned with each other. Therefore, a deep physical relationship between position coordinates and beam angles can be established, which solves the defects of logical contradiction and poor self-consistency of coordinate and beam angle prediction under traditional training methods. In summary, this application realizes all-round collaborative training of modal fusion optimization, vehicle position constraint, accurate beam angle fitting, and angle consistency alignment through a multi-modal dynamic adaptive fusion mechanism and a multi-dimensional joint multi-task loss constraint system. It overcomes the problems of poor modal tolerance, easy trajectory distortion, and low beam matching accuracy of existing training schemes, which can significantly improve the accuracy and reliability of millimeter-wave beam prediction in complex vehicle-road cooperative scenarios and ensure the stability of millimeter-wave communication links for high-speed vehicles.
[0059] Optionally, such as Figure 2 As shown, after updating the model parameters of the beam prediction model through backpropagation using the joint total loss function, this embodiment of the application can determine the predicted position coordinates and predicted beam direction at the next moment through the converged beam prediction model. The multi-task joint training method for the beam prediction model provided in this embodiment of the application may further include S201-S204: S201, the multi-task joint training device of the beam prediction model obtains the first multimodal fusion feature at the current moment and the second multimodal fusion feature at the previous moment.
[0060] The first multimodal fusion feature and the second multimodal fusion feature are both spatial perception information formed by fusing wireless channel data, visual image data and depth image data.
[0061] Specifically, the multi-task joint training device can invoke the dynamically fused feature model after training convergence. During the actual inference and deployment phase of the model, it collects wireless channel data, visual image data, and depth image data synchronously acquired by roadside units in real time, forming the original multimodal perception samples at the current moment. Relying on the MoE modal adaptive fusion mechanism learned during the training phase, it sequentially performs feature extraction, dimension alignment, dynamic weight allocation, and feature fusion processing on the three types of heterogeneous modal data to generate the first multimodal fused feature carrying the vehicle's spatial position, channel propagation characteristics, and depth distance information at the current moment. At the same time, it caches the multimodal fused feature generated by the same fusion logic at the previous inference moment as the second multimodal fused feature.
[0062] S202. The multi-task joint training device for the beam prediction model uses a temporal modeling network to perform temporal modeling on the first multimodal fusion feature and the second multimodal fusion feature in chronological order to obtain temporal features.
[0063] The time-series feature represents the historical movement trend of the vehicle.
[0064] Specifically, the multi-task joint training device can use chronological logic as a basis, taking the second multimodal fusion feature retained from the previous moment as historical temporal information and the first multimodal fusion feature acquired at the current moment as the latest temporal information, and sequentially splicing them to form a continuous short-time feature temporal sequence, covering the perceived change information within adjacent sampling intervals of the vehicle. Then, the temporal sequence is input into a temporal modeling network, which can adopt a gated recurrent unit (GRU) or a long short-term memory (LSTM) temporal structure. It uses a gating mechanism to selectively memorize and update historical motion features, adaptively filtering effective motion information such as vehicle displacement changes, driving deviations, and speed fluctuations, while filtering perceived noise and invalid redundant features.
[0065] S203. The multi-task joint training device for the beam prediction model concatenates the temporal features with the first multimodal fusion features to obtain joint spatial and temporal features.
[0066] Specifically, the multi-task joint training device can perform deep splicing and fusion of temporal features and the first multimodal fusion feature of the current frame in the feature channel dimension, realizing bidirectional coupling of temporal motion information and spatial perception information. Among them, the first multimodal fusion feature is used to characterize the vehicle's real-time spatial position, relative distance and channel matching status; the temporal feature is used to characterize the vehicle's historical motion trajectory, driving inertia and dynamic change patterns, providing dynamic temporal constraints.
[0067] S204. The multi-task joint training device for the beam prediction model inputs the joint features into the beam prediction model to obtain the predicted position coordinates of the vehicle at the next moment, and calculates the predicted beam direction of the vehicle at the next moment by combining the roadside unit coordinates.
[0068] The predicted location coordinates and the predicted beam direction are used to enable the roadside unit to perform millimeter-wave beam alignment in advance, so as to realize vehicle-road cooperative communication.
[0069] Specifically, the multi-task joint training device can input the aforementioned joint features into the trained and converged beam prediction model, perform coordinate regression inference through the joint position prediction head, output normalized coordinate results, and perform inverse normalization conversion to obtain the two-dimensional predicted position coordinates of the vehicle at the next moment. Subsequently, the coordinates of the roadside unit are retrieved, and the spatial offset of the vehicle relative to the roadside unit is calculated through the difference between the horizontal and vertical coordinates. The predicted beam direction at the next moment is obtained by combining the four-quadrant arctangent function.
[0070] In this embodiment, by acquiring the multimodal fusion features of the current moment and the previous moment, spatial feature representation is completed through multi-source perception information such as wireless channel data, visual image data, and depth image data. This effectively compensates for the shortcomings of traditional single-moment single-modal perception features, which have a single dimension and weak environmental anti-interference ability, and improves the accuracy and completeness of spatial perception in complex road scenarios. By performing temporal modeling on the multimodal fusion features of multiple moments and extracting temporal features that represent the historical movement trend of vehicles, the dynamic movement law of continuous vehicle driving can be fully explored, solving the problems of existing solutions that rely only on single-frame spatial features, cannot adapt to high-speed lane changes, dynamic driving conditions, and have poor prediction stability. By concatenating temporal features with current-moment multimodal fusion features to obtain spatiotemporal joint features, which integrate vehicle historical motion temporal information and current spatial perception information, this approach overcomes the problem of spatiotemporal feature fragmentation in traditional solutions. It can fully fit the spatial coupling relationship of vehicle two-dimensional motion and effectively suppress trajectory distortion and coordinate offset phenomena caused by independent coordinate regression. Furthermore, by uniformly predicting the vehicle's next-moment position coordinates based on spatiotemporal joint features and combining roadside unit coordinate geometry to predict beam direction, it achieves integrated joint prediction of vehicle position and millimeter-wave beam angle, establishing an inherent geometric and physical constraint relationship between position and beam angle. This solution significantly improves beam pre-alignment accuracy in millimeter-wave V2I vehicle-to-infrastructure (V2I) scenarios, effectively avoiding beam mismatch, communication attenuation, and link interruption issues, and significantly enhancing the stability and reliability of V2I millimeter-wave communication in high-speed moving scenarios.
[0071] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0072] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0073] It should be noted that the apparatus in the embodiments of this application includes a virtual device and a physical device. The virtual device can be a multi-task joint training device for beam prediction models, and the physical device can include electronic devices, computer storage media, and computer program products.
[0074] The multi-task joint training method for beam prediction models provided in this application can be executed by a multi-task joint training device for beam prediction models, or by a control module within that device for multi-task joint training of beam prediction models. This application uses the execution of the multi-task joint training method for beam prediction models by a multi-task joint training device as an example to illustrate the multi-task joint training device for beam prediction models provided in this application.
[0075] It should be noted that, according to the above method examples, the multi-task joint training device for beam prediction models can be divided into functional modules in this application embodiment. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0076] like Figure 3 As shown in the figure, this application embodiment provides a multi-task joint training device 300 for a beam prediction model. The multi-task joint training device 300 for the beam prediction model includes: an acquisition module 301 and a processing module 302.
[0077] The acquisition module 301 can be used to acquire a training dataset. The training samples of the training dataset include multimodal data at a first time, multimodal data at a second time, and the actual two-dimensional coordinates of the vehicle at a third time. The multimodal data includes wireless channel data, visual image data, and depth image data collected at the same time. The first time, the second time, and the third time are adjacent times. The processing module 302 can be used to perform feature stitching on the wireless channel data, visual image data, and depth image data at each sampling time, input the result into a dynamic feature fusion model based on the expert hybrid model MoE gated idea, obtain the dynamic fusion weights corresponding to each modality, and generate multimodal fusion features at the corresponding time based on the dynamic fusion weights; through a temporal modeling network, perform temporal modeling on the multimodal fusion features at the first time and the multimodal fusion features at the second time according to the chronological order to obtain temporal features, and stitch the temporal features with the multimodal fusion features at the second time to obtain joint spatial and temporal features; and then... The joint feature input beam prediction model performs forward inference to obtain the predicted position coordinates, lateral classification results, and predicted angle vectors of the vehicle at the third time step, and calculates the predicted beam direction based on the predicted position coordinates and roadside unit coordinates. The model parameters of the beam prediction model, the dynamic feature fusion model, and the temporal modeling network are updated through backpropagation using a joint total loss function. The joint total loss function includes coordinate regression loss, lateral classification auxiliary loss, angle joint constraint loss, gating weight balancing loss, and gating entropy regularization term. The angle joint constraint loss includes coordinate derived angle loss, angle vector supervision loss, and angle consistency loss.
[0078] Optionally, the beam prediction model includes: a joint position prediction head, a lateral classification auxiliary head, and an angle vector auxiliary head; the step of inputting the joint features into the beam prediction model for forward inference to obtain the predicted position coordinates, lateral classification result, and predicted angle vector of the vehicle at the third time step includes: predicting the predicted position coordinates of the vehicle using the joint position prediction head, predicting the lateral classification result using the lateral classification auxiliary head, and predicting the predicted angle vector using the angle vector auxiliary head.
[0079] Optionally, after updating the model parameters of the beam prediction model through backpropagation using the joint total loss function, the acquisition module 301 can be used to acquire the first multimodal fusion feature at the current moment and the second multimodal fusion feature at the previous moment. Both the first and second multimodal fusion features are spatial perception information fused from wireless channel data, visual image data, and depth image data. The processing module 302 can be used to perform temporal modeling on the first and second multimodal fusion features in chronological order through a temporal modeling network to obtain temporal features, which represent the historical movement trend of the vehicle. The temporal features are then concatenated with the first multimodal fusion features to obtain a joint feature of spatial and temporal characteristics. The joint feature is then input into the beam prediction model to obtain the predicted position coordinates of the vehicle at the next moment, and the predicted beam direction of the vehicle at the next moment is calculated by combining the roadside unit coordinates. The predicted position coordinates and the predicted beam direction are used to enable the roadside unit to perform millimeter-wave beam alignment in advance to achieve vehicle-road cooperative communication.
[0080] Optionally, the dynamic feature fusion model includes a MoE-gated network and a normalization module; after feature stitching of the wireless channel data, visual image data, and depth image data at each sampling time, the data is input into the dynamic feature fusion model based on the expert hybrid model MoE-gated idea to obtain the dynamic fusion weights corresponding to each mode, including: adding preset random noise to the original weight allocation coefficients output by the MoE-gated network, and using a normalization module with a temperature coefficient to smooth the original weight allocation coefficients before outputting the dynamic fusion weights corresponding to the three modes.
[0081] Optionally, after acquiring the training dataset, the processing module 302 can be used to perform modal degradation enhancement processing on the training samples in the training dataset to simulate a single-modal perception failure scenario; wherein, the modal degradation enhancement processing includes reducing the brightness of visual image data, blacking out the entire image, discarding visual image data, adding noise to wireless channel data, and adding noise to depth image data.
[0082] Optionally, the step of performing feature concatenation on the wireless channel data, visual image data, and depth image data at each sampling time, inputting them into a dynamic feature fusion model based on the expert hybrid model MoE gated idea, obtaining dynamic fusion weights corresponding to each modality, and generating multimodal fusion features corresponding to the time based on the dynamic fusion weights, includes: acquiring multimodal data at each time, the multimodal data including wireless channel data, visual image data, and depth image data; extracting the corresponding original modal features for each single modality data, mapping the original modal features corresponding to each modality to a feature space of a unified dimension, obtaining dimension-aligned standardized modal features; concatenating all the standardized modal features to obtain global context features, adaptively generating dynamic fusion weights corresponding to each modality based on the global context features; weighting each standardized modal feature based on the dynamic fusion weights to obtain dynamically weighted fusion features, performing equal-weighted averaging on all the standardized modal features to obtain residual average features; and fusing the dynamically weighted fusion features and the residual average features to obtain multimodal fusion features.
[0083] In this embodiment, a dynamic feature fusion model based on the expert hybrid model MoE gating concept is used to adaptively allocate and fuse multimodal features of wireless channels, visual images, and depth images. This model can dynamically adjust the contribution ratio of each modality feature according to the real-time scene conditions, thus effectively avoiding the problem that traditional fixed-weight fusion methods cannot adapt to complex interference scenarios and significantly suppressing feature pollution caused by single-modality anomalies. Simultaneously, the gating weight balancing loss and gating entropy regularization term during training can impose regularization constraints on the modality weight distribution, effectively solving the weight collapse problem during training and significantly improving the model's environmental generalization ability and perception fault tolerance. Furthermore, since coordinate regression and lateral classification multi-task supervision are performed simultaneously during training, the accuracy of vehicle 2D position prediction is optimized through coordinate regression loss, while lateral classification auxiliary loss is introduced to apply coarse-grained position constraints. This effectively suppresses coordinate drift and trajectory distortion caused by single-task coordinate regression. The problem is to improve the stability and rationality of vehicle position prediction. By setting up multi-angle joint constraint loss, the coordinate-derived angle loss is used to achieve accurate fitting of the coordinate-derived beam angle, and the angle vector supervision loss is used to achieve accurate supervision of the independent angle prediction branches. At the same time, the angle consistency loss is used to constrain the output results of different angles to be aligned with each other. Therefore, a deep physical relationship between position coordinates and beam angles can be established, which solves the defects of logical contradiction and poor self-consistency of coordinate and beam angle prediction under traditional training methods. In summary, this application realizes all-round collaborative training of modal fusion optimization, vehicle position constraint, accurate beam angle fitting, and angle consistency alignment through a multi-modal dynamic adaptive fusion mechanism and a multi-dimensional joint multi-task loss constraint system. It overcomes the problems of poor modal tolerance, easy trajectory distortion, and low beam matching accuracy of existing training schemes, which can significantly improve the accuracy and reliability of millimeter-wave beam prediction in complex vehicle-road cooperative scenarios and ensure the stability of millimeter-wave communication links for high-speed vehicles.
[0084] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logical instructions in the memory 430 to execute a multi-task joint training method for the beam prediction model. This method includes: acquiring a training dataset, wherein the training samples of the training dataset include multimodal data at a first time moment, multimodal data at a second time moment, and the actual two-dimensional coordinates of the vehicle at a third time moment. The multimodal data includes wireless channel data, visual image data, and depth image data collected at the same time moment, wherein the first time moment, the second time moment, and the third time moment are adjacent time moments; performing feature concatenation on the wireless channel data, visual image data, and depth image data at each sampling time moment, and inputting it into a dynamic feature fusion model based on the expert hybrid model MoE gate idea to obtain the dynamic fusion weights corresponding to each modality, and generating multimodal fusion features for the corresponding time moment according to the dynamic fusion weights; and using a temporal modeling network to process the first time moment in chronological order. The multimodal fusion features and the multimodal fusion features at the second time step are used for temporal modeling to obtain temporal features. These temporal features are then concatenated with the multimodal fusion features at the second time step to obtain joint spatial and temporal features. The joint features are input into the beam prediction model for forward inference to obtain the predicted position coordinates, lateral classification results, and predicted angle vectors of the vehicle at the third time step. The predicted beam direction is calculated based on the predicted position coordinates and roadside unit coordinates. The model parameters of the beam prediction model, the dynamic feature fusion model, and the temporal modeling network are updated through backpropagation using a joint total loss function. The joint total loss function includes coordinate regression loss, lateral classification auxiliary loss, angle joint constraint loss, gated weight balancing loss, and gated entropy regularization term. The angle joint constraint loss includes coordinate derived angle loss, angle vector supervision loss, and angle consistency loss.
[0085] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0086] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-task joint training method for the beam prediction model provided by the above methods. The method includes: acquiring a training dataset, wherein the training samples of the training dataset include multimodal data at a first time moment, multimodal data at a second time moment, and the actual two-dimensional coordinates of the vehicle at a third time moment. The multimodal data includes wireless channel data, visual image data, and depth image data collected at the same time moment, wherein the first time moment, the second time moment, and the third time moment are adjacent time moments; performing feature stitching on the wireless channel data, visual image data, and depth image data at each sampling time moment, inputting them into a dynamic feature fusion model based on the expert hybrid model MoE gating idea, obtaining dynamic fusion weights corresponding to each modality, and generating corresponding [data] based on the dynamic fusion weights. The system utilizes multimodal fusion features at different time points. Through a temporal modeling network, it performs temporal modeling on the multimodal fusion features of the first and second time points in chronological order to obtain temporal features. These temporal features are then concatenated with the multimodal fusion features of the second time point to obtain joint spatial and temporal features. These joint features are input into a beam prediction model for forward inference to obtain the predicted vehicle position coordinates, lateral classification results, and predicted angle vectors at the third time point. The predicted beam direction is calculated based on the predicted position coordinates and roadside unit coordinates. The model parameters of the beam prediction model, the dynamic feature fusion model, and the temporal modeling network are updated through backpropagation using a joint total loss function. The joint total loss function includes coordinate regression loss, lateral classification auxiliary loss, angle joint constraint loss, gated weight balancing loss, and gated entropy regularization. The angle joint constraint loss includes coordinate derived angle loss, angle vector supervision loss, and angle consistency loss.
[0087] On another front, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, this computer program implements a multi-task joint training method for the beam prediction model provided by the methods described above. This method includes: acquiring a training dataset, wherein the training samples of the training dataset include multimodal data at a first time step, multimodal data at a second time step, and the actual two-dimensional coordinates of the vehicle at a third time step. The multimodal data includes wireless channel data, visual image data, and depth image data collected at the same time step, wherein the first time step, the second time step, and the third time step are adjacent time steps; performing feature concatenation on the wireless channel data, visual image data, and depth image data at each sampling time step, inputting them into a dynamic feature fusion model based on the expert hybrid model MoE gating concept to obtain dynamic fusion weights corresponding to each modality, and generating multimodal fusion features corresponding to the time step based on the dynamic fusion weights; and using time... The sequential modeling network performs temporal modeling on the multimodal fusion features at the first and second time points according to time sequence to obtain temporal features. These temporal features are then concatenated with the multimodal fusion features at the second time point to obtain joint spatial and temporal features. These joint features are input into the beam prediction model for forward inference to obtain the predicted vehicle position coordinates, lateral classification results, and predicted angle vectors at the third time point. The predicted beam direction is calculated based on the predicted position coordinates and roadside unit coordinates. The model parameters of the beam prediction model, the dynamic feature fusion model, and the temporal modeling network are updated through backpropagation using a joint total loss function. The joint total loss function includes coordinate regression loss, lateral classification auxiliary loss, angle joint constraint loss, gated weight balancing loss, and gated entropy regularization term. The angle joint constraint loss includes coordinate derived angle loss, angle vector supervision loss, and angle consistency loss.
[0088] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0089] Obviously, those skilled in the art should understand that the various units or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps into a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.
[0090] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. A multi-task joint training method for a beam prediction model, characterized in that, include: Obtain a training dataset. The training samples of the training dataset include multimodal data at a first time step, multimodal data at a second time step, and the actual two-dimensional coordinates of the vehicle at a third time step. The multimodal data includes wireless channel data, visual image data, and depth image data collected at the same time step. The first time step, the second time step, and the third time step are adjacent time steps. After feature stitching of wireless channel data, visual image data, and depth image data at each sampling time, the data is input into a dynamic feature fusion model based on the expert hybrid model MoE gate idea to obtain the dynamic fusion weights corresponding to each modality, and multimodal fusion features at the corresponding time are generated according to the dynamic fusion weights. Using a temporal modeling network, the multimodal fusion features at the first and second time points are temporally modeled in chronological order to obtain temporal features. These temporal features are then concatenated with the multimodal fusion features at the second time point to obtain joint spatial and temporal features. The joint features are input into the beam prediction model for forward inference to obtain the predicted position coordinates, lateral classification results, and predicted angle vector of the vehicle at the third time point. The predicted beam direction is then calculated based on the predicted position coordinates and roadside unit coordinates. The model parameters of the beam prediction model, the dynamic feature fusion model, and the temporal modeling network are updated by backpropagation using the joint total loss function. The joint total loss function includes coordinate regression loss, lateral classification auxiliary loss, angle joint constraint loss, gated weight balancing loss, and gated entropy regularization term; the angle joint constraint loss includes coordinate derived angle loss, angle vector supervision loss, and angle consistency loss.
2. The multi-task joint training method for beam prediction models according to claim 1, characterized in that, The beam prediction model includes: a joint position prediction head, a lateral classification auxiliary head, and an angle vector auxiliary head; The step of inputting the joint features into the beam prediction model for forward inference to obtain the predicted position coordinates, lateral classification results, and predicted angle vector of the vehicle at the third time step includes: The vehicle's predicted position coordinates are predicted using the joint position prediction head, the lateral classification result is predicted using the lateral classification auxiliary head, and the predicted angle vector is predicted using the angle vector auxiliary head.
3. The multi-task joint training method for beam prediction models according to claim 2, characterized in that, After updating the model parameters of the beam prediction model, the dynamic feature fusion model, and the temporal modeling network through backpropagation using the joint total loss function, the method further includes: The first multimodal fusion feature at the current moment and the second multimodal fusion feature at the previous moment are obtained. Both the first multimodal fusion feature and the second multimodal fusion feature are spatial perception information fused from wireless channel data, visual image data and depth image data. By using a temporal modeling network, the first multimodal fusion feature and the second multimodal fusion feature are temporally modeled in chronological order to obtain temporal features, which represent the historical movement trend of the vehicle. The temporal features are concatenated with the first multimodal fusion features to obtain joint spatial and temporal features; The joint features are input into the beam prediction model to obtain the predicted position coordinates of the vehicle at the next moment, and the predicted beam direction of the vehicle at the next moment is calculated by combining the roadside unit coordinates. The predicted location coordinates and the predicted beam direction are used to enable the roadside unit to perform millimeter-wave beam alignment in advance, so as to realize vehicle-road cooperative communication.
4. The multi-task joint training method for beam prediction models according to claim 1, characterized in that, The dynamic feature fusion model includes a MoE gated network and a normalization module; After feature stitching of the wireless channel data, visual image data, and depth image data at each sampling time, the data is input into a dynamic feature fusion model based on the expert hybrid model MoE gating concept to obtain the dynamic fusion weights corresponding to each modality, including: Preset random noise is added to the original weight allocation coefficients output by the MoE gated network, and a normalization module with a temperature coefficient is used to smooth the original weight allocation coefficients before outputting the dynamic fusion weights corresponding to the three modes.
5. The multi-task joint training method for beam prediction models according to claim 4, characterized in that, After obtaining the training dataset, the method further includes: Modal degradation enhancement processing is performed on the training samples in the training dataset to simulate a single-modal perception failure scenario; The modal degradation enhancement process includes reducing the brightness of visual image data, blacking out the entire image, discarding visual image data, adding noise to wireless channel data, and adding noise to depth image data.
6. The multi-task joint training method for beam prediction models according to claim 5, characterized in that, After feature stitching of the wireless channel data, visual image data, and depth image data at each sampling time, the data is input into a dynamic feature fusion model based on the expert hybrid model MoE gating concept to obtain dynamic fusion weights corresponding to each modality. Then, multimodal fusion features are generated according to the dynamic fusion weights at the corresponding time, including: Acquire multimodal data at each moment, including wireless channel data, visual image data, and depth image data; For each single-modal data, the corresponding original modal features are extracted, and the original modal features corresponding to each modality are mapped to a feature space of the same dimension to obtain the dimension-aligned standardized modal features. All the standardized modal features are concatenated to obtain global context features, and dynamic fusion weights corresponding to each modality are adaptively generated based on the global context features. Based on the dynamic fusion weights, each of the standardized modal features is weighted to obtain the dynamic weighted fusion features. The standardized modal features are then averaged equally to obtain the residual average features. By fusing the dynamic weighted fusion feature and the residual average feature, a multimodal fusion feature is obtained.
7. A multi-task joint training device for a beam prediction model, characterized in that, include: Acquisition module and processing module; The acquisition module is used to acquire a training dataset. The training samples of the training dataset include multimodal data at a first time, multimodal data at a second time, and the actual two-dimensional coordinates of the vehicle at a third time. The multimodal data includes wireless channel data, visual image data, and depth image data collected at the same time. The first time, the second time, and the third time are adjacent times. The processing module is used to perform feature stitching on the wireless channel data, visual image data, and depth image data at each sampling time, input the data into a dynamic feature fusion model based on the expert hybrid model MoE gate idea, obtain the dynamic fusion weights corresponding to each modality, and generate multimodal fusion features at the corresponding time according to the dynamic fusion weights. A temporal modeling network is used to perform temporal modeling on the multimodal fusion features at the first and second time points according to time sequence, obtaining temporal features. These temporal features are then concatenated with the multimodal fusion features at the second time point to obtain joint spatial and temporal features. The joint features are input into a beam prediction model for forward inference to obtain the predicted vehicle position coordinates, lateral classification results, and predicted angle vector at the third time point. The predicted beam direction is calculated based on the predicted position coordinates and roadside unit coordinates. The model parameters of the beam prediction model, the dynamic feature fusion model, and the temporal modeling network are updated through backpropagation using a joint total loss function. The joint total loss function includes coordinate regression loss, lateral classification auxiliary loss, angle joint constraint loss, gated weight balancing loss, and gated entropy regularization term; the angle joint constraint loss includes coordinate derived angle loss, angle vector supervision loss, and angle consistency loss.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multi-task joint training method for the beam prediction model as described in any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-task joint training method for the beam prediction model as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-task joint training method for the beam prediction model as described in any one of claims 1 to 6.
