Vehicle-mounted communication beam prediction method, system and equipment based on multi-modal model

Abstract

The invention discloses a vehicle-mounted communication beam prediction method, system and device based on a multi-modal model, and relates to the technical field of wireless communication. The objective of the invention is to solve the problem of poor environment adaptability and single-mode sensing robustness of millimeter wave V2V communication. A receiver vehicle collects front / rear 180-degree RGB images of the vehicle, four-way FMCW radar data, 3D laser radar point cloud and GPS data of the two vehicles in parallel, a sensor is calibrated after preprocessing, and timestamps are aligned at millimeter wave power sampling moments; caching five frames of data to form a 1s observation window, and selecting a beam prediction model according to a driving environment; and inputting data to predict the optimal beam direction in the future 500ms and outputting an index, adjusting the direction of a 60GHz millimeter wave phased array of the receiver according to the index, optimizing a link through power verification, completing road environment information sharing, re-collecting data if continuous communication is needed, and otherwise, waiting. According to the invention, multi-modal data are fused to realize accurate beam prediction, different driving scenes are adapted, and the stability and efficiency of vehicle-mounted communication are improved.

Description

Technical Field

[0001] This invention relates to the field of wireless communication technology, and more specifically to a method, system, and device for beam prediction in vehicle communication based on a multimodal model. Background Technology

[0002] Existing Internet of Vehicles (IoV) relies on the high reliability and low latency of 5G mobile communication technology to achieve data exchange between vehicles (V2V) and between vehicles and infrastructure (V2I), providing a communication foundation for autonomous driving and intelligent transportation systems. However, with the exponential growth of service data volume, the Sub-6GHz band is becoming increasingly congested, making it difficult to meet the dual requirements of ultra-low latency and ultra-high bandwidth. To address this, the Third Generation Partnership Project (3GPP) officially introduced the millimeter wave (mmWave) band into the Cellular Vehicle-to-Everything (C-V2X) standard to utilize its continuous spectrum resources at the hundreds of megahertz and even gigahertz levels.

[0003] While millimeter waves offer ample bandwidth, their free-space path loss is significantly higher than that of Sub-6GHz, necessitating the use of large-scale antenna arrays to form high-gain, narrow beams. To quickly align the beam to the optimal direction at both the transmitting and receiving ends, existing technologies typically employ the following three approaches: Exhaustive Search: Traverses all transmit-receive beam pairs in a predefined codebook. It has high accuracy, but the training overhead increases twice with the codebook size, making it difficult to meet the real-time requirements of highly dynamic V2V scenarios.

[0004] Hierarchical Search (HS): Using multi-level wide-narrow beam codebooks to progressively reduce the search space can significantly reduce the number of training iterations. However, misclassification at any level will cause the error to propagate to the next level, ultimately leading to the missed detection of the optimal beam.

[0005] Two-stage budget allocation: First, a small budget is used for a coarse-grained scan of the entire space to identify potential optimal regions. Then, the remaining budget is concentrated on a fine-grained search within those regions. This method strikes a trade-off between cost and accuracy, but still requires multiple air-to-ground measurements.

[0006] In recent years, machine learning (ML) has been introduced to further reduce training overhead. Existing solutions often rely on a single onboard sensor (camera, millimeter-wave radar, LiDAR, or GPS) to extract environmental features and train a neural network to directly predict the optimal beam index. For example: Visual assistance: Using 360° camera images as input to CNN, the top-5 prediction accuracy is approximately 85%, but it is greatly affected by lighting, weather, and occlusion. Radar-assisted: The beam trajectory is predicted by RNN using the initial beam and continuous radar echo as input. However, radar resolution is limited and multipath propagation can easily cause false alarms. LiDAR-assisted: Point clouds are structured but sparse, making it difficult to depict the fine angular information of distant targets; GPS / RTK: Positioning accuracy is typically sub-meter level, which is too large in scenarios requiring centimeter-level beam alignment.

[0007] In summary, single-modal sensing data suffers from poor environmental adaptability, missing information dimensions, or insufficient accuracy, making it difficult to provide reliable beam prediction support in complex and dynamic traffic environments. Therefore, a beam prediction scheme is urgently needed to overcome the shortcomings of existing technologies. Summary of the Invention

[0008] The technical problem to be solved by this invention is how to address the issues of poor environmental adaptability and poor robustness of single-mode sensing and prediction in millimeter-wave V2V communication.

[0009] This invention solves the above-mentioned technical problems through the following technical means: a beam prediction method for vehicle communication based on a multimodal model, comprising: S1. The receiver vehicle uses its deployed multimodal sensors to collect 180° RGB images of the front and rear of the vehicle, raw FMCW radar data from the front, rear, left, and right directions, and 3D lidar point clouds in parallel. At the same time, it acquires GPS data through GPS sensors in each group of the receiver and transmitter vehicles. All collected data are preprocessed and the sensors are calibrated. The timestamps of all data are aligned using a unified system clock, with the millimeter-wave power sampling time as the reference. S2. Cache all preprocessed data in chronological order for 5 consecutive frames to form a 1-second observation window. Select a beam prediction model from the preset model library based on the current driving environment. Input all data in the observation window into the selected beam prediction model to predict the optimal beam direction of the receiver vehicle to the transmitter vehicle within the next 500ms and output the beam index. S3. According to the beam index, adjust the beam pointing of the 60GHz millimeter-wave phased array in the corresponding direction of the receiver vehicle to align it with the 60GHz quasi-omnidirectional antenna of the transmitter vehicle, and optimize the beam pointing through the power verification mechanism to establish and maintain the communication link to complete the task of sharing road environment information between the two vehicles; if communication needs to continue, return to S1 to re-execute the data acquisition and preprocessing process, otherwise maintain the link and wait for the next communication task.

[0010] This invention acquires front / rear RGB images, four-directional FMCW radar data, lidar point cloud data, and GPS data in parallel. After preprocessing to enhance feature effectiveness, calibration to eliminate sensor bias, and synchronization of timestamps, it captures timing information through a 1-second observation window. The current driving environment selects an appropriate beam prediction model from a preset model. The beam prediction model predicts the beam direction 500ms ahead and dynamically adjusts the alignment, ultimately achieving high precision, high robustness, and low latency in beam prediction under complex vehicle environments, providing reliable support for high-speed and stable vehicle-to-vehicle communication.

[0011] Furthermore, prior to performing S1, a device deployment step is also included: The transmitter vehicle is equipped with a 60GHz quasi-omnidirectional antenna and a GPS sensor. The receiver vehicle is equipped with an FMCW radar and a 60GHz millimeter-wave phased array in the left, right, front and rear directions, respectively. It is also equipped with two RGB cameras (180° front and 180° rear), one 3D LiDAR and one GPS sensor.

[0012] Furthermore, the preprocessing includes: The MIRNet-v2 model was used to enhance brightness and restore details in the 180° RGB images of the front and rear of the vehicle. Two-dimensional Fourier transforms were performed on the raw FMCW radar data from the four directions to generate range-Doppler maps, and the range-Doppler maps from the four directions were then stitched together. The 3D LiDAR point cloud is cleaned, downsampled, and mapped to a two-dimensional histogram; The latitude and longitude coordinates of the GPS data from the receiver and transmitter vehicles are converted into relative angle information between them.

[0013] This invention addresses the issue of detail loss in low-light scenes by enhancing image brightness using the MIRNet-v2 model, ensuring the recognition of key features such as vehicle outlines and road markings. It generates range-Doppler maps by performing a two-dimensional Fourier transform on the raw FMCW radar data from four directions, avoiding blind spots in single-direction radar. LiDAR preprocessing (denoising, downsampling, and projection) simplifies redundant data while retaining core features such as range-velocity correlation and spatial density distribution. GPS relative angle conversion maps global coordinates to local azimuth, better aligning with the angle requirements of beam prediction.

[0014] Furthermore, the beam prediction model selected from the preset model based on the current driving environment specifically refers to: If real-time vehicle speed And the density of point clouds of surrounding obstacles High-speed scene model is selected; If real-time vehicle speed And the density of point clouds of surrounding obstacles The urban traffic congestion model was selected. In all other cases, the default model is used. and The preset vehicle speed threshold, and This is a preset threshold for the point cloud density of surrounding obstacles.

[0015] This invention selects the corresponding beam prediction model from a preset model library based on the current driving environment, thus solving the pain point that "a single model cannot adapt to all scenarios".

[0016] Furthermore, the beam prediction model adopts the Transformer architecture, and its feature extraction process includes: Spatial texture features of RGB images are extracted using Layer 1-Layer 6 of EfficientNet-B0; The range-Doppler map of the FMCW radar and the two-dimensional histogram of the 3D LiDAR were used to extract local spatial features through ResNet18's conv1→bn1→relu→maxpool→layer1. GPS relative angle information is upscaled to 192 dimensions using MLP.

[0017] This invention employs layered feature extraction, preserving the core advantages of each modality. Images are rendered using the lower layers (Layer 1-Layer 6) of EfficientNet-B0 to retain spatial texture and accurately capture the visual outline of the transmitter vehicle. FMCW radar / 3D LiDAR utilizes a shallow layer of ResNet18 to preserve local features, enhancing the identification of distance-velocity peaks and abrupt changes in spatial density (obstacles). GPS features are upgraded in stages to prevent low-dimensional features from being "submerged," ensuring effective integration of global orientation information. The targeted feature extraction design maximizes the unique value of each modality.

[0018] Furthermore, the specific process of fusing multimodal features using the Transformer architecture in the beam prediction model is as follows: The features of each modality are unified to the same dimension through independent linear projection layers; All modal features are concatenated into a single sequence, and the single sequence is then combined with modality type encoding and temporal location encoding. Multi-Head Self-Attention is used to model the overall sequence and enable information exchange between modalities. The output sequence is global pooled to obtain a fused feature vector, which is then processed by a multilayer perceptron and Softmax to calculate the beam probability distribution and output the predicted beam index.

[0019] This invention solves the problem of dimensionality imbalance between modalities by unifying dimensions through independent linear projection; modal encoding and positional encoding distinguish feature sources and temporal order to avoid modal confusion and temporal disorder; multi-head self-attention captures cross-modal associations to achieve semantic-level fusion.

[0020] Furthermore, the sensor calibration includes: Joint calibration of LiDAR and camera: In static scenes, a checkerboard calibration board is used to simultaneously acquire LiDAR point cloud and camera images. The two-dimensional coordinates of the corner points of the calibration board in the image coordinate system and the three-dimensional coordinates in the point cloud coordinate system are extracted. The extrinsic parameter matrix is ​​optimized through a reprojection error minimization strategy to achieve spatial alignment of the coordinate systems of the two sensors. LiDAR and FMCW radar calibration: Using corner reflectors, the echo signal of FMCW radar is matched with the corresponding reflection point in the lidar point cloud. The external parameter relationship is solved through coordinate transformation to complete the spatial calibration. GPS calibration: Corrects GPS drift errors in real time during dynamic driving using SLAM technology; Millimeter-wave power calibration: The reference power from the transmitter vehicle to the transmitter's 60 GHz quasi-omnidirectional antenna is measured in an interference-free environment, while the receiver vehicle monitors the beam power vector in real time and normalizes it to the [0,1] range.

[0021] This invention eliminates spatial deviations between sensors through joint calibration of lidar and camera, lidar and radar calibration, GPS calibration and millimeter-wave power calibration, ensuring that multimodal features are fused in the same coordinate system and avoiding correlation errors caused by spatial misalignment.

[0022] Furthermore, the power verification mechanism is as follows: The beam prediction model outputs Top-K predicted beam directions in the codebook, where K is a positive integer and is dynamically adjusted according to the number of phased array beams. In the predicted Top-K beam set, the actual communication link received power of each beam is measured in sequence, and the beam with the highest received power is selected as the final alignment direction.

[0023] This invention verifies power in the predicted Top-K beam, which reduces the redundancy overhead of full codebook scanning and corrects prediction deviations through actual power measurement, ensuring that the received power in the final beam direction meets communication quality requirements.

[0024] This invention also provides a vehicle-mounted communication beam prediction system based on a multimodal model, comprising: The receiver vehicle is equipped with: two RGB cameras, used to acquire 180° RGB images from the front and rear of the vehicle respectively; one 3D LiDAR, used to acquire 3D LiDAR point clouds of the surrounding environment; four FMCW radars, deployed facing the front, rear, left, and right of the vehicle respectively, used to acquire raw radar echoes; one GPS sensor, used to obtain the position of the receiver vehicle; and four 60GHz millimeter-wave phased arrays, deployed facing the front, rear, left, and right of the vehicle respectively, used for beam adjustment. The transmitter vehicle is equipped with: one 60GHz quasi-omnidirectional antenna for transmitting communication signals; and one GPS sensor for obtaining the position of the transmitter vehicle. Multimodal perception module: used to acquire 180° RGB images of the front and rear of the vehicle, raw FMCW radar data in the front, rear, left and right directions, and 3D lidar point cloud in parallel; at the same time, it acquires GPS data through GPS sensors of each group of receiver and transmitter vehicles, and performs preprocessing and sensor calibration on all acquired data. Model selection module: used to cache all preprocessed data in chronological order for 5 consecutive frames to form a 1-second observation window; select the corresponding beam prediction model from the preset model library according to the current driving environment; Beam prediction module: Used to input all data in the observation window into the selected beam prediction model, predict the optimal beam direction of the receiver vehicle to the transmitter vehicle within the next 500ms, and output the beam index; The communication control module is used to adjust the beam pointing of the 60GHz millimeter-wave phased array of the receiver vehicle in the corresponding direction according to the beam index, so that it is aligned with the 60GHz quasi-omnidirectional antenna of the transmitter vehicle, and optimize the beam pointing through a power verification mechanism to establish and maintain a communication link to complete the task of sharing road environment information between the two vehicles; if communication needs to continue, it returns to the multimodal sensing module to re-execute the data acquisition and preprocessing process; otherwise, it maintains the link and waits for the next communication task. The time-space synchronization module is used to synchronize all data with a unified system clock, using the millimeter-wave power sampling time as a reference to complete the timestamp alignment.

[0025] The present invention also provides a processing device, including at least one processor and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor can execute the above-described method steps by calling the program instructions.

[0026] The advantages of this invention are: This invention integrates multimodal data (RGB images, FMCW radar data, 3D LiDAR point clouds, and GPS data), and with the help of beam prediction models adapted to different driving scenarios, it can more comprehensively and accurately capture the vehicle's surrounding environment and motion state, improve the accuracy and robustness of beam prediction, and solve the problem of poor prediction performance of single-modal perception in complex environments.

[0027] This invention predicts the optimal future beam direction and dynamically adjusts the beam pointing of the millimeter-wave phased array, and optimizes the communication link through power verification. This can effectively improve the stability and efficiency of millimeter-wave V2V communication, enhance environmental adaptability, and enable vehicles to maintain good communication quality in dynamic driving scenarios (such as lane changes, turns, and changes in vehicle distance).

[0028] This invention enables the sharing and communication of road environment information, meets the needs of vehicle-mounted collaborative tasks, and can flexibly trigger and maintain the link according to communication requirements, widely adapting to different driving scenarios and providing strong communication support for safe and efficient vehicle driving. Attached Figure Description

[0029] Figure 1 This is a flowchart of the vehicle communication beam prediction method based on a multimodal model according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram illustrating the operation of embodiment 1 of the present invention; Figure 3 This is a model architecture diagram of embodiment 1 of the present invention; Figure 4 This is a flowchart of vehicle communication beam prediction in Embodiment 1 of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Example 1 like Figure 1 As shown, the beam prediction method for vehicular communication based on a multimodal model includes: Step S1: The receiver vehicle uses its deployed multimodal sensors to collect parallel 180° RGB images of the front and rear of the vehicle, raw FMCW radar data from the front, rear, left, and right directions, and 3D lidar point clouds. Simultaneously, it acquires GPS data from the GPS sensors of each group on both the receiver and transmitter vehicles. All collected data undergoes preprocessing and sensor calibration, and is synchronized using a unified system clock, with the millimeter-wave power sampling time as the reference for timestamp alignment of all data. For example... Figure 2 The diagram shown illustrates the operation.

[0032] Specifically, before step S1, the process includes equipment deployment: deploying two RGB cameras (one capturing RGB images of the front 180° and the other capturing RGB images of the rear 180°), one 3D LiDAR, four FMCW radars (facing the front, rear, left, and right of the vehicle respectively), one GPS sensor, and four 60GHz millimeter-wave phased arrays facing four different directions (front, rear, left, and right) on the receiver vehicle. Deploying a 60GHz quasi-omnidirectional antenna and one GPS sensor on the transmitter vehicle. Sensor calibration is performed to ensure precise field-of-view matching of each sensor and phased array antenna, and to synchronize their data acquisition rates to achieve high data consistency and synchronization. The user selects the most suitable beam prediction model based on the current driving environment. Subsequently, the receiver and transmitter vehicles begin driving on the road.

[0033] Sensor calibration methods include: Joint calibration of LiDAR and camera: Calibration is performed using a checkerboard calibration board in a static scene. LiDAR point cloud and camera images are acquired simultaneously. The 2D and 3D coordinates of the corner points of the calibration board in their respective coordinate systems are extracted. The extrinsic parameter matrix is ​​optimized using a reprojection error minimization strategy to achieve accurate spatial alignment of the coordinate systems of the two sensors. The reprojection error is used as the evaluation metric. The extrinsic parameter matrix must ensure that the average reprojection error of the corner points of the calibration board is ≤1 pixel. If it exceeds this threshold, the extrinsic parameters are re-optimized.

[0034] LiDAR and radar calibration: Using a corner reflector, the high-intensity echo signal generated by the reflector is detected by millimeter-wave radar and matched with the corresponding reflection point in the lidar point cloud. The external parameter relationship between the two is solved through coordinate transformation to complete the spatial calibration. The spatial position error between the point cloud reflection point and the center of the millimeter-wave radar echo is required to be ≤10cm, based on the alignment error of the corner reflector position. If the deviation is greater than this value, the calibration is repeated.

[0035] GPS calibration: GPS drift is corrected in real time using SLAM technology during dynamic driving; GPS and SLAM fusion: Calibration is performed using trajectory overlap error, requiring the offset between the SLAM-corrected trajectory and the original GPS trajectory to be within 0.5m.

[0036] Millimeter-wave power calibration: The reference power of the quasi-omnidirectional transmitter is measured in an interference-free environment. The beam power vector is monitored in real time at the receiver and normalized to the same range. The received power is uniformly mapped to the [0,1] interval, thereby ensuring that the power reference for subsequent beam direction prediction is consistent.

[0037] Data synchronization: All collected data is synchronized with a unified system clock, based on approximate timestamp alignment. The millimeter-wave power sampling time is used as the reference, and other modal data samples are selected from those closest to the timestamp. All sensor data are collected within a fixed sampling period of 200ms. GPS delay is compensated using an interpolation strategy. When the delay of GPS data relative to the system clock exceeds 200ms, the interpolation strategy is activated.

[0038] Execution process: The system samples at millimeter-wave power. Based on this, find the timestamp of the most recent GPS data. .like Linear interpolation is then used to calculate the location information for the missing time periods. If the continuous loss time exceeds 500ms, the SLAM results completely replace the GPS data for that period to ensure the accuracy of the overall time alignment.

[0039] Step S2: Cache all preprocessed data in chronological order for 5 consecutive frames to form a 1-second observation window. Select a beam prediction model from the preset model library based on the current driving environment. Input all data in the observation window into the selected beam prediction model to predict the optimal beam direction of the receiver vehicle to the transmitter vehicle within the next 500ms and output the beam index.

[0040] Specifically, after five data acquisitions are completed, all of this preprocessed data will be fed into the beam prediction model to predict the beam direction of the receiver vehicle toward the transmitter vehicle within the next 500ms.

[0041] Five observation windows (1 second each, 5 frames, 200ms interval) were selected: This ensured the real-time performance of the system while acquiring a sufficiently long historical time series information. All data could effectively capture dynamic changes in vehicle behavior within a 1-second timeframe.

[0042] Choosing a 500ms prediction duration: In high-speed V2V scenarios, the relative position and attitude changes of vehicles within 500ms remain within a predictable range, preventing a significant reduction in prediction accuracy due to excessive motion uncertainty. The prediction results can be used in advance for beam pre-configuration in actual systems, effectively offsetting communication delays caused by channel detection and beam training, and improving the timeliness and stability of link establishment.

[0043] Selecting a beam prediction model refers to choosing the beam prediction model that best matches the characteristics of the current environment and has the optimal prediction performance under different driving conditions. Independent models were pre-trained for various driving states and lighting conditions (day / night). The model training process was configured as follows: using the cross-entropy loss function, optimized by the AdamW optimizer, with a batch size of 4 and a learning rate of 1e-4, dynamically switching based on perceived data during runtime. This mainly includes: High-speed scene model (day / night): Applicable conditions: Vehicle speed is greater than 70 km / h and traffic density is low (point cloud density ρ < 0.05).

[0044] Emphasis: Enhance the ability to predict the trajectory of vehicles at high speeds, improve the accuracy of relative angle estimation at the receiver, and ensure stable beam alignment under conditions of rapid relative displacement.

[0045] City congestion model (day / night): Applicable conditions: Vehicle speed is less than 50 km / h and traffic density is high (point cloud density ρ > 0.25).

[0046] Focus areas: Enhance the ability to model the spatial layout of surrounding obstacles and optimize beam orientation selection under non-line-of-sight (NLOS) conditions to improve link robustness in complex environments.

[0047] Default model: Applicable conditions: Scenarios where vehicle speed is 50–70 km / h and traffic density is 0.05≤ρ≤0.25.

[0048] Focus: To achieve a balance between prediction accuracy and computational resource consumption when the vehicle speed and environmental complexity are moderate.

[0049] The point cloud density ρ is calculated as follows: A fan-shaped region centered on the receiver vehicle, with a forward and backward azimuth angle of ±60° and a distance of 0~30 meters, is defined as the Region of Interest (ROI); this fan-shaped region is divided into a 1-meter × 1-meter grid; all non-ground point clouds within the ROI are extracted and projected onto the grid on the horizontal plane, and the number of grids occupied by at least one point cloud is counted; the point cloud density ρ is the ratio of the number of occupied grids to the total number of grids in the fan-shaped region.

[0050] The system automatically selects a matching beam prediction model from the model library based on real-time vehicle speed, point cloud density, and lighting conditions. For example, when the millimeter-wave radar detects a vehicle speed of 80 km / h, a point cloud density ρ of less than 0.05, and nighttime lighting conditions, the system automatically calls a high-speed nighttime scene model as the currently used beam prediction model.

[0051] The specific implementation method of beam prediction for vehicle communication in this embodiment is as follows: Figure 4 As shown.

[0052] The receiver vehicle and the transmitter vehicle need to communicate. For example, the receiver vehicle needs to share information about the road environment with the transmitter vehicle, such as water accumulation and road obstacles.

[0053] The multimodal module of the receiver vehicle preprocesses the captured 180° front and rear images of the vehicle, and uses the MIRNet-v2 model to effectively enhance the brightness and restore the details of the darker areas in the images.

[0054] The receiver vehicle's multi-mode module preprocesses the raw data acquired by the four FMCW radars, generates range-Doppler maps through two-dimensional Fourier transform, and stitches these four maps together to obtain comprehensive radar information.

[0055] The receiver vehicle's multimodal module cleans and downsamples the point cloud data acquired by the 3D LiDAR, and maps it to a two-dimensional histogram to simplify data processing and preserve key spatial features.

[0056] The receiver vehicle's multimodal module converts the obtained GPS latitude and longitude coordinates of the two vehicles into relative angle information of the two vehicles.

[0057] The calculation of relative angle information is as follows: Calculation of latitude and longitude difference: Obtain the GPS coordinates of the transmitter and receiver vehicles. , .

[0058] Calculate the difference in latitude and longitude: .

[0059] Longitude difference correction: Since the longitude distance scales with latitude, the longitude difference needs to be corrected when calculating relative directions. .

[0060] Latitude difference can be directly used as the north-south component: .

[0061] Azimuth calculation: using the arctangent two-parameter function The azimuth angle of the receiver relative to the transmitter is obtained. , The range is .

[0062] Angle normalization: unifying angles to... 0° is defined as due north, and the angle values ​​increase clockwise.

[0063] Beam prediction models employ a Transformer architecture. Although all beam prediction models use the same Transformer structure in their network architecture, the weight parameters learned by the models through backpropagation during independent training will inevitably differ due to significant differences in the distribution of the data used for training (such as vehicle speed range, traffic density, and lighting conditions). The Transformer weights, including the Q, K, and V mapping matrices of the self-attention layer and the parameters of the multilayer perceptron, are iteratively optimized using the gradient descent algorithm under the constraint of the objective loss function. When the input distribution and target label distribution of the training data are different, the gradient direction, weight update path, and local optima converged by the network during the optimization process will also change accordingly. Therefore, the final weight parameters of the models cannot be completely consistent.

[0064] For example, in datasets with low nighttime lighting, visual features are noisier, and the model adaptively reduces the attention intensity of the visual branch, resulting in a parameter distribution that is drastically different from the daytime model. In urban congestion scenarios, the spatial geometric relationship between vision and LiDAR is more prominent, and during model training, the parameter update direction of the corresponding layers will be biased towards enhancing the connection weights of the spatial interaction parts.

[0065] Although the models for different scenarios are structurally identical, the different statistical characteristics of the training data lead to different parameter spaces during the optimization process, resulting in independent models with significantly different weight parameters. Independent training across multiple scenarios allows each model to achieve better beam prediction performance in its target environment, while training a single model on mixed data can lead to a decrease in model generalization ability due to conflicting feature distributions, making it difficult to simultaneously achieve optimal performance across different scenarios.

[0066] Model architecture diagram as follows Figure 3 As shown, it includes: 1. Feature extraction from image data (based on EfficientNet-B0) The feature extraction of the preprocessed RGB images (5 consecutive frames in each direction, 256×256×3 pixels, for a total of 10 frames) from the front and rear 180° of the vehicle were performed using Layer 1 to Layer 6 of EfficientNet-B0, rather than the complete network or the top fully connected layer. The specific design is as follows: EfficientNet-B0's lower and middle layers (Layer 1-Layer 6) retain richer spatial texture and edge information, making it sensitive to the geometric contours of vehicles, road markings, and obstacles. Using semantic features extracted across all layers (including deep layers) would result in spatial resolution loss due to multiple downsampling steps (e.g., excessive compression of feature map sizes), failing to accurately capture the location details of the transmitting vehicle. Furthermore, compared to EfficientNet-B3~B7 or the ResNet series, EfficientNet-B0, through a composite scaling strategy (coordinated adjustment of depth, width, and resolution), reduces the number of parameters and inference latency while achieving slightly better prediction accuracy, making it more suitable for the computing power limitations of in-vehicle terminals.

[0067] After processing 10 frames of RGB images (256×256×3) through Layer1-Layer6, the output feature is 10×24×64×64 (10 is the number of frames, 24 is the number of channels, and 64×64 is the feature map size), which not only retains sufficient spatial resolution, but also achieves preliminary feature compression through 24 channels.

[0068] 2. Feature extraction for radar and lidar (based on shallow ResNet18 layers) The range-Doppler image (5 frames, size 256×128×1) from the FMCW radar and the BEV (bird's-eye view, 5 frames, size 256×256×1) from the LiDAR were extracted using the ResNet18 conv1→bn1→relu→maxpool→layer1 module, with the specific design as follows: ResNet18's layer 1 is the first-stage residual block with a small receptive field (approximately 32×32), suitable for extracting local spatial information such as velocity peaks in the range-Doppler image and obstacle edges in the BEV image. Using deeper blocks (such as layer 2 and above) would lead to excessive compression of spatial information (a sharp reduction in feature map size) due to multiple convolutions, resulting in the loss of fine range-velocity features from radar echoes and density distribution details of laser point clouds. Simultaneously, using only shallow modules reduces redundant computation, resulting in lower overall inference latency and meeting the real-time requirements of automotive applications.

[0069] After the input data from radar and lidar are processed by the above modules, they both output 5×24×64×64 features (5 is the number of frames, 24 is the number of channels, and 64×64 is the feature map size), which is completely aligned with the number of channels and spatial size of image features, laying the foundation for subsequent fusion.

[0070] 3. Feature mapping of GPS relative angle information (phased dimensionality increase) The relative angle information between the receiver and transmitter (5 frames, 1D angle value per frame) is not directly concatenated with other modal features, but is upgraded in stages through 3 linear layers, as detailed below: Selection criteria: GPS angle information is a low-dimensional scalar (1D), which differs significantly from the high-dimensional features (24×64×64) of images / radar in terms of dimensionality and data distribution. Direct concatenation would lead to an imbalance in intermodal weights (GPS features would be "submerged"). Phased dimensionality upscaling can gradually integrate GPS features into spatial coding characteristics, enabling them to interact effectively with image and radar features at different resolutions.

[0071] 4. Transformer Multimodal Fusion Taking the first-layer Transformer (d_model=24) as an example, the fusion process is explained in detail: 4.1 Independent Linear Projection (Unified Feature Dimension) After each modality is extracted by the backbone network, the spatial resolution is aligned to a fixed anchor point (8×8) using Adaptive AvgPool2d, then flattened into a token sequence, and each branch feature is mapped to the same dimension d_model=24 through independent linear layers. Image: F_img (10×24×8×8) → Flattened to 10×64×24 → Linear projection layer Linear_img → Output F_img_proj=10×64×24.

[0072] LiDAR: F_lidar (5×24×8×8) → flattened to 5×64×24 → Linear_lidar → F_lidar_proj=5×64×24.

[0073] Millimeter-wave radar: Similarly → F_radar_proj=5×64×24.

[0074] GPS: The original input is 5×1, which is linearly upscaled to 24 dimensions (5×24) through a fully connected network → F_gps_proj=5×24. Purpose: To eliminate the difference in feature dimensions between modalities, enabling the spatial features of the image and the distance features of the radar to interact.

[0075] 4.2 Sequence Construction and Dual Encoding (Label Feature Source and Temporal Sequence) Serialization: Converting "time × space × feature" into a one-dimensional sequence. Image sequence: Seq_img=F_img_proj → Length 10×64=640 → Dimension 640×24.

[0076] LiDAR sequence: Seq_lidar=F_lidar_proj→length 5×64=320→320×24.

[0077] Millimeter-wave radar: Length is 320→320×24.

[0078] GPS sequence: length 5→5×24.

[0079] Positional Encoding: Uses learnable positional embeddings pos_emb to add positional information to the token embedding by adding them element by element.

[0080] 4.3 Multi-Head Self-Attention Interaction Global sequence concatenation: Seq global=[Seqimg_enc;Seqlidar_enc;Seqradar_enc;Seqgps_enc], total length: 640+320×2+5=1285→dimension 1285×24.

[0081] Multi-head attention computation 4 attention heads (each with 24 / 4=6 dimensions) map the global sequence to , , (All are 1285×24); Each head independently calculates attention weights; for example, the first head focuses on associating "vehicle outlines in the image" with "vehicle speed from the radar."

[0082] Where h represents the h-th attention head, and in this embodiment there are 4 attention heads. (Query) is the query matrix with the h-th head, used to represent the focus of the current token on other tokens. (Key) is the key matrix of the h-th head, used to measure the similarity or matching degree between different tokens. (Value) is the value matrix of the h-th head, representing the feature information carried by the attention token, and d is the feature dimension of single-head attention. In this embodiment, d=6 (24 / 4=6).

[0083] 4-head output concatenation → linear layer transformation → resulting in the interactive global sequence Seq_{global_trans}=1285×24.

[0084] 4.4 Branch Feature Reconstruction and Residual Connection Sequence splitting: Split Seq_{global_trans} according to its original length to restore the interaction features of each modality: Image interaction features: Take the first 640 elements → reshape them into 10×24×8×8 → denoted as F_img_trans.

[0085] LiDAR / Radar: Similarly, F_lidar_trans, F_radar_trans.

[0086] GPS: Retain the last 5×24, do not perform spatial reshape → F_gps_trans.

[0087] Residual connection: First, use F.interpolate to upsample the interactive features to the same spatial size as the backbone features (64×64), then use element-wise addition to achieve residual connection (preserving the original feature details): Fimg_out = Fimg_trans.interpolate + Fimg (both are 10×24×64×64) The same applies to other modes.

[0088] Subsequent Transformer layers (layers 2-4) Repeat the above procedure, adjusting only the following parameters: Transformer2 layer: d_model=40, resolution 32×32, number of heads 4 (10 dimensions per head).

[0089] Transformer3 layer: d_model=80, resolution 16×16, number of heads 4 (20 dimensions per head).

[0090] Transformer 4 layers: d_model=192, resolution 8×8, number of heads 4 (48 dimensions per head).

[0091] 4.5 Beam Prediction Output Take the global fusion feature output from the last Transformer layer (d_model=192) (obtain a 1×192 vector by global averaging or other pooling of spatial-temporal tokens), input this 192-dimensional vector into the MLP (192→256) to obtain the final codebook prediction score, and select the beam index corresponding to the highest score.

[0092] Through the above design, multimodal features retain their respective advantages (texture details of images, range-velocity information of radar, global orientation of GPS), and cross-modal information complementarity is achieved through a hierarchical Transformer, ultimately improving beam prediction accuracy while ensuring real-time performance.

[0093] Step S3: Based on the beam index, adjust the beam pointing of the 60GHz millimeter-wave phased array of the receiver vehicle in the corresponding direction to align it with the 60GHz quasi-omnidirectional antenna of the transmitter vehicle. Optimize the beam pointing through a power verification mechanism to establish and maintain a communication link, thereby completing the task of sharing road environment information between the two vehicles. If continued communication is required, return to S1 to re-execute the data acquisition and preprocessing process; otherwise, maintain the link and wait for the next communication task.

[0094] Specifically, the model output codebook contains the Top-K predicted beam directions (K can be set according to system requirements, such as Top-5). In experimental verification, the process is as follows: Model prediction phase: Input multimodal sensing data, and the model outputs the K most likely beam indices and their confidence ranking.

[0095] Beam Sweeping within Top-K: In the predicted Top-K beam set, the power of each beam is measured sequentially using the actual communication link to find the beam with the highest received power. This balances prediction efficiency and accuracy verification, reducing the overhead of exhaustive full codebook scanning.

[0096] If communication needs to continue, return to S1 to re-execute the data acquisition and processing flow; otherwise, maintain the link and wait for the next communication task.

[0097] Specific conditions for communication: Vehicle relative position and distance threshold: When the distance between two vehicles is less than 100m and they are in front of or behind each other or parallel to each other.

[0098] Vehicle behavior prediction: If the transmitter vehicle is detected to be about to change lanes or turn, it is necessary to communicate and exchange intentions in advance.

[0099] Communication requirement triggers: such as vehicle-mounted tasks that require synchronous collaboration (cooperative driving, route negotiation, etc.).

[0100] Historical link status: When link quality degrades, the system attempts to re-establish communication. The system uses Received Signal Strength (RSSI) as a link quality indicator. Measurement method: The millimeter-wave phased array of the receiver vehicle measures the average power intensity received in real time when receiving the signal of each predicted beam.

[0101] Application standard: When RSSI is below -80dB, the system determines that the link strength is insufficient.

[0102] Example 2 Based on Embodiment 1, Embodiment 2 of the present invention also provides a vehicle-mounted communication beam prediction system based on a multimodal model, including: The receiver vehicle is equipped with: two RGB cameras, used to acquire 180° RGB images from the front and rear of the vehicle respectively; one 3D LiDAR, used to acquire 3D LiDAR point clouds of the surrounding environment; four FMCW radars, deployed in the front, rear, left, and right directions facing the vehicle respectively, used to acquire raw radar echoes; one GPS sensor, used to obtain the position of the receiver vehicle; and four 60GHz millimeter-wave phased arrays, deployed in the front, rear, left, and right directions facing the vehicle respectively, used for beam adjustment.

[0103] The transmitter vehicle is equipped with: one 60GHz quasi-omnidirectional antenna for transmitting communication signals; and one GPS sensor for obtaining the position of the transmitter vehicle.

[0104] Multimodal perception module: used to acquire 180° RGB images of the front and rear of the vehicle, raw FMCW radar data in the front, rear, left and right directions, and 3D lidar point cloud in parallel; at the same time, it acquires GPS data through GPS sensors of each group of receiver and transmitter vehicles, and performs preprocessing and sensor calibration on all acquired data.

[0105] Includes: Image acquisition unit: Deploys 2 RGB cameras (180° in front of the vehicle and 180° behind the vehicle) to output raw RGB images for capturing the texture and contour information around the vehicle.

[0106] Radar acquisition unit: Deploys 4 FMCW radars (front, rear, left, and right directions) to output raw echo data for extracting target distance and velocity characteristics.

[0107] LiDAR acquisition unit: Deploys one 3D LiDAR to output point cloud data, which is used to construct the spatial density distribution of the surrounding environment.

[0108] GPS positioning unit: The receiver and transmitter are each equipped with one GPS sensor, which outputs latitude and longitude coordinates to calculate the relative positions of the two vehicles.

[0109] Millimeter-wave communication unit: The receiver is equipped with four 60GHz millimeter-wave phased arrays (front, rear, left, and right directions), and the transmitter is equipped with one 60GHz quasi-omnidirectional antenna for signal transmission.

[0110] Image preprocessing unit: Used to perform brightness enhancement and detail restoration on the front / rear RGB images of the vehicle using the MIRNet-v2 model.

[0111] Radar data preprocessing unit: used to perform two-dimensional Fourier transform on the raw FMCW radar data in four directions respectively, generate range-Doppler maps, and stitch the range-Doppler maps in the four directions together.

[0112] LiDAR preprocessing unit: used to clean and downsample the 3D LiDAR point cloud and map it to a two-dimensional histogram.

[0113] GPS preprocessing unit: Used to convert the latitude and longitude coordinates of GPS data from the receiver and transmitter into relative angles.

[0114] The joint calibration unit for LiDAR and camera is used to simultaneously acquire LiDAR point clouds and camera images in static scenes using a checkerboard calibration board. It extracts the two-dimensional and three-dimensional coordinates of the corner points of the calibration board in their respective coordinate systems and optimizes the extrinsic parameter matrix through a reprojection error minimization strategy to achieve spatial alignment of the coordinate systems of the two sensors.

[0115] LiDAR and radar calibration unit: Used to match the echo signal of millimeter-wave radar with the corresponding reflection point in the lidar point cloud using a corner reflector, and solve the external parameter relationship between the two through coordinate transformation to complete spatial calibration.

[0116] GPS calibration unit: Used to correct GPS drift in real time during dynamic driving using SLAM technology.

[0117] Millimeter-wave power calibration unit: used to measure the reference power of a quasi-omnidirectional transmitter in an interference-free environment, and the receiver monitors the beam power vector in real time and normalizes it to a uniform range.

[0118] Model selection module: This module caches five consecutive frames of preprocessed data in chronological order, forming a 1-second observation window; it selects the corresponding beam prediction model from a preset model library based on the current driving environment; including: Data caching unit: used to cache preprocessed multimodal data in chronological order, continuously storing 5 frames (each frame interval 200ms) to form a 1s observation window, ensuring that it contains enough time sequence information to capture vehicle dynamics.

[0119] Model selection unit: used to select the corresponding beam prediction model from the preset model library based on the current driving environment; The preset model library contains three types of models: high-speed scene model (day / night), urban congestion model (day / night), and default model, which are selected according to the following logic: The beam prediction model selected from the preset model based on the current driving environment is as follows: If real-time vehicle speed And the density of point clouds of surrounding obstacles A high-speed scene model was selected.

[0120] If real-time vehicle speed And the density of point clouds of surrounding obstacles An urban traffic congestion model was selected.

[0121] In all other cases, the default model is used. and The preset vehicle speed threshold, and This is a preset threshold for the point cloud density of surrounding obstacles.

[0122] Beam prediction module: This module takes all data within the observation window and inputs it into the selected beam prediction model to predict the optimal beam direction from the receiver vehicle to the transmitter vehicle within the next 500ms, and outputs the beam index; it includes: Image feature extraction unit: used to extract spatial texture features from RGB images via Layer 1-Layer 6 of EfficientNet-B0.

[0123] Radar / Laser Extraction Unit: Used to extract local spatial features from radar range-Doppler images and lidar 2D histograms via ResNet18's conv1→bn1→relu→maxpool→layer1.

[0124] GPS Feature Upscaling Unit: GPS relative angle information is upscaled to 192 dimensions via MLP.

[0125] Independent linear projection unit: The feature input for each modality is first unified to the same dimension through an independent linear projection layer.

[0126] Sequence encoding unit: used to concatenate all modal features into a whole sequence, and the whole sequence is combined with modality type encoding and time position encoding.

[0127] Attention interaction unit: Used to model the overall sequence using Multi-Head Self-Attention to achieve information interaction between modalities.

[0128] Output beam index unit: used to output the fused feature vector obtained by global pooling of the output sequence, calculate the beam probability distribution through multilayer perceptron and Softmax, and output the predicted beam index.

[0129] The communication control module is used to adjust the beam pointing of the 60GHz millimeter-wave phased array of the receiver vehicle in the corresponding direction according to the beam index, so that it is aligned with the 60GHz quasi-omnidirectional antenna of the transmitter vehicle. It also optimizes the beam pointing through a power verification mechanism to establish and maintain a communication link to complete the task of sharing road environment information between the two vehicles. If communication needs to continue, it returns to the multimodal sensing module to re-execute the data acquisition and preprocessing process; otherwise, it maintains the link and waits for the next communication task.

[0130] Beam adjustment unit: Used to control the receiver's four millimeter-wave phased arrays to switch to the corresponding direction according to the predicted beam index, so as to achieve initial beam alignment.

[0131] Power verification unit: used for the Top-K predicted beam directions in the output codebook of the beam prediction model, where K is an integer and is dynamically adjusted according to the number of phased array beams. In the predicted Top-K beam set, the actual communication link received power of each beam is measured in sequence, and the beam with the highest received power is selected as the final alignment direction.

[0132] Communication Judgment Unit: Used to trigger the communication process according to preset communication conditions: when the distance between the two vehicles is less than 100m and they are in a front-to-back / parallel position, or when the transmitter vehicle is detected to be about to change lanes / turn, or when the vehicle-mounted task needs to be synchronized (cooperative driving, path negotiation), or when the historical link quality deteriorates, communication is initiated.

[0133] The time-space synchronization module is used to align the timestamps of all data based on the power sampling time of the 60GHz millimeter-wave phased array and to interpolate and compensate for GPS data delays.

[0134] Example 3 Based on Embodiment 1, Embodiment 3 of the present invention also provides a processing device, including at least one processor and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor can execute the method steps of Embodiment 1 by calling the program instructions.

[0135] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A beam prediction method for vehicular communication based on a multimodal model, characterized in that, include: S1. The receiver vehicle uses its deployed multimodal sensors to collect 180° RGB images of the front and rear of the vehicle, raw FMCW radar data in the front, rear, left, and right directions, and 3D lidar point clouds in parallel. At the same time, it acquires GPS data through the GPS sensors of each group of the receiver and transmitter vehicles. All the collected data are preprocessed and the sensors are calibrated. The timestamps of all data are aligned using a unified system clock and the sampling time of the millimeter-wave power as the reference. S2. Cache all preprocessed data in chronological order for 5 consecutive frames to form a 1-second observation window. Select a beam prediction model from the preset model library based on the current driving environment. Input all data in the observation window into the selected beam prediction model to predict the optimal beam direction of the receiver vehicle to the transmitter vehicle within the next 500ms and output the beam index. S3. According to the beam index, adjust the beam pointing of the 60GHz millimeter-wave phased array of the receiver vehicle in the corresponding direction so that it is aligned with the 60GHz quasi-omnidirectional antenna of the transmitter vehicle, and optimize the beam pointing through the power verification mechanism to establish and maintain the communication link in order to complete the task of sharing road environment information between the two vehicles. If communication needs to continue, return to S1 to re-execute the data acquisition and preprocessing process; otherwise, maintain the link and wait for the next communication task.

2. The method for beam prediction in vehicle communication based on a multimodal model according to claim 1, characterized in that, Prior to performing S1, a device deployment step is also included: The transmitter vehicle is equipped with a 60GHz quasi-omnidirectional antenna and a GPS sensor. The receiver vehicle is equipped with an FMCW radar and a 60GHz millimeter-wave phased array in the left, right, front and rear directions, respectively. It is also equipped with two RGB cameras (180° front and 180° rear), one 3D LiDAR and one GPS sensor.

3. The method for beam prediction in vehicle communication based on a multimodal model according to claim 1, characterized in that, The preprocessing includes: The MIRNet-v2 model was used to enhance brightness and restore details in the 180° RGB images of the front and rear of the vehicle. Two-dimensional Fourier transforms were performed on the raw FMCW radar data from the four directions to generate range-Doppler maps, and the range-Doppler maps from the four directions were then stitched together. The 3D LiDAR point cloud is cleaned, downsampled, and mapped to a two-dimensional histogram; The latitude and longitude coordinates of the GPS data from the receiver and transmitter vehicles are converted into relative angle information between them.

4. The method for beam prediction in vehicle communication based on a multimodal model according to claim 1, characterized in that, The beam prediction model selected from the preset model based on the current driving environment is specifically as follows: If real-time vehicle speed And the density of point clouds of surrounding obstacles High-speed scene model is selected; If real-time vehicle speed And the density of point clouds of surrounding obstacles The urban traffic congestion model was selected. In all other cases, the default model is used. and The preset vehicle speed threshold, and This is the preset threshold for the point cloud density of surrounding obstacles.

5. The method for beam prediction in vehicle communication based on a multimodal model according to claim 1, characterized in that, The beam prediction model adopts the Transformer architecture, and its feature extraction process includes: Spatial texture features of RGB images are extracted using Layer 1-Layer 6 of EfficientNet-B0; The range-Doppler map of the FMCW radar and the two-dimensional histogram of the 3D LiDAR were used to extract local spatial features through ResNet18's conv1→bn1→relu→maxpool→layer1. GPS relative angle information is upscaled to 192 dimensions using MLP.

6. The method for beam prediction in vehicle communication based on a multimodal model according to claim 5, characterized in that, The specific process of fusing multimodal features using the Transformer architecture in the beam prediction model is as follows: The features of each modality are unified to the same dimension through independent linear projection layers; All modal features are concatenated into a single sequence, and the single sequence is then combined with modality type encoding and temporal location encoding. Multi-Head Self-Attention is used to model the overall sequence and enable information exchange between modalities. The output sequence is global pooled to obtain a fused feature vector, which is then processed by a multilayer perceptron and Softmax to calculate the beam probability distribution and output the predicted beam index.

7. The method for beam prediction in vehicle communication based on a multimodal model according to claim 1, characterized in that, The sensor calibration includes: Joint calibration of LiDAR and camera: In static scenes, a checkerboard calibration board is used to simultaneously acquire LiDAR point cloud and camera images. The two-dimensional coordinates of the corner points of the calibration board in the image coordinate system and the three-dimensional coordinates in the point cloud coordinate system are extracted. The extrinsic parameter matrix is ​​optimized through a reprojection error minimization strategy to achieve spatial alignment of the coordinate systems of the two sensors. LiDAR and FMCW radar calibration: Using corner reflectors, the echo signal of FMCW radar is matched with the corresponding reflection point in the lidar point cloud. The external parameter relationship is solved through coordinate transformation to complete the spatial calibration. GPS calibration: Corrects GPS drift errors in real time during dynamic driving using SLAM technology; Millimeter-wave power calibration: The reference power from the transmitter vehicle to the transmitter's 60 GHz quasi-omnidirectional antenna is measured in an interference-free environment, while the receiver vehicle monitors the beam power vector in real time and normalizes it to the [0,1] range.

8. The method for beam prediction in vehicle communication based on a multimodal model according to claim 1, characterized in that, The power verification mechanism is as follows: The beam prediction model outputs Top-K predicted beam directions in the codebook, where K is a positive integer and is dynamically adjusted according to the number of phased array beams. In the predicted Top-K beam set, the actual communication link received power of each beam is measured in sequence, and the beam with the highest received power is selected as the final alignment direction.

9. A vehicle-mounted communication beam prediction system based on a multimodal model, characterized in that, include: The receiver vehicle is equipped with: two RGB cameras, used to acquire 180° RGB images from the front and rear of the vehicle respectively; one 3D LiDAR, used to acquire 3D LiDAR point clouds of the surrounding environment of the vehicle; and four FMCW radars, deployed in the front, rear, left and right directions of the vehicle respectively, used to acquire raw radar echoes. One GPS sensor is used to obtain the location of the receiver vehicle; four 60GHz millimeter-wave phased arrays are deployed in the front, rear, left and right directions facing the vehicle for beam adjustment. The transmitter vehicle is equipped with: one 60GHz quasi-omnidirectional antenna for transmitting communication signals; and one GPS sensor for obtaining the position of the transmitter vehicle. Multimodal perception module: used to acquire 180° RGB images of the front and rear of the vehicle, raw FMCW radar data in the front, rear, left and right directions, and 3D lidar point cloud in parallel; at the same time, it acquires GPS data through GPS sensors of each group of receiver and transmitter vehicles, and performs preprocessing and sensor calibration on all acquired data. Model selection module: used to cache all preprocessed data in chronological order for 5 consecutive frames to form a 1-second observation window; select the corresponding beam prediction model from the preset model library according to the current driving environment; Beam prediction module: Used to input all data in the observation window into the selected beam prediction model, predict the optimal beam direction of the receiver vehicle to the transmitter vehicle within the next 500ms, and output the beam index; Communication control module: Used to adjust the beam pointing of the 60GHz millimeter-wave phased array of the receiver vehicle in the corresponding direction according to the beam index, so that it is aligned with the 60GHz quasi-omnidirectional antenna of the transmitter vehicle, and optimize the beam pointing through the power verification mechanism to establish and maintain the communication link in order to complete the task of sharing road environment information between the two vehicles. If communication needs to continue, return to the multimodal sensing module to re-execute the data acquisition and preprocessing process; otherwise, maintain the link and wait for the next communication task. The time-space synchronization module is used to synchronize all data with a unified system clock, using the millimeter-wave power sampling time as a reference to complete the timestamp alignment.

10. A processing apparatus, characterized in that, The method includes at least one processor and at least one memory communicatively connected to the processor, wherein the memory stores program instructions executable by the processor, and the processor can execute the method as described in any one of claims 1 to 8 by invoking the program instructions.

Images