A vehicle network channel prediction method based on multimodal fusion and related equipment

Abstract

The present invention relates to the field of vehicle network communication technology, and discloses a vehicle network channel prediction method based on multimodal fusion and related equipment. The method includes constructing an urban road scene of the vehicle network channel, obtaining known time frame data and unknown time frame data to be predicted in the urban road scene in combination with the weather scene; performing data enhancement and data vector representation and position information processing to obtain a multimodal vector, performing feature extraction and splicing processing on the multimodal vector to obtain a multimodal feature vector; performing deep inter-modal fusion on the multimodal feature vector to obtain a joint feature vector; performing nonlinear mapping on the joint feature vector through a multi-layer perceptron module, and outputting a channel index probability distribution, and predicting the vehicle network channel with the channel index probability distribution. The present invention solves the technical problems existing in the prior art, such as limited single-modal perception capability, poor quality of original data, and insufficient robustness in complex scenarios.

Description

Technical Field

[0001] The present invention relates to the field of vehicle networking communication technology, and specifically to a vehicle networking channel prediction method based on multimodal fusion and related equipment. Background Art

[0002] With the rapid development of connected vehicle (IoV) technology, the importance of channel prediction in improving the efficiency and reliability of vehicle-infrastructure cooperative communications has become increasingly prominent. Thanks to policy support and the gradual improvement of technical standards, IoV communication systems are moving towards ultra-high frequency bands and multimodal integration. With the ongoing deployment of 5G technology and the foresight of 6G communications, IoV is gradually acquiring the ability to support the efficient transmission and perception of multimodal data. Furthermore, the proliferation of intelligent sensors (such as LiDAR and millimeter-wave radar) and edge computing has increased the feasibility of incorporating multimodal perception data into communication system design, further promoting the research and application of channel prediction technology.

[0003] Currently, many studies have explored the potential and advantages of incorporating sensory data into channel prediction. Existing technologies for IoV channel prediction based on multimodal enhanced data primarily focus on the following aspects:

[0004] 1. Channel Prediction Method Based on Visual Sensor Data

[0005] Vision sensors (such as cameras) capture visible or infrared light in the environment and generate two-dimensional images or video data. Their technical characteristics are their ability to provide rich scene information, such as object shape, color, and texture, making them suitable for perception in both static and dynamic environments. In channel prediction, environmental features in the data (such as buildings, vegetation, and vehicles) are often used to infer signal propagation paths and attenuation characteristics.

[0006] 2. Channel Prediction Method Based on Optical Sensor Data

[0007] Optical sensors (such as LiDAR) generate high-precision 3D point cloud data by emitting laser beams and receiving reflected signals. Their technical characteristics are their ability to provide precise distance and shape information, making them suitable for high-resolution environmental modeling. In channel prediction, environmental structures (such as buildings, obstacles, and terrain) in point cloud data are often used to model signal propagation paths and reflection characteristics.

[0008] 3. Channel Prediction Method Based on Positioning Data

[0009] Positioning data typically comes from a Global Positioning System (GPS), an Inertial Measurement Unit (IMU), or a fused positioning system, providing information on the device's location, velocity, and orientation. Its technical characteristics are its ability to provide precise position and motion information, making it suitable for channel prediction in dynamic environments. Channel prediction typically uses the device's location and motion trajectory, combined with map data or environmental models, to infer signal propagation paths and dynamic changes.

[0010] 4. Channel Prediction Method Based on Modal Fusion

[0011] Modal fusion methods integrate heterogeneous data from multiple sources, such as vision, optics, and positioning, and leverage the complementary properties of different modalities to improve the robustness and accuracy of channel prediction. Its technical characteristics lie in its use of a deep learning framework to achieve feature alignment and joint modeling of heterogeneous data.

[0012] However, different multimodal enhanced data still have limitations in terms of environmental adaptability and diversity. First, the perception ability of visual sensors is significantly reduced in low light, extreme weather or occluded scenes, and they cannot directly provide distance information. They need to rely on complex algorithms for indirect deduction, which increases computational complexity and uncertainty. Second, optical sensors are easily affected by weather and reflective surface characteristics. The large amount of data, high computing resource requirements and expensive hardware costs limit their large-scale application. Third, in occluded environments (such as urban canyons, tunnels, etc.) or indoor scenes, GPS positioning accuracy may be greatly reduced or even fail due to the obstruction or attenuation of satellite signals. Fourth, existing research has not effectively integrated multi-source heterogeneous information, resulting in limited adaptability to diverse traffic conditions. Summary of the Invention

[0013] In order to overcome the defects of the above-mentioned prior art, the purpose of the present invention is to provide a vehicle network channel prediction method and related equipment based on multimodal fusion, so as to solve the technical problems existing in the prior art such as limited single-modal perception capability, poor quality of original data, and insufficient robustness in complex scenarios.

[0014] The present invention is achieved through the following technical solutions:

[0015] In a first aspect, the present invention provides a vehicle network channel prediction method based on multimodal fusion, comprising:

[0016] Construct an urban road scenario for the Internet of Vehicles channel, and combine the weather scenario with the urban road scenario to obtain known time frame data and unknown time frame data to be predicted;

[0017] The known time frame data and the unknown data to be predicted are subjected to data enhancement, data vector representation, and position information processing to obtain a multimodal vector, and the multimodal vector is subjected to feature extraction and splicing processing to obtain a multimodal feature vector;

[0018] The multimodal feature vector is input into the Transformer model, and the deep fusion between modalities is performed through the Transformer model's multi-head attention mechanism and feedforward neural network to obtain a joint feature vector;

[0019] The joint feature vector is nonlinearly mapped through a multi-layer perceptron module, and the channel index probability distribution is output to predict the Internet of Vehicles channel.

[0020] Preferably, in the urban road scene, the known time frame data and the unknown time frame data to be predicted are obtained in combination with the weather scene, and the weather scene includes a sunny scene, a rainy scene, and a snowy scene;

[0021] The known time frame data includes channel index value, communication signal time frame sequence number, vehicle and RSU position coordinates, RGB image, LiDAR point cloud;

[0022] The unknown time frame data to be predicted includes the communication signal time frame sequence number, the position coordinates of the vehicle and RSU, the RGB image, and the LiDAR point cloud.

[0023] Preferably, data enhancement includes image enhancement and point cloud enhancement;

[0024] Among them, image enhancement includes brightness adjustment, contrast enhancement, gamma correction, Gaussian blur and background masking;

[0025] The point cloud enhancement process involves filtering the background by referencing a 2D bird's-eye view, mapping building locations to the point cloud data, removing point clouds in the building location area, and concentrating the remaining point clouds in areas of vehicle motion and road structure to reduce interference from static objects on the model.

[0026] The data vector representation includes an image vector representation and a point cloud vector representation;

[0027] The image vector representation process includes using RGB three-channel format to represent the image data as a vector and then normalizing it as the model input.

[0028] The process of vector representation of point clouds includes first dividing the area of ​​interest into grids to achieve effective vector representation of point cloud data; limiting the range of the point cloud by setting the area of ​​interest, and counting the density information within each grid as a vector representation; secondly, setting a maximum density threshold. When the number of points in a grid exceeds the maximum density threshold, it is fixed to the maximum density threshold to suppress the interference of outliers on the vector representation and model performance; finally, the number of points in all grids is normalized and used as model input;

[0029] The position information processes the spatial consistency of multimodal data, aligns them through a unified coordinate system standardization method, and is used as a model input after normalization.

[0030] Preferably, the specific process of extracting and concatenating the multimodal vectors to obtain the multimodal feature vectors is as follows:

[0031] The multimodal feature vectors are respectively input into the ResNet34 module and the ResNet18 module for feature extraction, and the multimodal vectors are spliced ​​to obtain the multimodal feature vector.

[0032] Preferably, the Transformer model includes an encoder and a decoder. The encoder performs multi-level abstraction and expression on the input multimodal feature vector through a multi-head attention mechanism and a feedforward neural network, and then combines the output with the target context through the decoder to obtain a joint feature vector.

[0033] Furthermore, the specific process of the multi-head attention mechanism is as follows:

[0034] Project the input multimodal feature vector into multiple subspaces, calculate the attention weights based on the multiple subspaces, and establish the relationship between multiple feature dimensions;

[0035] Based on the relationship between multiple feature dimensions, each attention is independently generated into query, key and value vectors, and the weight is calculated by scaling the dot product attention. The multi-head results are spliced ​​into a unified output according to the obtained weight;

[0036] Among them, scaled dot product attention scales the multi-head results by measuring the correlation through the dot product of the query and key vectors to avoid numerical instability caused by high-dimensional features.

[0037] Preferably, the joint feature vector is nonlinearly mapped through a multi-layer perceptron module, wherein the multi-layer perceptron module includes multiple fully connected layers and a Softmax activation function for dimensionality reduction and outputting a normalized probability distribution; the calculation formula of the Softmax activation function includes:

[0038]

[0039] Here, x is a vector of length K and the output is a probability distribution.

[0040] In a second aspect, the present invention further provides a vehicle network channel prediction system based on multimodal fusion, comprising:

[0041] The multimodal raw data acquisition module is used to construct the urban road scene of the Internet of Vehicles channel. In the urban road scene, the known time frame data and the unknown time frame data to be predicted are acquired in combination with the weather scene.

[0042] The data processing and conversion module is used to perform data enhancement and data vector representation and position information processing on the known time frame data and the unknown data to be predicted to obtain a multimodal vector, and perform feature extraction and splicing processing on the multimodal vector to obtain a multimodal feature vector;

[0043] The model processing module is used to input the multimodal feature vector into the Transformer model and perform deep fusion between the modalities through the Transformer model's multi-head attention mechanism and feedforward neural network to obtain a joint feature vector;

[0044] The channel prediction module is used to perform nonlinear mapping on the joint feature vector through the multi-layer perceptron module, and output the channel index probability distribution, and use the channel index probability distribution to predict the Internet of Vehicles channel.

[0045] In a third aspect, the present invention also provides a mobile terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the vehicle network channel prediction method based on multimodal fusion as described above is implemented.

[0046] In a fourth aspect, the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the vehicle network channel prediction method based on multimodal fusion as described above.

[0047] Compared with the prior art, the present invention has the following beneficial technical effects:

[0048] The present invention provides a vehicle network channel prediction method based on multimodal fusion. By constructing urban road scenarios and combining them with weather scenarios to obtain multimodal raw data, the robustness of channel prediction in multiple scenarios is improved through the efficient fusion capability of the Transformer model and the optimization of data enhancement strategies under different traffic densities and various weather conditions. The quality of the raw data is significantly improved through the preprocessing process, providing reliable input for the model's high-precision prediction. Based on the Transformer model, deep fusion and dynamic modeling of multimodal data are achieved. The computational efficiency of the model is significantly improved through parallel computing characteristics, enabling it to quickly process large-scale multimodal data and providing technical support for real-time channel prediction. In network slicing, the present invention solves the dynamic modeling problem of wireless channels at the access network (AN) level. By constructing urban road scenarios and combining them with weather scenarios to obtain multimodal data, data enhancement, feature extraction, and deep fusion are performed to achieve accurate prediction of channel status. The present invention is connected with access network security authentication to jointly serve the integrity of the end-to-end architecture of the slice, ensuring the stability and security of the network slice in the access link.

[0049] Furthermore, the present invention uses the Transformer model to perform multimodal data fusion, significantly improving the accuracy of channel prediction. Traditional single-modal methods often perform poorly in complex scenarios. For example, visual information cannot capture depth, lidar data lacks semantic understanding, and position information is difficult to independently reflect dynamic changes. The Transformer model fully utilizes the advantages of each modality through multimodal data fusion: visual information provides rich spatial details, lidar data supplements depth and speed information, and position information dynamically reflects scene changes. This fusion method overcomes the limitations of single-modal methods and makes channel predictions more accurate and reliable.

[0050] Furthermore, the present invention effectively solves the problem of insufficient raw data quality through data enhancement strategies, providing higher quality input for channel prediction. Raw multimodal data often have quality defects. For example, images are easily affected by lighting and background interference, the lidar point cloud has a lot of noise, and the position information lacks consistency. These problems directly affect the effect of model training and prediction accuracy. To address these deficiencies, the present invention adopts a series of optimization measures: image quality improvement and background masking to reduce irrelevant interference and improve image quality; masking, downsampling and noise filtering of the lidar point cloud to enhance the robustness of the data; and unified coordinate alignment of the position information to ensure the spatial consistency of multimodal data. Through these optimizations, the present invention effectively solves the problem of poor raw data quality and provides cleaner and more consistent input data for the model.

[0051] Furthermore, the present invention systematically experiments the proposed channel prediction method under various weather conditions and traffic density. Traditional channel prediction models are typically tested in idealized environments, ignoring the variability of real-world application scenarios, resulting in significantly reduced performance in complex environments. The present invention designed an experimental environment encompassing different weather conditions (such as sunny, rainy, and snowy) and traffic densities (such as medium and high density), comprehensively verifying the robustness and stability of the model under various real-world conditions. The experimental results demonstrate that the proposed method maintains excellent prediction performance across multiple scenarios, fully demonstrating its robustness in complex real-world environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Flowchart of a method for predicting a vehicle network channel based on multimodal fusion in an embodiment of the present invention;

[0053] Figure 2 This is a V2I channel prediction scene diagram in an embodiment of the present invention;

[0054] Figure 3 This is a diagram of a V2I channel prediction algorithm model in an embodiment of the present invention;

[0055] Figure 4 This is a diagram showing multimodal raw data in an embodiment of the present invention;

[0056] Figure 5 This is a diagram showing the RGB image data enhancement processing in an embodiment of the present invention;

[0057] Figure 6 This is a diagram showing the LiDAR point cloud data enhancement processing in an embodiment of the present invention;

[0058] Figure 7 This is a visualization diagram of the process of inputting multimodal data into the Transformer in an embodiment of the present invention;

[0059] Figure 8 Schematic diagram of the Transformer infrastructure and multi-head attention mechanism in an embodiment of the present invention;

[0060] Figure 9 This is a schematic diagram showing the comparison of the accuracy values ​​of different experimental schemes under different weather conditions and traffic volumes in an embodiment of the present invention;

[0061] Figure 10 Schematic diagram of a vehicle network channel prediction system based on multimodal fusion in an embodiment of the present invention;

[0062] In the figure: 1. Multimodal raw data acquisition module; 2. Data processing and conversion module; 3. Model processing module; 4. Channel prediction module. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0064] The purpose of the present invention is to provide a vehicle network channel prediction method and related equipment based on multimodal fusion to solve the technical problems existing in the prior art, such as limited single-modal perception capability, poor original data quality, and insufficient robustness in complex scenarios.

[0065] The terms involved in this invention are explained as follows:

[0066] 1. Transformer model: self-attention transformation model;

[0067] 2. RGB image: red, green, and blue image, Red-Green-Blue image;

[0068] 3. LiDAR point cloud: LiDAR point cloud, Light Detection And Ranging point cloud;

[0069] 4: ResNet module: Residual Network module, Residual Network module;

[0070] 5: GPS: Global Positioning System;

[0071] 6: IMU: Inertial Measurement Unit;

[0072] 7: V2I communication: Vehicle-to-Infrastructure communication;

[0073] 8: Air-Sim: Air-Sim is an open-source, cross-platform simulator based on a game engine that can be used for physical and visual simulation of robots such as drones and unmanned vehicles;

[0074] WaveFarer: WaveFarer is a high-fidelity radar simulator that accounts for multipath and scattering from structures and vehicles in the radar system's surrounding environment, as well as critical atmospheric and scattering effects at frequencies up to and beyond 100 GHz. Applications include simulating automotive driving scenarios, indoor sensors, and far-field radar cross-sections.

[0075] 10: Wireless InSite: Wireless InSite is a prediction tool for understanding wireless coverage, channel multipath, and data throughput for 5G, 6G, and WiFi networks. Using advanced accelerated 3D ray tracing and fast ray-based surrogacy methods, along with empirical models, it efficiently and accurately predicts multipath channel characteristics in indoor, urban, and rural environments. Dynamic scenario modeling of vehicles and pedestrians captures time-varying attenuation, while frequency sweeps account for broadband effects. Communication analysis capabilities apply multi-input, multi-output algorithms to channel prediction to estimate wireless network coverage and throughput.

[0076] The present invention is described in further detail below with reference to the accompanying drawings:

[0077] Example 1

[0078] See also Figure 1 In one embodiment of the present invention, a method for predicting a vehicle network channel based on multimodal fusion is provided, comprising:

[0079] Step 1: Construct an urban road scenario for the Internet of Vehicles channel, and obtain known time frame data and unknown time frame data to be predicted in the urban road scenario in combination with the weather scenario;

[0080] Specifically, according to Figure 2 As shown in the figure, the urban road scenarios of the constructed Internet of Vehicles channel cover experimental environments with different weather conditions (such as sunny, rainy, snowy, etc.) and traffic densities (such as medium density, high density, etc.), which fully verifies the robustness and stability of the model under various actual conditions.

[0081] In the medium-density scenario, the scene contains 10 cars, 3 buses, and 11 RSUs; in the high-density scenario, the number of vehicles increases to 15 cars and 6 buses, while the number of RSUs is kept at 11. The original data is collected by sensing devices in three weather scenarios: sunny, rainy, and snowy days. Each scenario has 1,000 time frames, including known time frames (75%) and unknown data to be predicted (25%). Each known time frame contains the top three channel index values ​​of the signal-to-noise ratio of V2I communication, the communication signal time frame sequence number, the position coordinates of the vehicle and RSU, RGB images, and LiDAR point clouds. Each unknown time frame to be predicted only contains the communication signal time frame sequence number, the position coordinates of the vehicle and RSU, RGB images, and LiDAR point clouds, such as Figure 4 shown.

[0082] Step 2: Perform data enhancement, data vector representation, and position information processing on the known time frame data and the unknown data to be predicted to obtain a multimodal vector, and then perform feature extraction and splicing on the multimodal vector to obtain a multimodal feature vector;

[0083] Specifically, according to Figure 3 As shown, the multimodal original data is preprocessed to obtain multimodal enhanced data, and the preprocessing process includes image enhancement, image vector representation, point cloud enhancement, point cloud vector representation and preprocessing of position information.

[0084] Specifically, according to Figure 5 As shown in the figure, the specific image enhancement process is as follows: Image improvement techniques include brightness adjustment, contrast enhancement, gamma correction, Gaussian blurring, and background masking. These processing methods can significantly improve image clarity and consistency in different weather conditions (such as sunny, rainy, and snowy days). The main purpose of background masking is to mask buildings and other irrelevant background information to reduce the interference of invalid features on the model, thereby enhancing the model's focus on roads, vehicles, and dynamic traffic targets. If a pixel area is completely within the masked area, it is directly cropped to further reduce the computational burden and optimize data quality.

[0085] Specifically, the image vector representation process is as follows: image data is represented as a vector using a three-channel RGB format and normalized before use as model input. The data for each channel is normalized to reduce the impact of numerical differences between channels on model training, thereby improving training stability and effectiveness. During normalization, each channel is processed using a commonly used fixed value of [0.229, 0.224, 0.225]. This approach not only adapts to common models but also provides a more stable input data distribution for subsequent feature extraction. The processed RGB data is input into the model as the basic data representation for further feature learning, effectively supporting channel prediction tasks in complex scenarios.

[0086] The specific process of point cloud enhancement is as follows: background filtering maps the building locations to the point cloud data by referring to the two-dimensional bird's-eye view, and removes the point clouds in these areas to ensure that the retained point cloud information is mainly concentrated in key areas such as vehicle movement and road structure, thereby effectively reducing the interference of static objects on the model. In addition, in order to reduce the data scale while maintaining the integrity of the point cloud geometric features, the study introduces a downsampling operation to reduce the point cloud density by 20%, improve data processing efficiency and reduce computational overhead. At the same time, a random noise perturbation between -0.4 and 0.4 is appropriately introduced to simulate the measurement uncertainty in actual scenes and enhance the robustness of the model, such as Figure 6 shown.

[0087] The specific process of point cloud vector representation is as follows: the area of ​​interest is divided into grids to achieve effective vector representation of point cloud data. The point cloud range is limited by setting the area of ​​interest, and the density information in each grid is counted and represented as a vector. In this process, in order to prevent outliers caused by too many local points, a maximum density threshold is set. When the number of points in the grid exceeds the threshold, it is fixed to the threshold to suppress the interference of outliers on the vector representation and model performance. Finally, the number of points in all grids is normalized to ensure that the vector representation has a consistent data distribution, thereby improving the convergence and prediction performance of the model. The above steps effectively convert point cloud data into usable vectors, laying a data foundation for subsequent multimodal fusion and dynamic modeling.

[0088] Position information preprocessing focuses on the spatial consistency of multimodal data, aligning it through a unified coordinate system normalization method. Specifically, the vehicle and RSU positions are mapped to a coordinate system with the RSU as the origin, and the coordinates are normalized based on the maximum detection range of the sensing device. Normalization divides the actual coordinate values ​​by the maximum detection range of the sensing device, aligning the data to a range of 0 to 1 and eliminating the influence of absolute coordinate values ​​on model training.

[0089] After all multimodal augmented data are converted to vector representation, the RGB image vector is input to the ResNet34 module, and the LiDAR point cloud vector is input to the ResNet18 module for feature extraction, and then the multimodal vectors are spliced. Through the residual connection mechanism, the ResNet module can effectively capture local and global characteristics, while gradually compressing high-dimensional feature representations, making them more compact and having stronger semantic expression capabilities, such as Figure 7 shown.

[0090] Step 3: Input the multimodal feature vector into the Transformer model, and perform deep fusion between modalities through the multi-head attention mechanism and feedforward neural network of the Transformer model to obtain the joint feature vector, as shown in Figure 8 As shown;

[0091] Specifically, according to Figure 3As shown in the figure, the Transformer first performs a linear projection on the input features to generate a query vector (Q), a key vector (K), and a value vector (V). Subsequently, the attention weight is calculated through the dot product operation between the query vector and the key vector, and the value vector is weighted summed based on the weight, thereby achieving deep fusion of modal features and aggregation of related information. This multimodal fusion process is iterated 4 times in the Transformer model, and each iteration further improves the abstract expression ability of the features and the depth of interactive information. Ultimately, the fused RGB image, LiDAR point cloud, and coordinate features are represented as a 512×1 joint feature vector to form an enhanced multimodal feature representation.

[0092] In step 4, the joint feature vector is nonlinearly mapped through a multi-layer perceptron module, and the channel index probability distribution is output to predict the IoV channel.

[0093] Specifically, according to Figure 3 As shown in Figure 2, during the channel prediction phase, the joint feature vector undergoes nonlinear mapping via a Multi-Layer Perceptron (MLP) module. The MLP module consists of multiple fully connected layers and a Softmax activation function for dimensionality reduction and outputting a normalized probability distribution.

[0094] The calculation formula of the Softmax activation function includes:

[0095]

[0096] Here, x is a vector of length K and the output is a probability distribution.

[0097] Ultimately, the algorithm maps the joint feature vector into a 128×1 channel index probability distribution, representing the predicted target channel. Specifically, the channel index divides the 180-degree plane between the RSU and the road into 128 evenly spaced sections, with the channel index representing one of the directions. For example, index values ​​of 0 and 127 represent the leftmost and rightmost directions, respectively. The channel index accurately represents the communication direction, further improving communication system performance.

[0098] In order to verify the beneficial effect of the proposed Transformer model in the channel prediction task, based on the public dataset A series of simulation experiments were conducted to compare the present embodiment with Solution 1 using position information alone and Solution 2 using position information and image data.

[0099] Data source: The dataset uses Air-Sim and WaveFarer to collect multimodal perception data, and Wireless InSite to collect communication data. The dataset achieves a communication perception integrated system by deeply integrating and precisely aligning AirSim, WaveFarer, and WirelessInSite. The dataset also covers various weather conditions, multiplex frequency bands, and times of the day.

[0100] Hardware environment and software platform: The hardware environment uses Intel Xeon W-3235 12-core processor and RTX 3090 GPU, and the programming language uses Python 3.9.

[0101] In terms of dataset selection and experimental setup, to ensure the reliability of experimental results and the scientific nature of model evaluation, the dataset was divided into training, validation, and test sets. Specifically, the training set accounted for 72%, the validation set accounted for 8%, and the test set accounted for 20%. During the model training phase, the training set was used for model optimization, while the validation set was used to monitor model performance and perform hyperparameter tuning to avoid overfitting. During the testing phase, the test set was used to conduct a final assessment of the model's generalization performance, thereby verifying its predictive power on unseen data.

[0102] To simulate real-world connected vehicle scenarios, different vehicle densities and RSU distributions were set to reflect typical urban road communication environments. In the medium-density scenario, the vehicle population consisted of 10 cars, 3 buses, and 11 RSUs. In the high-density scenario, the vehicle population increased to 15 cars and 6 buses, while maintaining the same 11 RSUs. These two density scenarios enabled a comprehensive evaluation of the model's adaptability to varying traffic conditions. Furthermore, to simulate realistic road conditions, two weather scenarios, rainy and snowy, were introduced. In the rainy scenario, the rainfall was set to 50 mm / hr and the road humidity was 100%. In the snowy scenario, the snowfall was set to 10 mm / hr and the road humidity was also 100%. Furthermore, to more accurately reflect the impact of inclement weather on channel propagation, the experiment also modeled the physical and electromagnetic space parameters under different weather conditions. For example, in the rainy scenario, the ambient temperature was set to 22.2°C and the humidity was 100%. In the snowy scenario, the temperature was set to -10°C and the humidity was 20%.

[0103] In the communication parameter setting, the experiment adopted an integrated design of communication and perception to ensure the accuracy of channel prediction and the efficiency of model training. Specifically, each RSU and vehicle is equipped with 128 and 32 antenna units, respectively, to support high-precision beamforming and channel prediction. In terms of frequency band selection, the experiment simulated the communication performance of the Sub-6 GHz band and the millimeter wave band at the same time. In the Sub-6 GHz band, the carrier frequency is set to 5.9 GHz and the communication bandwidth is 20 MHz; in the millimeter wave band, the carrier frequency is set to 28 GHz and the communication bandwidth is 2 GHz. Through a comprehensive comparison of the communication performance of different frequency bands, the experiment further verified the adaptability and performance of the model in a multi-band environment.

[0104] The simulation experiment parameter settings are summarized in Table 1:

[0105] Table 1 Channel prediction simulation parameter settings based on the Transformer model

[0106]

[0107] according to Figure 9 As shown in the figure, the channel prediction accuracy of three schemes under different weather and traffic flow scenarios is compared, including comparison scheme 1, comparison scheme 2 and the multimodal method of the present invention. The experimental results show that the performance of scheme 1, which uses location information alone, is limited in complex scenarios. For example, the accuracy is only 0.58 in medium-density scenarios on snowy days, and only 0.63 in high-density scenarios on sunny days. When combined with camera features, the accuracy of scheme 2 is improved, for example, from 0.69 to 0.78 in medium-density scenarios on sunny days. The scheme proposed in this paper not only achieves deep feature fusion based on multimodal data, but also combines targeted data enhancement strategies, so that the model shows significant performance advantages in all scenarios. In the high-density scenario on sunny days, the accuracy reaches 0.91, and in the medium-density scenario on snowy days, the accuracy is even improved to 0.98. These results fully demonstrate the robustness and superiority of the V2I channel prediction strategy based on the Transformer model proposed in this paper under different traffic flows and severe weather conditions, and outperform other existing schemes.

[0108] Table 2 shows the runtime performance of channel prediction based on the Transformer model under different input data types and augmentation strategies. By analyzing the input data types and model runtime, we can comprehensively evaluate the model's real-time performance in multimodal data fusion and expansion scenarios. First, for the original input data, which includes vehicle position, RSU position, a single RGB image, and a LiDAR point cloud, the size of each sample is 5.31 MB. Under these conditions, the model's training time per sample is 0.399 ms, and its inference time per sample is 2.247 ms. In contrast, after data augmentation, the input data includes more RGB images (8) and LiDAR point clouds (2), reducing the size of each sample to 4.14 MB. This data compression is primarily due to the data augmentation strategy's optimization of information redundancy and reduction of sample complexity. The impact of data augmentation on the model's runtime performance is further analyzed. After data augmentation, the training time for a single sample increased slightly, from 0.399 ms for the original data to 0.466 ms, while the inference time for a single sample decreased from 2.247 ms to 2.073 ms. This phenomenon indicates that while the data augmentation strategy slightly increases training time, it significantly improves model inference efficiency and reduces computational overhead during inference, demonstrating particularly good performance in highly dynamic scenarios requiring real-time prediction. Experimental results demonstrate that by optimizing the input data structure, the data augmentation strategy effectively reduces inference time while ensuring data quality, providing strong support for efficient real-time prediction in multimodal data scenarios.

[0109] Table 2 Time performance of the channel prediction model based on the Transformer model

[0110]

[0111] In summary, the experimental results verify that this embodiment effectively overcomes the difficulties of multimodal heterogeneity and channel dynamics by fully exploiting the spatiotemporal correlation and semantic features in multimodal data, and achieves high precision and high robustness in channel prediction.

[0112] In summary, the Transformer model in this embodiment achieves deep fusion and dynamic modeling of multimodal data through its multi-head attention mechanism, position encoding capability and parallel computing characteristics. Among them, the multi-head attention mechanism can effectively capture the spatial characteristics of visual information, the depth and speed characteristics of lidar data, and the dynamic change characteristics of position data, providing a basis for the efficient fusion of multimodal data. The position encoding capability enhances the model's modeling ability for dynamic time series scenes by introducing sequence position information, enabling it to better adapt to changes in complex scenes. The parallel computing characteristics significantly improve the computational efficiency of the model, enabling it to quickly process large-scale multimodal data, providing technical support for real-time channel prediction.

[0113] For image data, the RGB image input quality is optimized through quality improvement and background masking techniques. Vector representation is performed in a three-channel RGB format and normalized before serving as model input. For LiDAR point cloud data, strategies such as background filtering, downsampling, and random noise perturbation are employed. The region of interest is divided into a grid during the vector representation process to achieve effective vector representation of the point cloud data. Position information is aligned using a unified coordinate system normalization method to ensure spatial consistency of multimodal data. These data enhancement strategies significantly improve the quality of the raw data and provide reliable input for the model's high-precision predictions.

[0114] This embodiment demonstrates excellent robustness in multiple scenarios, primarily due to the Transformer model's efficient fusion capabilities and optimized data augmentation strategies. Experiments covered varying traffic densities (e.g., medium density, high density, etc.) and various weather conditions (e.g., sunny, rainy, snowy, etc.). When traffic density is high, channel prediction accuracy can easily decline due to vehicle occlusion. However, this embodiment, through multimodal data fusion, fully leverages the complementary advantages of vision, lidar, and position information, effectively compensating for vehicle occlusion. Furthermore, for inclement weather, the data augmentation strategy significantly improves the quality of input data by optimizing image brightness and contrast and reducing denoising on the lidar point cloud, enabling the model to maintain stable performance in complex weather environments. These technical advantages collectively ensure the high robustness of this embodiment in multiple scenarios.

[0115] Example 2

[0116] according to Figure 10 As shown, the present invention also provides a vehicle network channel prediction system based on multimodal fusion, including:

[0117] Multimodal raw data acquisition module 1 is used to construct an urban road scene for the Internet of Vehicles channel, and obtain known time frame data and unknown time frame data to be predicted in the urban road scene in combination with the weather scene;

[0118] Data processing and conversion module 2 is used to perform data enhancement and data vector representation and position information processing on the known time frame data and the unknown data to be predicted to obtain a multimodal vector, and perform feature extraction and splicing processing on the multimodal vector to obtain a multimodal feature vector;

[0119] Model processing module 3 is used to input the multimodal feature vector into the Transformer model, and perform deep fusion between the modalities through the Transformer model's multi-head attention mechanism and feedforward neural network to obtain a joint feature vector;

[0120] The channel prediction module 4 is used to perform nonlinear mapping on the joint feature vector through the multi-layer perceptron module, and output the channel index probability distribution, and predict the vehicle network channel based on the channel index probability distribution.

[0121] Example 3

[0122] The present invention also provides a mobile terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, such as a vehicle network channel prediction program based on multimodal fusion.

[0123] When the processor executes the computer program, the above-mentioned vehicle network channel prediction method based on multimodal fusion is implemented, for example:

[0124] Construct an urban road scenario for the Internet of Vehicles channel, and combine the weather scenario with the urban road scenario to obtain known time frame data and unknown time frame data to be predicted;

[0125] The known time frame data and the unknown data to be predicted are subjected to data enhancement, data vector representation, and position information processing to obtain a multimodal vector, and the multimodal vector is subjected to feature extraction and splicing processing to obtain a multimodal feature vector;

[0126] The multimodal feature vector is input into the Transformer model, and the deep fusion between modalities is performed through the Transformer model's multi-head attention mechanism and feedforward neural network to obtain a joint feature vector;

[0127] The joint feature vector is nonlinearly mapped through a multi-layer perceptron module, and the channel index probability distribution is output to predict the Internet of Vehicles channel.

[0128] Alternatively, when the processor executes the computer program, the functions of each module in the above system are realized, for example:

[0129] Multimodal raw data acquisition module 1 is used to construct an urban road scene for the Internet of Vehicles channel, and obtain known time frame data and unknown time frame data to be predicted in the urban road scene in combination with the weather scene;

[0130] Data processing and conversion module 2 is used to perform data enhancement and data vector representation and position information processing on the known time frame data and the unknown data to be predicted to obtain a multimodal vector, and perform feature extraction and splicing processing on the multimodal vector to obtain a multimodal feature vector;

[0131] Model processing module 3 is used to input the multimodal feature vector into the Transformer model, and perform deep fusion between the modalities through the Transformer model's multi-head attention mechanism and feedforward neural network to obtain a joint feature vector;

[0132] The channel prediction module 4 is used to perform nonlinear mapping on the joint feature vector through the multi-layer perceptron module, and output the channel index probability distribution, and predict the vehicle network channel based on the channel index probability distribution.

[0133] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the mobile terminal.

[0134] For example, the computer program may be divided into a multimodal raw data acquisition module 1, a data processing and conversion module 2, a model processing module 3, and a channel prediction module 4;

[0135] The specific functions of each module are as follows:

[0136] Multimodal raw data acquisition module 1 is used to construct an urban road scene for the Internet of Vehicles channel, and obtain known time frame data and unknown time frame data to be predicted in the urban road scene in combination with the weather scene;

[0137] Data processing and conversion module 2 is used to perform data enhancement and data vector representation and position information processing on the known time frame data and the unknown data to be predicted to obtain a multimodal vector, and perform feature extraction and splicing processing on the multimodal vector to obtain a multimodal feature vector;

[0138] Model processing module 3 is used to input the multimodal feature vector into the Transformer model, and perform deep fusion between the modalities through the Transformer model's multi-head attention mechanism and feedforward neural network to obtain a joint feature vector;

[0139] The channel prediction module 4 is used to perform nonlinear mapping on the joint feature vector through the multi-layer perceptron module, and output the channel index probability distribution, and predict the vehicle network channel based on the channel index probability distribution.

[0140] The mobile terminal may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The mobile terminal may include, but is not limited to, a processor and a memory.

[0141] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the mobile terminal and connects various parts of the entire mobile terminal using various interfaces and lines.

[0142] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the mobile terminal by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0143] The memory may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as sound playback or image playback); the data storage area may store data generated based on the use of the mobile phone (such as audio data and a phone book). Furthermore, the memory may include high-speed random access memory (RAM) and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0144] Example 4

[0145] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the vehicle network channel prediction method based on multimodal fusion.

[0146] If the module / unit integrated in the mobile terminal is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0147] Based on this understanding, the present invention can implement all or part of the processes in the above-mentioned method by means of a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the above-mentioned aggregated reinforcement learning resource scheduling method. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form.

[0148] The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0149] It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0150] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A vehicle network channel prediction method based on multimodal fusion, characterized in that: include: Construct an urban road scenario for the Internet of Vehicles channel, and combine the weather scenario with the urban road scenario to obtain known time frame data and unknown time frame data to be predicted; The weather scenes include sunny scenes, rainy scenes and snowy scenes; The known time frame data includes channel index value, communication signal time frame sequence number, vehicle and RSU position coordinates, RGB image, LiDAR point cloud; The unknown time frame data to be predicted includes the communication signal time frame sequence number, the position coordinates of the vehicle and RSU, the RGB image, and the LiDAR point cloud; The known time frame data and the unknown data to be predicted are subjected to data enhancement, data vector representation, and position information processing to obtain a multimodal vector, and the multimodal vector is subjected to feature extraction and splicing processing to obtain a multimodal feature vector; Wherein, the data enhancement includes image enhancement and point cloud enhancement; Among them, image enhancement includes brightness adjustment, contrast enhancement, gamma correction, Gaussian blur and background masking; The point cloud enhancement process includes background filtering by referring to a 2D bird's-eye view image, mapping building locations into the point cloud data, removing point clouds in the building location area, and concentrating the remaining point clouds in the area of ​​vehicle motion and road structure; The data vector representation includes an image vector representation and a point cloud vector representation; The image vector representation process includes using RGB three-channel format to represent the image data as a vector and then normalizing it as the model input. The process of vector representation of point cloud includes first dividing the area of ​​interest into grids to achieve effective vector representation of point cloud data; limiting the range of point cloud by setting the area of ​​interest, and counting the density information within each grid, which is represented as a vector; secondly, setting a maximum density threshold. When the number of points in a grid exceeds the maximum density threshold, it is fixed to the maximum density threshold; finally, the number of points in all grids is normalized and used as model input; The position information processes the spatial consistency of multimodal data, aligns them through a unified coordinate system standardization method, and uses them as model input after normalization; The multimodal feature vector is input into the Transformer model, and the deep fusion between modalities is performed through the Transformer model's multi-head attention mechanism and feedforward neural network to obtain a joint feature vector; The joint feature vector is nonlinearly mapped through a multi-layer perceptron module, and the channel index probability distribution is output to predict the Internet of Vehicles channel.

2. The method for predicting a vehicle network channel based on multimodal fusion according to claim 1, characterized in that: The specific process of extracting and concatenating the multimodal vectors to obtain the multimodal feature vectors is as follows: The multimodal feature vectors are respectively input into the ResNet34 module and the ResNet18 module for feature extraction, and the multimodal vectors are spliced ​​to obtain the multimodal feature vector.

3. The method for predicting a vehicle network channel based on multimodal fusion according to claim 1, characterized in that: The Transformer model includes an encoder and a decoder. The encoder performs multi-level abstraction and expression on the input multimodal feature vector through a multi-head attention mechanism and a feedforward neural network, and then combines the decoder with the target context to output a joint feature vector.

4. The method for predicting a vehicle network channel based on multimodal fusion according to claim 3, characterized in that: The specific process of the multi-head attention mechanism is as follows: Project the input multimodal feature vector into multiple subspaces, calculate the attention weights based on the multiple subspaces, and establish the relationship between multiple feature dimensions; Based on the relationship between multiple feature dimensions, each attention is independently generated into query, key and value vectors, and the weight is calculated by scaling the dot product attention. The multi-head results are spliced ​​into a unified output according to the obtained weight; Among them, scaled dot product attention scales the multi-head results by measuring the relevance by the dot product of the query and key vectors.

5. The method for predicting a vehicle network channel based on multimodal fusion according to claim 1, characterized in that: The joint feature vector is nonlinearly mapped through a multi-layer perceptron module, wherein the multi-layer perceptron module includes multiple fully connected layers and a Softmax activation function for dimensionality reduction and outputting a normalized probability distribution; The calculation formula of the Softmax activation function is include: Here, x is a vector of length K and the output is a probability distribution.

6. A vehicle network channel prediction system based on multimodal fusion, based on a vehicle network channel prediction method based on multimodal fusion according to any one of claims 1 to 5, characterized in that: include: The multimodal raw data acquisition module is used to construct the urban road scene of the Internet of Vehicles channel. In the urban road scene, the known time frame data and the unknown time frame data to be predicted are acquired in combination with the weather scene. The data processing and conversion module is used to perform data enhancement and data vector representation and position information processing on the known time frame data and the unknown data to be predicted to obtain a multimodal vector, and perform feature extraction and splicing processing on the multimodal vector to obtain a multimodal feature vector; The model processing module is used to input the multimodal feature vector into the Transformer model and perform deep fusion between the modalities through the Transformer model's multi-head attention mechanism and feedforward neural network to obtain a joint feature vector; The channel prediction module is used to perform nonlinear mapping on the joint feature vector through the multi-layer perceptron module, and output the channel index probability distribution, and use the channel index probability distribution to predict the Internet of Vehicles channel.

7. A mobile terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the vehicle network channel prediction method based on multimodal fusion as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the vehicle network channel prediction method based on multimodal fusion as described in any one of claims 1 to 5 is implemented.

Images