Internet of vehicles channel state information prediction method, system, medium and electronic equipment

Abstract

The application provides a vehicle networking channel state information prediction method, system, medium and electronic equipment. The prediction method comprises: acquiring multi-modal input information of a vehicle in vehicle networking; inputting the multi-modal input information into a trained vehicle networking channel state information prediction model to obtain a prediction result and outputting the prediction result. Correspondingly, the model training step comprises: acquiring a training set, a verification set and a constructed prediction model; inputting the training set into the prediction model for training, wherein feature extraction is performed based on the training set to obtain target features; fusing the target features to obtain fused features, and performing channel state decoding to obtain an amplitude prediction result; and verifying the verification set after each round of training to obtain the trained prediction model based on the model parameters with the lowest verification error. The application can improve the prediction accuracy of vehicle networking channel state information and enhance the prediction robustness under the condition of camera image degradation.

Description

Technical Field

[0001] This invention belongs to the field of communication control technology, and in particular relates to a method, system, medium and electronic equipment for predicting channel state information in vehicle networking. Background Technology

[0002] Vehicle-to-everything (V2X) communication is considered a crucial component of future sixth-generation mobile communication and intelligent transportation systems. In V2X scenarios, base stations or roadside units need to acquire accurate channel state information to support operations such as beamforming, link adaptation, and resource scheduling. Traditional channel state information acquisition schemes heavily rely on pilot-aided channel estimation (DAS), which measures the wireless channel response by periodically transmitting pilot signals. However, in high-speed vehicle scenarios, the wireless channel exhibits rapidly changing time-varying characteristics, and road structures, traffic facilities, oncoming vehicles, and various dynamic obstructions can trigger complex multipath propagation and fading. Continuing to use the traditional pilot-aided channel estimation mechanism to acquire channel state information not only consumes significant pilot resources and incurs high pilot overhead, but also generates additional feedback overhead in systems requiring terminal channel reporting, making it difficult to meet the low-latency and high-reliability communication requirements of high-speed V2X.

[0003] To reduce communication link overhead, existing research attempts to introduce external environment perception data to assist in channel state information acquisition and utilize environmental priors to uncover implicit correlations in wireless channels. Specifically, vehicle location can characterize the relative spatial relationship between the vehicle and the base station, used to establish a mapping between location and channel response; camera images can provide visual content such as roads, vehicles, buildings, and obstructions, used to characterize the visible environment of wireless propagation; and lidar point cloud observations can provide distance and spatial geometry data, used to describe the surrounding scene structure.

[0004] To overcome the insufficient representation capabilities of a single sensing modality, existing technologies further employ joint modeling of multi-source heterogeneous sensing information, leveraging the complementarity of information from different modalities to enhance channel state information acquisition. This demonstrates a correlation between external environment sensing data and wireless propagation status, and multi-modal sensing fusion can provide effective assistance for predicting channel state information in vehicle-to-everything (V2X) networks. Even so, existing technologies still suffer from the following shortcomings:

[0005] First, relying solely on vehicle location information can only reflect the two-dimensional or three-dimensional position of the vehicle, and cannot fully restore the spatial distribution characteristics of buildings, scattering bodies, and obstructions.

[0006] Second, relying solely on camera images makes them susceptible to factors such as changes in perspective, occlusion, blurring, noise, low light, rain, and image loss, resulting in unstable visual characteristics.

[0007] Third, when using lidar data alone, if there is a lack of effective spatial structure coding and effective point processing mechanisms, it is difficult to fully extract the geometric and statistical information in lidar observations;

[0008] Fourth, existing multimodal fusion methods typically employ simple splicing or fixed-weight fusion, which cannot adaptively adjust the contribution of each modality based on the reliability of information in different scenarios. This leads to a decrease in prediction accuracy when perceptual information degrades or modal quality is unbalanced. Summary of the Invention

[0009] The purpose of this invention is to provide a method, system, medium, and electronic device for predicting channel state information in vehicle-to-everything (V2X) networks, which solves the problems of high pilot overhead, insufficient single-sensor mode representation, and lack of adaptability in multi-modal fixed fusion in existing V2X channel state information acquisition schemes.

[0010] In a first aspect, the present invention provides a method for training a vehicle-to-everything (V2X) channel state information prediction model, the method comprising:

[0011] Obtain a training set, a validation set, and a constructed prediction model, wherein the training set includes at least sample data and sample labels;

[0012] The training set is input into the prediction model for training, wherein...

[0013] Based on the training set, feature extraction is performed to obtain target features, which include at least vehicle position features, camera fusion features, and lidar fusion features.

[0014] Multiple target features are fused to obtain a fused feature, and channel state decoding is performed to obtain the amplitude prediction result; and

[0015] After each round of training, the validation set is used for validation to obtain a trained prediction model based on the model parameters with the lowest validation error.

[0016] In some embodiments of the first aspect of this application, obtaining the training set specifically includes:

[0017] The sample data includes vehicle location, three-view camera images, and raw lidar observation data, among which,

[0018] The vehicle location is used to characterize the spatial relationship between the vehicle and the roadside base station, and the calculation formula is as follows:

[0019] ;

[0020] in, This represents the two-dimensional position of the vehicle in the nth sample. and These represent the vehicle's horizontal and vertical coordinates in the scene coordinate system, respectively.

[0021] The three-view camera image is represented as ,in, and These represent the left, middle, and right view camera images of the nth sample, respectively.

[0022] The original lidar observation data is represented as follows: ,in, This represents the original lidar observation corresponding to the nth sample;

[0023] The sample label includes the original complex channel state information corresponding to the sample data. The original complex channel state information is converted into amplitude and normalized to obtain the channel state information label, calculated as follows:

[0024] ;

[0025] in, This represents the original complex channel state information corresponding to the nth sample. This represents the amplitude of the channel state information for the nth sample. This indicates normalization processing. This represents the normalized channel state information amplitude label of the nth sample.

[0026] In some embodiments of the first aspect of this application, constructing the prediction model specifically includes:

[0027] The prediction model includes a vehicle location feature extraction module, a camera feature extraction module, a view-gated fusion module, a LiDAR range view module, a geometric contour bi-branch coding module, a scalar confidence-gated residual fusion module, and a channel state information decoding module.

[0028] The vehicle location feature extraction module is used to map the vehicle location into vehicle location features using a multilayer perceptron structure.

[0029] The camera feature extraction module is used to process the three-view camera images to obtain three-way camera features;

[0030] The view gating fusion module is used to generate left, center, and right view weights based on the vehicle position features, and to perform weighted fusion of the three camera features to obtain camera fusion features.

[0031] The lidar range view module is used to convert the original lidar observation data into a range view and generate an effective point mask corresponding to the range view;

[0032] The geometric contour dual-branch encoding module is used to extract spatial structure information and geometric statistical information, and to combine them with the vehicle position features and the camera fusion features to obtain the lidar fusion features.

[0033] The scalar confidence gated residual fusion module is used to fuse the vehicle position features, the camera fusion features, and the lidar fusion features to generate the final fused features;

[0034] The channel state information decoding module is used to map the final fused features into a normalized channel state information amplitude prediction result.

[0035] In some embodiments of the first aspect of this application, the step of extracting features based on the training set to obtain target features specifically includes:

[0036] The vehicle position features are extracted by preprocessing the vehicle position coordinates and then inputting them into a vehicle position feature extraction network to obtain the vehicle position features. The calculation formula is as follows:

[0037] ;

[0038] in, This represents the vehicle location feature extraction network. This represents the vehicle position coordinates of the nth sample. This represents the vehicle location features of the nth sample;

[0039] The camera fusion features are extracted by weighting and fusing the features from the three camera views according to the three-view weights, as shown in the following formula:

[0040] ;

[0041] ;

[0042] in, This represents the camera fusion feature of the nth sample. and Let be the three-view weights of the nth sample, and , , and These represent the feature extraction branches for the left, center, and right view cameras, respectively. and These represent the left, middle, and right view camera images of the nth sample, respectively.

[0043] Extracting the fusion features of the lidar, wherein the joint features are calculated and input into the lidar fusion network to obtain the lidar fusion features, the calculation formula is as follows:

[0044] ;

[0045] ;

[0046] , ;

[0047] ;

[0048] ;

[0049] ;

[0050] ;

[0051] in, This represents the LiDAR fusion feature of the nth sample. This indicates a lidar fusion network. Represents the joint features of the nth sample. This represents the concatenation function. Represents the geometric contour coding branch, This represents the geometric contour encoding features of the nth sample. Indicates the distance view encoding branch, This represents the distance view feature of the nth sample. Represents the geometric contour features of the nth sample. This represents the nearest distance feature of the nth sample. This represents the average distance feature of the nth sample. This represents the distance standard deviation characteristic of the nth sample. This represents the proportion of valid points in the nth sample. This represents the minimum effective pitch position feature of the nth sample. This represents the normalized azimuth location encoding feature of the nth sample. This represents the input of the LiDAR range view branch for the nth sample. This represents the normalized distance view of the nth sample. This indicates that the valid points have been normalized. This indicates data transformation and processing. This represents the original lidar observation data of the nth sample. This represents the distance view of the nth sample obtained by converting the original lidar observations. This represents the effective point mask corresponding to the distance view for the nth sample.

[0052] In some embodiments of the first aspect of this application, the step of fusing multiple target features to obtain fused features and performing channel state decoding to obtain amplitude prediction results specifically includes:

[0053] Multiple target features are fused to obtain a fused feature, calculated as follows:

[0054] ;

[0055] ;

[0056] ;

[0057] ;

[0058] in, This represents the fusion feature of the nth sample. This represents the basic representation of the nth sample, generated from vehicle location features and camera fusion features. Let represent the scalar confidence gating coefficient corresponding to the nth sample. This represents the residual feature of the nth sample generated with the participation of LiDAR information. This represents the vehicle location features of the nth sample. This represents the camera fusion feature of the nth sample. This represents the concatenation function. Representing the basic representation network, Indicates a gating network. express Activation function This represents the preset minimum gate value. This represents the LiDAR fusion feature of the nth sample. Represents the residual feature network;

[0059] The fused features are input into the channel state information decoder to obtain the normalized channel state information amplitude prediction result, calculated as follows:

[0060] ;

[0061] in, This represents the normalized channel state information amplitude prediction result for the nth sample. The channel state information decoder, This represents the fusion feature of the nth sample.

[0062] In some embodiments of the first aspect of this application, the total loss during training is calculated as follows:

[0063] ;

[0064] ;

[0065] ;

[0066] ;

[0067] in, Indicates the total loss. Indicates the main loss. Indicates the auxiliary loss weight, This indicates that lidar-assisted loss prediction is performed. Indicates the gating regularization weight. Indicates the gating regularization term. This represents the normalized channel state information amplitude label of the nth sample. This represents the normalized channel state information amplitude prediction result for the nth sample. This represents the normalized channel state information amplitude prediction result of the nth sample output by the auxiliary prediction head. Indicates the auxiliary prediction head. This represents the LiDAR fusion feature of the nth sample. This represents the mean square error.

[0068] To achieve the above and other related objectives, a second aspect of this application provides a method for predicting channel state information in a vehicle-to-everything (V2X) network, applied to a V2X channel state information prediction model trained by the V2X channel state information prediction model training method described in any one of the preceding claims, wherein the method includes:

[0069] Acquire multimodal input information from vehicles in the Internet of Vehicles (IoV);

[0070] The multimodal input information is input into the trained vehicle network channel state information prediction model to obtain the prediction result;

[0071] Output the prediction result, wherein the prediction result includes the normalized channel state information amplitude.

[0072] To achieve the above and other related objectives, a third aspect of this application provides a vehicle network channel state information prediction system, the system comprising:

[0073] The acquisition module is used to acquire multimodal input information of vehicles in the Internet of Vehicles (IoV).

[0074] The prediction module is used to input the multimodal input information into the trained vehicle network channel state information prediction model to obtain the prediction result;

[0075] The output module is used to output the prediction result, wherein the prediction result includes the normalized channel state information amplitude.

[0076] To achieve the above and other related objectives, a fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the vehicle network channel state information prediction model training method and / or the vehicle network channel state information prediction method described in any of the above claims.

[0077] To achieve the above and other related objectives, a fifth aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the vehicle network channel state information prediction model training method and / or the vehicle network channel state information prediction method described in any of the above claims.

[0078] As described above, the vehicle network channel state information prediction method, system, medium, and electronic device of the present invention have the following beneficial effects:

[0079] 1. This invention, by jointly modeling the vehicle position, three-view camera images, and the distance view and effective point mask converted from the original lidar observations, can extract feature information related to the wireless propagation environment from the vehicle's spatial position, visual environment structure, and lidar geometry. This reduces the reliance on traditional pilot-assisted channel estimation and terminal feedback mechanisms, lowers the communication overhead caused by frequent pilot measurements, and alleviates the problem of delayed acquisition of channel state information caused by rapid time-varying channels in high-speed mobile vehicle-to-everything (V2X) scenarios.

[0080] 2. This invention uses three-view camera feature fusion based on location conditions to adaptively adjust the contribution of left, center, and right camera view features in prediction according to the vehicle's position, reducing the instability of feature representation caused by occlusion, view changes, or image quality degradation of single-view images. By converting the original LiDAR observations into a range view and effective point mask, and combining effective point normalization, range view encoding branch, and geometric contour encoding branch, spatial structure information, effective point distribution information, and geometric statistical information are extracted simultaneously, improving the ability to characterize scatterers, occlusions, and road environment structures around the vehicle.

[0081] 3. This invention further employs a scalar confidence-gated residual fusion method, using vehicle position and camera fusion features as the basic representation. By adaptively controlling the injection intensity of LiDAR residual information through gating coefficients, compared to simple splicing or fixed-weight fusion, it can dynamically adjust the contribution according to different samples and different modal reliability, improving the stability and robustness of multimodal fusion. It also assists in predicting loss and gating regularization terms to constrain LiDAR branch learning of channel-related geometric features and suppresses gating coefficient collapse, ensuring that LiDAR information is effectively utilized during training.

[0082] 4. In outdoor experiments in the Internet of Vehicles (IoV) scenario, the normalized channel state information amplitude prediction results of this invention are as follows: the mean square error (MSE) for a clean scenario is 0.4734±0.0066, which is 57.58% lower than the baseline method (1.1158) without using multimodal sensing information, and 2.58% lower than the model using only vehicle position and camera (0.4860±0.0069); the MSE under mixed camera degradation conditions is 0.5468±0.0231, which is 2.69% lower than the model using only vehicle position and camera (0.5617±0.0097); compared with the fixed gating fusion method, adaptive gating fusion reduces the MSE for a clean scenario by 1.60%. Therefore, the results show that this invention can improve the accuracy and anti-sensory degradation capability of channel state information amplitude prediction in IoV scenarios, and can provide more reliable channel state information for beamforming, link adaptation, and wireless resource scheduling. Attached Figure Description

[0083] Figure 1 The diagram shows a scenario application of the vehicle network channel state information prediction model training method of the present invention in one embodiment;

[0084] Figure 2 The diagram shown is a flowchart of an embodiment of the vehicle network channel state information prediction model training method of the present invention.

[0085] Figure 3 The diagram shows a prediction model module in one embodiment of the vehicle network channel state information prediction model training method of the present invention.

[0086] Figure 4 The diagram shows the laser radar distance view and geometric contour feature construction in one embodiment of the vehicle network channel state information prediction model training method of the present invention.

[0087] Figure 5 The diagram shows a camera image degradation mode in one embodiment of the vehicle network channel state information prediction model training method of the present invention.

[0088] Figure 6The diagram shows the model training convergence curve in one embodiment of the vehicle network channel state information prediction model training method of the present invention.

[0089] Figure 7 The diagram shows a step-by-step illustration of the vehicle network channel state information prediction method of the present invention in one embodiment;

[0090] Figure 8 The diagram shown is a structural schematic of the vehicle network channel state information prediction system of the present invention in one embodiment.

[0091] Figure 9 The diagram shown is a structural schematic of an embodiment of the electronic device of the present invention. Detailed Implementation

[0092] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0093] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0094] Furthermore, the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0095] In light of the aforementioned background technologies, there is an urgent need for a multimodal fusion method for predicting channel state information in vehicle-to-everything (V2X) networks that can comprehensively utilize vehicle location, three-view camera images, and LiDAR geometric observation information. This method should improve the accuracy and robustness of channel state information prediction while reducing dependence on traditional pilot and feedback mechanisms. For example, Figure 1The diagram illustrates a vehicle-to-everything (V2X) communication scenario. A target vehicle moves along a road. The base station acquires the vehicle's position (e.g., its relative position to the base station, or its location within the base station's coordinate system), three-view camera images, and raw LiDAR observation data through a perception fusion unit. Based on this environmental perception information, it predicts the channel state information between the target vehicle and the base station. Furthermore, this invention constructs a vehicle position-based three-view camera adaptive fusion mechanism, a LiDAR range view and geometric contour dual-branch coding mechanism, and a scalar confidence-gated residual fusion mechanism. This enables end-to-end prediction from multimodal environmental perception information to a normalized channel state information amplitude representation. This invention reduces reliance on traditional pilot-assisted channel estimation, improves the accuracy and robustness of channel state information prediction under conditions of high-speed V2X movement and degraded perception information, and ultimately provides reliable channel state information for beamforming, link adaptation, and wireless resource scheduling. The technical solutions in the embodiments of this invention will be described in detail below with reference to the accompanying drawings.

[0096] like Figure 2 As shown, in one embodiment of the invention, the vehicle-to-everything (V2X) channel state information prediction model training method of the present invention includes the following steps:

[0097] Obtain a training set, a validation set, and a constructed prediction model, wherein the training set includes at least sample data and sample labels;

[0098] The training set is input into the prediction model for training, wherein...

[0099] Based on the training set, feature extraction is performed to obtain target features, which include at least vehicle position features, camera fusion features, and lidar fusion features.

[0100] Multiple target features are fused to obtain a fused feature, and channel state decoding is performed to obtain the amplitude prediction result; and

[0101] After each round of training, the validation set is used for validation to obtain a trained prediction model based on the model parameters with the lowest validation error.

[0102] It should be noted that, in this embodiment, a training set, a validation set, and a prediction model are first obtained. For example, the training set includes 11,000 samples, and the validation set includes 2,500 samples. In this embodiment, DeepVerse 6G O1 outdoor vehicle-to-everything (V2X) scenario data is used for validation. The data used for training and validation includes synchronized wireless channel data, three-view camera images, LiDAR observation data, and a scene parameter file. The scene parameter file can extract vehicle location information.

[0103] In the sample data construction stage, vehicle location information, three-view camera images, and lidar observation data are used as multimodal inputs, and the original complex channel state information corresponding to the synchronization is used as the label source. The original complex channel state information is first converted into amplitude, and then normalized to obtain normalized channel state information amplitude labels, which are used to supervise model training.

[0104] Furthermore, in this embodiment, in the vehicle-to-everything (V2X) scenario, for the nth sample, the vehicle position, left-view camera image, middle-view camera image, right-view camera image, original LiDAR observation data, and original complex channel state information corresponding to the above multimodal input data are obtained. Among them, the vehicle position, three-view camera images, and original LiDAR observation data are used as model inputs, and the original complex channel state information is used as model training labels after amplitude conversion and normalization processing.

[0105] Vehicle location is used to characterize the spatial relationship between the vehicle and the roadside base station, and can be represented as:

[0106] ;

[0107] in, This represents the two-dimensional position of the vehicle in the nth sample. and These represent the vehicle's horizontal and vertical coordinates in the scene coordinate system, respectively.

[0108] The three-view camera image is represented as follows:

[0109] ;

[0110] in, and These represent the left, middle, and right view camera images of the nth sample, respectively.

[0111] The raw lidar observation data is represented as follows:

[0112] ;

[0113] in, This represents the original lidar observation corresponding to the nth sample.

[0114] The corresponding original complex channel state information is converted into amplitude and normalized to obtain the channel state information label:

[0115] ;

[0116] in, This represents the original complex channel state information corresponding to the nth sample. This represents the amplitude of the channel state information for the nth sample. This indicates normalization processing. This represents the normalized channel state information amplitude label of the nth sample.

[0117] The channel state information label is used to supervise the training of the multimodal fusion model. Accordingly, the model's final output is the normalized channel state information amplitude prediction result, rather than directly predicting the complex channel state information.

[0118] Furthermore, in one embodiment of the invention, constructing the prediction model specifically includes:

[0119] The prediction model includes a vehicle location feature extraction module, a camera feature extraction module, a view-gated fusion module, a LiDAR range view module, a geometric contour bi-branch coding module, a scalar confidence-gated residual fusion module, and a channel state information decoding module.

[0120] The vehicle location feature extraction module is used to map the vehicle location into vehicle location features using a multilayer perceptron structure.

[0121] The camera feature extraction module is used to process the three-view camera images to obtain three-way camera features;

[0122] The view gating fusion module is used to generate left, center, and right view weights based on the vehicle position features, and to perform weighted fusion of the three camera features to obtain camera fusion features.

[0123] The lidar range view module is used to convert the original lidar observation data into a range view and generate an effective point mask corresponding to the range view;

[0124] The geometric contour dual-branch encoding module is used to extract spatial structure information and geometric statistical information, and to combine them with the vehicle position features and the camera fusion features to obtain the lidar fusion features.

[0125] The scalar confidence gated residual fusion module is used to fuse the vehicle position features, the camera fusion features, and the lidar fusion features to generate the final fused features;

[0126] The channel state information decoding module is used to map the final fused features into a normalized channel state information amplitude prediction result.

[0127] It should be noted that, in this embodiment, as Figure 3As shown, the vehicle position feature extraction module adopts a multilayer perceptron structure to map the two-dimensional vehicle position into vehicle position features. The module includes coordinate standardization, multilayer perceptron operation, and layer normalization. Specifically, firstly, the two-dimensional vehicle position is standardized using the mean and standard deviation of the vehicle position coordinates in the training set, with 1e-6 added to the standard deviation to ensure numerical stability. Then, the standardized two-dimensional vehicle position is input into a multilayer perceptron consisting of three fully connected layers, with feature dimensions of 2→32→32→16 respectively. The first two fully connected layers are followed by a linear rectified unit activation function for nonlinear transformation. Finally, layer normalization is performed on the multilayer perceptron output to obtain 16-dimensional vehicle position features.

[0128] Furthermore, in this embodiment, the camera feature extraction module processes the left, center, and right view images respectively. Each camera branch includes convolutional feature extraction, multi-receptive field feature mixing, global average pooling, feature projection, and normalization processing to finally obtain the camera features for each view. The left, center, and right view camera feature extraction branches use the same network structure and are configured with independent network parameters. Specifically, before inputting into the camera feature extraction branches, the target region in each view image is determined based on the vehicle position, and a 150×150 pixel area is cropped from the left, center, and right view images respectively. For a three-channel RGB image, with each channel having a size of 150*150, two convolutional feature extraction units are first used to increase the number of channels from 3 to 16 and 24 respectively. Then, the 24-channel features are sequentially processed by two multi-receptive field feature mixing units. The processing result is then mapped to 32-channel features by the convolutional feature extraction unit and processed by a multi-receptive field feature mixing unit. Finally, the two-dimensional feature map is converted into a feature vector by global average pooling, and then processed by multi-layer perceptron projection and layer normalization with dimensions of 32→32→16 to obtain 16-dimensional camera features.

[0129] Furthermore, in this embodiment, the subsequent view-gated fusion module generates corresponding three-view weights based on the vehicle position features and the features of the left, center, and right cameras, and performs weighted fusion of the three camera features to obtain camera fusion features. Specifically, the 16-dimensional vehicle position features are concatenated with the 16-dimensional features of the left, center, and right cameras to form a 64-dimensional view-gated input. The view-gated input is first converted into 32-dimensional features through a fully connected mapping and then nonlinearly transformed using the GELU activation function. Subsequently, three unnormalized view scores are generated through the next fully connected mapping. After the three scores are normalized by the Softmax function, the left, center, and right three-view weights are obtained. Finally, the three camera features are weighted and summed according to the view weights to obtain the 16-dimensional camera fusion features.

[0130] Further, in this embodiment, the original LiDAR observation data is preprocessed and converted into a range view, and a valid point mask corresponding to the range view is generated. The LiDAR range view and geometric contour dual-branch encoding module takes the range view and valid point mask as input, first normalizes the valid points of the range view, and then splices the normalized range view with the valid point mask. Specifically, the original LiDAR observation data is preprocessed and converted into a range view with a size of 1*64*1024, and a valid point mask of the same size is generated to mark whether there are valid measurement points at each grid position. Subsequently, the mean and variance of the range view are calculated only based on the valid points, the valid measurement values ​​are standardized, and invalid positions are set to zero. Finally, the normalized range view and the valid point mask are spliced ​​along the channel dimension to form a range view branch input with a size of 2*64*1024.

[0131] Further, in this embodiment, the geometric contour dual-branch encoding module is then used to extract spatial structure information and geometric statistical information. Finally, the LiDAR branch features are jointly fused with vehicle position features and camera fusion features to obtain LiDAR fusion features. Specifically, the distance view encoding branch module takes a stitched result of size 2*64*1024 as input and uses a two-dimensional convolutional network with channel numbers sequentially from 2 to 16 to 24 to 32 to 48 for encoding. Residual units are set at the 24, 32, and 48 channel stages. The convolutional output is then global average pooled and projected through a multilayer perceptron of dimensions 48 to 32 to 16, and then processed by layer normalization to obtain 16-dimensional distance view features. Further, the geometric contour encoding branch module takes a 6*1024... The geometric contour features are used as input and encoded sequentially using one-dimensional convolutions with 6→32→64 channels. After each convolution, batch normalization and GELU activation are performed. Subsequently, the encoded results are subjected to one-dimensional global average pooling and projection onto a multilayer perceptron with dimensions of 64→32→16 to obtain 16-dimensional geometric contour features. Then, the 16-dimensional distance view features, 16-dimensional geometric contour features, 16-dimensional vehicle position features, and 16-dimensional camera fusion features are concatenated to form a 64-dimensional joint feature. The joint feature is first converted into a 64-dimensional feature through a fully connected mapping and then subjected to a nonlinear transformation using the GELU activation function. It is then converted into a 16-dimensional feature through another fully connected mapping. Finally, the mapping result is subjected to layer normalization to obtain the 16-dimensional LiDAR fusion feature.

[0132] Furthermore, in this embodiment, the scalar confidence-gated residual fusion module uses vehicle position features, camera fusion features, and lidar fusion features as inputs to fuse them to generate a 64-dimensional final fusion feature. Correspondingly, the channel state information decoding module maps the final fusion feature to a 1×64×64 normalized channel state information amplitude prediction result.

[0133] Furthermore, in one embodiment of the invention, the step of extracting features based on the training set to obtain target features specifically includes:

[0134] The vehicle position features are extracted by preprocessing the vehicle position coordinates and then inputting them into a vehicle position feature extraction network to obtain the vehicle position features. The calculation formula is as follows:

[0135] ;

[0136] in, This represents the vehicle location feature extraction network. This represents the vehicle position coordinates of the nth sample. This represents the vehicle location features of the nth sample;

[0137] The camera fusion features are extracted by weighting and fusing the features from the three camera views according to the three-view weights, as shown in the following formula:

[0138] ;

[0139] ;

[0140] in, This represents the camera fusion feature of the nth sample. and Let be the three-view weights of the nth sample, and , , and These represent the feature extraction branches for the left, center, and right view cameras, respectively. and These represent the left, middle, and right view camera images of the nth sample, respectively.

[0141] Extracting the fusion features of the lidar, wherein the joint features are calculated and input into the lidar fusion network to obtain the lidar fusion features, the calculation formula is as follows:

[0142] ;

[0143] ;

[0144] , ;

[0145] ;

[0146] ;

[0147] ;

[0148] ;

[0149] in, This represents the LiDAR fusion feature of the nth sample. This indicates a lidar fusion network. Represents the joint features of the nth sample. This represents the concatenation function. Represents the geometric contour coding branch, This represents the geometric contour encoding features of the nth sample. Indicates the distance view encoding branch, This represents the distance view feature of the nth sample. Represents the geometric contour features of the nth sample. This represents the nearest distance feature of the nth sample. This represents the average distance feature of the nth sample. This represents the distance standard deviation characteristic of the nth sample. This represents the proportion of valid points in the nth sample. This represents the minimum effective pitch position feature of the nth sample. This represents the normalized azimuth location encoding feature of the nth sample. This represents the input of the LiDAR range view branch for the nth sample. This represents the normalized distance view of the nth sample. This represents the original lidar observation data of the nth sample. This represents the distance view of the nth sample obtained by converting the original lidar observations. This represents the effective point mask corresponding to the distance view for the nth sample.

[0150] It should be noted that, in this embodiment, the acquired vehicle position, three-view camera images, and original lidar observations are subjected to feature processing to obtain vehicle position features, camera fusion features, and lidar fusion features.

[0151] The specific process is as follows:

[0152] (1) Vehicle location feature extraction:

[0153] The vehicle position coordinates are preprocessed and then input into the vehicle position feature extraction network to obtain the vehicle position features:

[0154] ;

[0155] in, This represents the vehicle location feature extraction network. This represents the vehicle position coordinates of the nth sample. The input preprocessing, which represents vehicle location features, includes standardizing the vehicle location coordinates to reduce the impact of different coordinate scales on feature extraction. These vehicle location features characterize the spatial relationship between the vehicle and the roadside base station and participate in subsequent camera viewpoint weight generation and multimodal fusion. The vehicle location feature extraction network includes multiple fully connected layers and activation layers for nonlinear mapping of the standardized vehicle location coordinates. After layer normalization, the mapping results are used to obtain the vehicle location features.

[0156] (2) Adaptive fusion of three-view cameras based on vehicle position:

[0157] The left, center, and right view camera images are input into the corresponding camera feature extraction branches to obtain the three view features:

[0158] ;

[0159] in, , and These represent the feature extraction branches for the left, center, and right view cameras, respectively. , and These represent the camera features from the left, center, and right viewpoints, respectively. The camera feature extraction branches can include convolutional feature extraction, multi-receptive field feature fusion, pooling, feature projection, and normalization processing to extract visual propagation environment features from different viewpoints. The three camera feature extraction branches have the same network structure but are configured with independent network parameters. Each camera feature extraction branch includes multiple convolutional feature extraction units and multiple multi-receptive field feature fusion units, followed by a global average pooling layer and a multilayer perceptron projection layer. The projection results are further normalized to obtain the camera features for the corresponding viewpoint. The multi-receptive field feature fusion unit includes two local feature extraction branches with different convolutional kernel sizes and a global context branch. The outputs of the three branches are concatenated and fused using pointwise convolution, and then the residuals are added to the input of the multi-receptive field feature fusion unit.

[0160] To adaptively determine the importance of different camera views based on the vehicle's location, the features from the left-view camera, the middle-view camera, the right-view camera, and the vehicle's position are concatenated to obtain view weight generation features. These features are then input into a view gating network to obtain unnormalized scores for the left, middle, and right views.

[0161] ;

[0162] in, The view-gated network is described, consisting of two fully connected layers and a Gaussian error linear unit activation layer between the two fully connected layers. This network maps the stitched results of vehicle position features and three camera features into three unnormalized view scores. The unnormalized score corresponding to the three perspectives can be represented as: .

[0163] The unnormalized scores are normalized to obtain the three-view weights:

[0164] ;

[0165] in, The normalized three-view weights can be represented as: and full In this system, the viewpoint weights are not pre-set manually, but are adaptively generated by the viewpoint gating network. This network takes the vehicle position features and the global features from the left, center, and right cameras as input, outputs the confidence scores of each viewpoint through two fully connected layers, and then normalizes them using the Softmax function to obtain viewpoint weights that sum to 1. During the training phase, the parameters of the viewpoint gating network are updated synchronously with the total loss function through backpropagation, thereby automatically learning the mapping relationship between the vehicle's spatial position, visual features, and the contribution of each viewpoint. Therefore, in the inference phase of later applications, the gating network can generate weights in real time based on the vehicle position and three-view image features of the current sample.

[0166] Subsequently, the camera viewpoint features from the three cameras are weighted and fused according to the aforementioned three-viewpoint weights to obtain the camera fusion features:

[0167] ;

[0168] in, The fused camera features are represented by the image. This processing can utilize vehicle position features to adjust the contribution of left, center, and right view camera features in channel state information prediction, thereby improving the adaptability of camera feature representation under different vehicle positions and view conditions.

[0169] (3) Laser radar feature processing:

[0170] 1) Generation of distance view and effective point mask:

[0171] Convert the raw lidar observation data into a range view and corresponding effective point mask:

[0172] ;

[0173] in, This represents the original lidar observation data of the nth sample. This represents the range view obtained by converting the original lidar observations. This represents the effective point mask corresponding to the distance view. This indicates data conversion and processing. The valid point mask is used to mark whether there are valid LiDAR measurement points at the corresponding positions in the distance view.

[0174] 2) Valid point normalization and distance view branch input construction:

[0175] Perform effective point normalization on the distance view to obtain a normalized distance view:

[0176] ;

[0177] in, This indicates that the valid point normalization process is performed, meaning that only the valid point mask is used. The normalized statistics of the valid measurement points are calculated, and the invalid locations are set to zero. This process can reduce the impact of invalid lidar measurement points on subsequent feature extraction.

[0178] Subsequently, the normalized range view and the effective point mask are stitched together along the channel dimension to obtain the LiDAR range view branch input:

[0179] ;

[0180] in, This indicates the input for the lidar range view branch. This input contains both lidar range information and effective point distribution information, used to preserve the spatial organization of lidar observations in the elevation and azimuth directions.

[0181] 3) Construction of geometric contour features:

[0182] Constructing the geometric contour features of the lidar based on normalized distance views and effective point masks:

[0183] ;

[0184] in, This function represents the constructor for geometric contour features. It processes the normalized distance view and valid point masks column-by-column, using the azimuth column of the distance view as the unit. For each azimuth column, valid measurement points are first selected based on the valid point mask, and the nearest distance, average distance, distance standard deviation, and valid point ratio are calculated. Based on this, the minimum pitch row index corresponding to the valid points in that azimuth column is extracted and normalized to construct the row-direction position contour feature. Simultaneously, the column indices of each azimuth column are normalized to construct the column-direction position encoding feature. This represents the geometric contour features obtained by constructing a normalized distance view and an effective point mask.

[0185] The geometric contour features include six categories: nearest distance, average distance, distance standard deviation, effective point ratio, minimum effective pitch position, and normalized azimuth position encoding, which can be represented as:

[0186] ;

[0187] in, This indicates the closest distance feature, used to describe the effective lidar measurement information at the corresponding azimuth. This represents the average distance characteristic, used to describe the average distance of effective lidar measurement points in the corresponding orientation; This represents the standard deviation of distance, used to describe the degree of dispersion of the distance distribution; This indicates the proportion of effective points, used to describe the effectiveness of lidar observations at the corresponding azimuth. The minimum effective pitch row position feature is used to describe the distribution of effective points in the pitch direction. For each azimuth column, the minimum value of the pitch row index corresponding to the effective point is extracted and normalized to describe how the position of the effective area in the pitch direction changes with the azimuth column. This represents the normalized azimuth position coding feature, used to describe the position or distribution information of different azimuth columns. It is obtained by normalizing the column index of each azimuth column and is used to describe the relative position of each azimuth column in the azimuth direction.

[0188] By using the aforementioned geometric contour features, more compact geometric statistical information can be extracted while preserving the spatial structure information of the lidar, thus providing a supplement for subsequent lidar feature encoding.

[0189] 4) Fusion of dual-branch coding and LiDAR features:

[0190] Input the LiDAR distance view branch Input the distance view encoding branch to obtain the distance view features:

[0191] ;

[0192] in, Indicates the distance view encoding branch, This branch represents the range view features and is used to extract spatial structure features from the LiDAR range view. The range view encoding branch consists of multiple two-dimensional convolutional units, residual units, global average pooling layers, and multilayer perceptron projection units. The two-dimensional convolutional units sequentially extract spatial structure features of the range view in the pitch and azimuth directions through two-dimensional convolution, batch normalization, and the GELU activation function. The residual units add the convolution transformation result to the unit input as a residual to preserve input information and enhance deep feature propagation. Subsequently, the extracted result is converted into a one-dimensional feature vector through global average pooling and flattening, and then processed by multilayer perceptron projection and layer normalization to obtain the range view features.

[0193] At the same time, geometric contour features Input the geometric contour encoding branch to obtain the geometric contour features:

[0194] ;

[0195] in, Represents the geometric contour coding branch, This represents the geometric contour encoding features. This branch is used to extract distance statistics, effective point distribution features, and spatial geometric contour information from lidar observations. The geometric contour encoding branch includes multiple one-dimensional convolutional units, a global average pooling layer, and a multilayer perceptron projection layer. Each one-dimensional convolutional unit sequentially uses one-dimensional convolution, batch normalization, and the GELU activation function to extract the geometric statistical correlation between different orientation columns. The extraction results are then processed by global average pooling, multilayer perceptron projection, and layer normalization to obtain the geometric contour features.

[0196] Then, the distance view features, geometric contour encoding features, vehicle position features, and camera fusion features are concatenated to obtain the joint features for the LiDAR fusion network:

[0197] ;

[0198] in, This represents the joint feature obtained by fusing distance view information, geometric contour information, vehicle position information, and camera visual information.

[0199] The joint features are input into the lidar fusion network to obtain the lidar fusion features:

[0200] ;

[0201] in, This indicates a lidar fusion network. This represents the fusion features of the LiDAR system. The process jointly utilizes spatial structure information from the full-range range view of the LiDAR, statistical information from geometric contour features, and environmental context information provided by vehicle position features and camera fusion features. The LiDAR fusion network includes multiple fully connected layers, a GELU activation function, and layer normalization layers to fuse range view features, geometric contour features, vehicle position features, and camera fusion features, ultimately yielding the LiDAR fusion features.

[0202] Specifically, in this embodiment, as Figure 4 As shown, the raw lidar observation data is first converted into a range view and a corresponding effective point mask. In this embodiment, both the range view and the effective point mask have a size of 1×64×1024, where 64 corresponds to the elevation direction and 1024 corresponds to the azimuth direction. Subsequently, the range view undergoes effective point normalization processing to obtain a normalized range view. Based on the normalized range view and the effective point mask, two types of lidar branch inputs are generated. The normalized range view and the effective point mask are concatenated along the channel dimension to form the range view branch input, with a size of 2×64×1024. Simultaneously, a six-channel geometric contour feature is constructed using the same normalized range view and effective point mask, including the nearest distance, average distance, distance standard deviation, effective point ratio, minimum effective elevation position, and normalized azimuth position encoding, forming the geometric contour branch input with a size of 6×1024.

[0203] Furthermore, in one embodiment of the invention, the step of fusing multiple target features to obtain fused features and performing channel state decoding to obtain amplitude prediction results specifically includes:

[0204] Multiple target features are fused to obtain a fused feature, calculated as follows:

[0205] ;

[0206] ;

[0207] ;

[0208] ;

[0209] in, This represents the fusion feature of the nth sample. This represents the basic representation of the nth sample, generated from vehicle location features and camera fusion features. Let represent the scalar confidence gating coefficient corresponding to the nth sample. This represents the residual feature of the nth sample generated with the participation of LiDAR information. This represents the vehicle location features of the nth sample. This represents the camera fusion feature of the nth sample. This represents the concatenation function. Representing the basic representation network, Indicates a gating network. express Activation function This represents the preset minimum gate value. This represents the LiDAR fusion feature of the nth sample. Represents the residual feature network;

[0210] The fused features are input into the channel state information decoder to obtain the normalized channel state information amplitude prediction result, calculated as follows:

[0211] ;

[0212] in, This represents the normalized channel state information amplitude prediction result for the nth sample. The channel state information decoder, This represents the fusion feature of the nth sample.

[0213] It should be noted that, in this embodiment, based on the obtained vehicle position features Camera fusion features Fusion features with lidar The system constructs a vehicle position-camera basic representation, LiDAR residual features, and scalar confidence gating coefficients. First, the vehicle position features and camera fusion features are concatenated and input into the basic representation network to obtain the vehicle position-camera basic representation.

[0214] ;

[0215] in, Representing the basic representation network, The basic representation is generated from vehicle position features and camera fusion features. The basic representation network includes multiple fully connected layers and a GELU activation function to fuse vehicle position features and camera fusion features to generate a vehicle position-camera basic representation that does not depend on LiDAR information.

[0216] Then, the vehicle position features, camera fusion features, and LiDAR fusion features are concatenated and input into the residual feature network to obtain the LiDAR residual features:

[0217] ;

[0218] in, Represents the residual feature network, The residual features generated by LiDAR information are represented. The residual feature network includes multiple fully connected layers and a GELU activation function, which are used to generate residual features with the same dimension as the basic representation based on vehicle position features, camera fusion features and LiDAR fusion features. The residual features are used to represent the supplementation and correction of the vehicle position-camera basic representation after the introduction of LiDAR information.

[0219] Simultaneously, vehicle position features, camera fusion features, and LiDAR fusion features are input into the gating network to generate scalar confidence gating coefficients:

[0220] ;

[0221] in, Indicates a gating network. express Activation function This represents the preset minimum threshold value, used to prevent LiDAR information from being completely ignored. Let represent the scalar confidence gating coefficient corresponding to the nth sample. The scalar confidence gating network includes multiple fully connected layers and a GELU activation function, which are used to generate a scalar score based on vehicle position features, camera fusion features, and LiDAR fusion features. Correspondingly, the scalar score is subjected to a Sigmoid function and a minimum gating value constraint to obtain the scalar confidence gating coefficient 'an', which is used to control the injection intensity of the residual features.

[0222] The final fusion feature is represented as:

[0223] ;

[0224] in, This represents the final fused feature after fusing vehicle location, camera, and LiDAR information. This step adaptively controls the injection intensity of LiDAR residual information through scalar confidence gating coefficients, allowing the LiDAR residual information to participate in the fusion according to the sample state, thereby mitigating the performance degradation problem caused by simple stitching or fixed-weight fusion when modal quality changes.

[0225] Specifically, in this embodiment, the 16-dimensional vehicle position features and the 16-dimensional camera fusion features are first concatenated to form a 32-dimensional input. This input is then processed by a basic representation network with dimensions of 32→96→64. After each fully connected mapping, the GELU activation function is applied to obtain a 64-dimensional vehicle position-camera basic representation. Subsequently, the 16-dimensional vehicle position features, the 16-dimensional camera fusion features, and the 16-dimensional LiDAR fusion features are concatenated to form a 48-dimensional joint input. This joint input is then processed by a residual feature network with dimensions of 48→64→64 and the GELU activation function to obtain 64-dimensional residual features. Simultaneously, the 48-dimensional joint input is fed into a scalar confidence gating network with dimensions 48→32→1 to generate a scalar score. This score is mapped by a Sigmoid function and then adjusted according to a minimum gating value of 0.1 to obtain a scalar confidence gating coefficient with a value range of (0.1,1). Finally, the injection intensity of the residual features is adjusted according to the gating coefficient, and the adjusted residual features are added to the vehicle position-camera basic representation to obtain the final 64-dimensional fused features.

[0226] Furthermore, it should be noted that in this embodiment, the final fused feature is input into the channel state information decoder to obtain the normalized channel state information amplitude prediction result:

[0227] ;

[0228] in, The channel state information decoder, This represents the normalized channel state information amplitude prediction result for the nth sample. The channel state information decoder includes multiple fully connected layers and a ReLU activation function, used to map the final fused features into a channel state information amplitude vector, and reconstruct the amplitude vector into a two-dimensional normalized channel state information amplitude representation. It should be noted that the decoder in this embodiment is used to map the fused environmental features into a two-dimensional channel amplitude structure, and the output result is a normalized channel state information amplitude representation, rather than complex channel state information.

[0229] Specifically, further in this embodiment, the 64-dimensional final fused feature is input into the channel state information decoder. The decoder first generates a 1024-dimensional feature through a fully connected mapping and performs a nonlinear transformation using a linear rectified unit activation function. Subsequently, a 4096-dimensional output vector is generated through another fully connected mapping, and the output is again constrained to a non-negative value using a linear rectified unit activation function. Finally, the 4096-dimensional output vector is reconstructed into a normalized channel state information amplitude prediction result of size 1×64×64. Accordingly, the prediction result is a normalized amplitude representation of the channel state information.

[0230] Furthermore, in one embodiment of the invention, the total loss during the training process is calculated as follows:

[0231] ;

[0232] ;

[0233] ;

[0234] ;

[0235] in, Indicates the total loss. Indicates the main loss. Indicates the auxiliary loss weight, This indicates that lidar-assisted loss prediction is performed. Indicates the gating regularization weight. Indicates the gating regularization term. This represents the normalized channel state information amplitude label of the nth sample. This represents the normalized channel state information amplitude prediction result for the nth sample. This represents the normalized channel state information amplitude prediction result of the nth sample output by the auxiliary prediction head. Indicates the auxiliary prediction head. This represents the LiDAR fusion feature of the nth sample. This represents the mean square error.

[0236] It should be noted that, in this embodiment, the error between the model's main decoder output and the actual normalized channel state information amplitude label is used as the main loss. :

[0237]

[0238] in, This represents the true normalized channel state information amplitude label for the nth sample. This represents the normalized channel state information amplitude prediction result output by the master decoder for the nth sample.

[0239] During the training phase, an auxiliary prediction head is set after the lidar features are fused to obtain auxiliary prediction results:

[0240]

[0241] And construct a lidar-assisted loss prediction system. :

[0242]

[0243] in, This represents the normalized channel state information amplitude prediction result output by the auxiliary prediction head. The auxiliary prediction head comprises multiple fully connected layers and a GELU activation function, used to generate a normalized channel state information amplitude prediction result based on the lidar fusion features. This result, as the output of the auxiliary prediction branch, is only used to calculate the lidar auxiliary prediction loss to constrain the lidar branch to learn geometric features related to the channel state information, and is not used as the final prediction result of the model.

[0244] Simultaneously, a gating regularization term is applied to the scalar confidence gating coefficients to suppress gating coefficient collapse, ensuring the model maintains effective utilization of LiDAR information. The total training loss is expressed as:

[0245]

[0246] in, Indicates the auxiliary loss weight, Indicates the gating regularization weight. This represents the gating regularization term, where the main loss is... This is used to measure the difference between the main decoder's prediction result and the actual normalized channel state information amplitude label, directly constraining the model's final prediction accuracy. LiDAR-assisted prediction loss. This is used to constrain the retention of geometric information related to channel state information in the fused features of LiDAR, preventing the LiDAR branches from being weakened or ignored during multimodal training, and the gating regularization term... This is used to guide the average gating coefficient within a training batch to approach a set value, suppress the gating coefficient from approaching the lower or upper limit for a long time, and at the same time preserve the adaptive variation of the gating coefficient between different samples.

[0247] Accordingly, in this embodiment, the auxiliary prediction head is only used during the training phase to constrain the learning of geometric features related to channel state information by the lidar branch; during the testing or actual deployment phase, the model output is given by the main decoder.

[0248] Furthermore, in one embodiment of the invention, as Figure 5 As shown, to verify the robustness of the present invention under conditions of visual perception quality degradation, this embodiment applies hybrid degradation processing to some three-view camera images in the verification sample. The degradation types include blurring, Gaussian noise, low light, rain-like interference, and camera blackout. The left, middle, and right view images are treated with the same degradation type to simulate image quality degradation or failure of the camera sensor in complex road environments.

[0249] In this embodiment, camera degradation is applied to 50.96% of the validation samples. Degradation samples are selected using a fixed random seed of 999, with each degradation type allocated with equal probability. This includes 1226 clean samples, 278 blurred samples, 251 Gaussian noise samples, 252 low-light samples, 242 rain-like interference samples, and 251 camera blackout samples. Specific degradation settings include: blurring is handled using average pooling with a kernel size of 9; the standard deviation of Gaussian noise is set to 0.24 times the image scale; low-light degradation multiplies the image intensity by 0.30; rain-like interference adds 105 random rain lines with a length of 36; and camera blackout sets the camera image to zero. Since this invention utilizes both vehicle position and LiDAR geometric information, when camera images are blurred, noisy, in low-light, or missing, the model can still supplement the propagation environment information through other modalities, thereby improving the robustness of channel state information prediction.

[0250] Furthermore, in this embodiment, during the training phase, the main decoder generates the main prediction result based on the final fusion features, while the LiDAR auxiliary prediction head generates the auxiliary branch prediction result based on the LiDAR fusion features. The main prediction loss is calculated using element-wise mean square error between the main prediction result and the true normalized channel state information amplitude label, and the LiDAR auxiliary prediction loss is calculated using element-wise mean square error between the auxiliary branch prediction result and the label. The model uses a total training loss consisting of the main prediction loss, the LiDAR auxiliary prediction loss, and the gating regularization term for backpropagation and parameter updates. The weight of the LiDAR auxiliary prediction loss is set to 0.1, the weight of the gating regularization loss is set to 0.02, and the regularization target of the average gating coefficient within the training batch is set to 0.35. The LiDAR auxiliary prediction loss is used to enhance the learning ability of the LiDAR branch to channel-related geometric features, and the gating regularization term is used to suppress the long-term convergence of the gating coefficient to the boundary of the value range, so that the LiDAR residual information stably participates in the fusion. During the training phase, the output of the main decoder is used as the channel state information prediction result of the model, and the auxiliary prediction head is only used to provide additional supervision.

[0251] Furthermore, during the inference phases such as verification and actual deployment, the model retains the scalar confidence-gated residual fusion mechanism, and the normalized channel state information amplitude prediction result is generated by the main decoder. The auxiliary prediction head does not participate in the generation of the prediction result. During the verification phase, only the mean square error between the main decoder prediction result and the real label is calculated. The lidar-assisted prediction loss and gating regularization term are not calculated, and backpropagation and parameter updates are not performed. During the actual deployment phase, the prediction result of the main decoder is directly output, and the loss function is not calculated.

[0252] The specific training configuration is as follows: the initial learning rate is set to 0.004, the batch size is set to 256, the maximum number of training rounds is set to 100, and an early stopping mechanism is adopted with an early stopping patience value of 35. To reduce the impact of randomness, this embodiment uses three random seeds, 2024, 2025, and 2026, for repeated verification.

[0253] Specifically, such as Figure 6 As shown, the training loss decreases overall during the training process, and the validation loss gradually stabilizes, indicating that the multimodal fusion model can converge effectively.

[0254] Therefore, to verify the technical effect of this embodiment, the method of the present invention is compared with a benchmark method that does not use multimodal perception information and a method that only uses vehicle position and camera. The results are shown in Table 1 below.

[0255] Table 1. Comparison of mean square error in channel state information prediction using different methods

[0256] method Mean square error of clean scene Hybrid camera degradation mean square error Benchmark methods that do not use multimodal sensing information 1.1158 — Method using vehicle location and three-view camera information 0.4860 ± 0.0069 0.5617 ± 0.0097 Method of the present invention 0.4734 ± 0.0066 0.5468 ± 0.0231

[0257] As shown in Table 1, compared with the benchmark method that does not use multimodal perception information, the method of the present invention reduces the mean square error by 57.58% in clean scenes; compared with vehicle position and camera methods, the mean square error is reduced by 2.58% and 2.69% in clean scenes and mixed camera degradation scenes, respectively. The above experimental results show that the present invention can improve the prediction accuracy of vehicle network channel state information and enhance the prediction robustness under camera image degradation conditions.

[0258] The scope of protection of the vehicle network channel state information prediction method described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0259] This application embodiment also provides a method for predicting vehicle network channel state information, applied to a vehicle network channel state information prediction model trained by the vehicle network channel state information prediction model training method described in any of the above claims, wherein, as Figure 7 As shown, the method includes the following steps:

[0260] Step S702: Obtain multimodal input information of the vehicle in the vehicle network;

[0261] Step S704: Input the multimodal input information into the trained vehicle network channel state information prediction model to obtain the prediction result;

[0262] Step S706: Output the prediction result, wherein the prediction result includes the normalized channel state information amplitude.

[0263] It should be noted that, in this embodiment, during the inference application stage, the prediction model has already been trained. Therefore, the multimodal input information of vehicles in the Internet of Vehicles can be directly obtained and input into the trained Internet of Vehicles channel state information prediction model. Based on the same model processing process as in the model training process, the prediction result is obtained and then the prediction result can be output. Accordingly, the prediction result includes the normalized channel state information amplitude. The data processing process in the inference stage is the same as the training process described above, so it will not be described in detail in this embodiment.

[0264] This application also provides a vehicle network channel state information prediction system. The vehicle network channel state information prediction system can implement the vehicle network channel state information prediction method described in this application. However, the implementation device of the vehicle network channel state information prediction method described in this application includes, but is not limited to, the structure of the vehicle network channel state information prediction system listed in this embodiment. All structural modifications and substitutions of the prior art made based on the principles of this application are included within the protection scope of this application.

[0265] Please see Figure 8 In one embodiment, this embodiment provides a vehicle-to-everything (V2X) channel state information prediction system 80, the system comprising:

[0266] Module 81 is used to acquire multimodal input information of vehicles in the Internet of Vehicles.

[0267] Prediction module 82 is used to input the multimodal input information into the trained vehicle network channel state information prediction model to obtain the prediction result;

[0268] Output module 83 is used to output the prediction result, wherein the prediction result includes the normalized channel state information amplitude.

[0269] Since the specific implementation of this embodiment corresponds to the aforementioned method embodiment, the same details will not be repeated here, and those skilled in the art should also understand this. Figure 8 The division of the modules in the embodiments is only a logical functional division. In actual implementation, they can be fully or partially integrated into one or more physical entities. These modules can be fully implemented in software through processing element calls, fully implemented in hardware, or some modules can be implemented in software through processing element calls and some modules can be implemented in hardware.

[0270] In the embodiments provided by this invention, it should be understood that the disclosed apparatus or method can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of the apparatus or module or unit may be electrical, mechanical, or other forms.

[0271] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of the present invention, depending on actual needs. For example, the functional modules / units in the various embodiments of the present invention may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0272] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0273] This invention also provides an electronic device, such as... Figure 9 As shown, the electronic device includes a processor and a memory.

[0274] like Figure 9 As shown, the electronic device includes at least one processor 901, a memory 902, at least one network interface 903, and a user interface 905. The various components in the device are coupled together via a bus system 904. It is understood that the bus system 904 is used to implement communication between these components. In addition to a data bus, the bus system 904 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 9 The general will label all buses as bus systems.

[0275] The user interface 905 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0276] It is understood that memory 902 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0277] In this embodiment of the invention, the memory 902 is used to store various categories of data to support the operation of the electronic device 900. Examples of this data include: any executable program for operation on the electronic device 900, such as the operating system 9021 and application programs 9022; the operating system 9021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 9022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The methods provided in this embodiment of the invention can be included in the application program 9022.

[0278] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 901. Processor 901 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 901 or by instructions in software form. The processor 901 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 901 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 901 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0279] In an exemplary embodiment, the electronic device 900 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0280] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code that, when run on a computer, causes the computer to execute any of the vehicle network channel state information prediction model training methods and / or any of the methods in the embodiments of the vehicle network channel state information prediction method.

[0281] The terms “component,” “module,” “system,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0282] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0283] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0284] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0285] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0286] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0287] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0288] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0289] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0290] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A method for training a vehicle-to-everything (V2X) channel state information prediction model, characterized in that, include: Obtain a training set, a validation set, and a constructed prediction model, wherein the training set includes at least sample data and sample labels; The training set is input into the prediction model for training, wherein... Based on the training set, feature extraction is performed to obtain target features, which include at least vehicle position features, camera fusion features, and lidar fusion features. Multiple target features are fused to obtain a fused feature, and channel state decoding is performed to obtain the amplitude prediction result; and After each round of training, the validation set is used for validation to obtain a trained prediction model based on the model parameters with the lowest validation error.

2. The method for training a vehicle-to-everything (V2X) channel state information prediction model according to claim 1, characterized in that, The acquisition of the training set specifically includes: The sample data includes vehicle location, three-view camera images, and raw lidar observation data, among which, The vehicle location is used to characterize the spatial relationship between the vehicle and the roadside base station, and the calculation formula is as follows: ； in, This represents the two-dimensional position of the vehicle in the nth sample. and These represent the vehicle's horizontal and vertical coordinates in the scene coordinate system, respectively. The three-view camera image is represented as ,in, and These represent the left, middle, and right view camera images of the nth sample, respectively. The original lidar observation data is represented as follows: ,in, This represents the original lidar observation corresponding to the nth sample; The sample label includes the original complex channel state information corresponding to the sample data. The original complex channel state information is converted into amplitude and normalized to obtain the channel state information label, calculated as follows: ； in, This represents the original complex channel state information corresponding to the nth sample. This represents the amplitude of the channel state information for the nth sample. This indicates normalization processing. This represents the normalized channel state information amplitude label of the nth sample.

3. The method for training a vehicle-to-everything (V2X) channel state information prediction model according to claim 2, characterized in that, Constructing the prediction model specifically includes: The prediction model includes a vehicle location feature extraction module, a camera feature extraction module, a view-gated fusion module, a LiDAR range view module, a geometric contour bi-branch coding module, a scalar confidence-gated residual fusion module, and a channel state information decoding module. The vehicle location feature extraction module is used to map the vehicle location into vehicle location features using a multilayer perceptron structure. The camera feature extraction module is used to process the three-view camera images to obtain three-way camera features; The view gating fusion module is used to generate left, center, and right view weights based on the vehicle position features, and to perform weighted fusion of the three camera features to obtain camera fusion features. The lidar range view module is used to convert the original lidar observation data into a range view and generate an effective point mask corresponding to the range view; The geometric contour dual-branch encoding module is used to extract spatial structure information and geometric statistical information, and to combine them with the vehicle position features and the camera fusion features to obtain the lidar fusion features. The scalar confidence gated residual fusion module is used to fuse the vehicle position features, the camera fusion features, and the lidar fusion features to generate the final fused features; The channel state information decoding module is used to map the final fused features into a normalized channel state information amplitude prediction result.

4. The method for training a vehicle-to-everything (V2X) channel state information prediction model according to claim 1, characterized in that, The step of extracting target features based on the training set specifically includes: The vehicle position features are extracted by preprocessing the vehicle position coordinates and then inputting them into a vehicle position feature extraction network to obtain the vehicle position features. The calculation formula is as follows: ； in, This represents the vehicle location feature extraction network. This represents the vehicle position coordinates of the nth sample. This represents the vehicle location features of the nth sample; The camera fusion features are extracted by weighting and fusing the features from the three camera views according to the three-view weights, as shown in the following formula: ； ； in, This represents the camera fusion feature of the nth sample. and Let be the three-view weights of the nth sample, and , , and These represent the feature extraction branches for the left, center, and right view cameras, respectively. and These represent the left, middle, and right view camera images of the nth sample, respectively. Extracting the fusion features of the lidar, wherein the joint features are calculated and input into the lidar fusion network to obtain the lidar fusion features, the calculation formula is as follows: ； ； ， ； ； ； ； ； in, This represents the LiDAR fusion feature of the nth sample. This indicates a lidar fusion network. Represents the joint features of the nth sample. This represents the concatenation function. Indicates the geometric contour coding branch, This represents the geometric contour encoding features of the nth sample. Indicates the distance view encoding branch, This represents the distance view feature of the nth sample. Represents the geometric contour features of the nth sample. This represents the nearest distance feature of the nth sample. This represents the average distance feature of the nth sample. This represents the distance standard deviation characteristic of the nth sample. This represents the proportion of valid points in the nth sample. This represents the minimum effective pitch position feature of the nth sample. This represents the normalized azimuth location encoding feature of the nth sample. This represents the input of the LiDAR range view branch for the nth sample. This represents the normalized distance view of the nth sample. This indicates that the valid points have been normalized. This indicates data transformation and processing. This represents the original lidar observation data of the nth sample. This represents the distance view of the nth sample obtained by converting the original lidar observations. This represents the effective point mask corresponding to the distance view for the nth sample.

5. The method for training a vehicle-to-everything (V2X) channel state information prediction model according to claim 1, characterized in that, The process of fusing multiple target features to obtain a fused feature and performing channel state decoding to obtain an amplitude prediction result specifically includes: Multiple target features are fused to obtain a fused feature, calculated as follows: ； ； ； ； in, This represents the fusion feature of the nth sample. This represents the basic representation of the nth sample, generated from vehicle location features and camera fusion features. Let represent the scalar confidence gating coefficient corresponding to the nth sample. This represents the residual feature of the nth sample generated with the participation of LiDAR information. This represents the vehicle location features of the nth sample. This represents the camera fusion feature of the nth sample. This represents the concatenation function. Representing the basic representation network, Indicates a gating network. express Activation function This represents the preset minimum gate value. This represents the LiDAR fusion feature of the nth sample. Represents the residual feature network; The fused features are input into the channel state information decoder to obtain the normalized channel state information amplitude prediction result, calculated as follows: ； in, This represents the normalized channel state information amplitude prediction result for the nth sample. The channel state information decoder, This represents the fusion feature of the nth sample.

6. The method for training a vehicle-to-everything (V2X) channel state information prediction model according to claim 1, characterized in that, The total loss during training is calculated as follows: ； ； ； ； in, Indicates the total loss. Indicates the main loss. Indicates the auxiliary loss weight, This indicates that lidar-assisted loss prediction is performed. Indicates the gating regularization weight. Indicates the gating regularization term. This represents the normalized channel state information amplitude label of the nth sample. This represents the normalized channel state information amplitude prediction result for the nth sample. This represents the normalized channel state information amplitude prediction result of the nth sample output by the auxiliary prediction head. Indicates the auxiliary prediction head. This represents the LiDAR fusion feature of the nth sample. This represents the mean square error.

7. A method for predicting channel state information in a vehicle-to-everything (V2X) network, characterized in that, The method for training a vehicle-to-everything (V2X) channel state information prediction model according to any one of claims 1-6 is applied to the V2X channel state information prediction model training method, wherein the method includes: Acquire multimodal input information from vehicles in the Internet of Vehicles (IoV); The multimodal input information is input into the trained vehicle network channel state information prediction model to obtain the prediction result; Output the prediction result, wherein the prediction result includes the normalized channel state information amplitude.

8. A vehicle-to-everything (V2X) channel state information prediction system, characterized in that, include: The acquisition module is used to acquire multimodal input information of vehicles in the Internet of Vehicles (IoV). The prediction module is used to input the multimodal input information into the trained vehicle network channel state information prediction model to obtain the prediction result; The output module is used to output the prediction result, wherein the prediction result includes the normalized channel state information amplitude.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the vehicle network channel state information prediction model training method according to any one of claims 1 to 6 and / or the vehicle network channel state information prediction method according to claim 7.

10. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the vehicle network channel state information prediction model training method as described in any one of claims 1 to 6 and / or the vehicle network channel state information prediction method as described in claim 7.

Images