Beam prediction method and device based on multi-modal large model, equipment and medium

Abstract

The application provides a beam prediction method and device based on a multi-modal large model, equipment and a medium. A first multi-modal environment-beam dataset is used to perform fine-tuning training on a beam prediction model based on a multi-modal large model. A small sample training set is used to perform secondary fine-tuning training on the beam prediction model based on the multi-modal large model after the first fine-tuning training. The beam corresponding to a target terminal at a first time is obtained according to the beam prediction model based on the multi-modal large model after the secondary fine-tuning training and target multi-modal environment data of the target terminal at the first time. The beam prediction model based on the multi-modal large model is used to obtain text embedding features and image embedding features according to the target multi-modal environment data, obtain multi-modal features related to beam selection, and map the multi-modal features to a probability distribution of all beams in a beam codebook to select a beam with the maximum probability corresponding to a beam index. The scheme has high prediction accuracy and strong generalization ability.

Description

Technical Field

[0001] The present application relates to the field of wireless communication technology, and in particular to a beam prediction method, apparatus, device and medium based on a multimodal large model. Background Art

[0002] In 5G (5th Generation Mobile Communication Technology), beamforming and Massive MIMO (Massive Multiple Input Multiple Output) technologies are key to addressing the high path loss problem in high-frequency bands. By forming narrow beams, base stations can focus their transmission energy along the desired direction, thereby improving the strength of the received signal. Currently, in mobile scenarios such as the Internet of Vehicles (IoV), the optimal beam direction is constantly updated due to the rapidly changing environment surrounding the vehicle. Two methods exist for obtaining the optimal beam: beam scanning and deep learning-based beam prediction. Beam scanning requires traversing all beams in a predefined codebook, resulting in significant time and resource overhead due to frequent training. Furthermore, it struggles to meet real-time requirements in mobile scenarios. While deep learning-based beam prediction methods can significantly reduce training overhead, they are often limited to specific environments due to the limited size of model parameters and lack generalization capabilities to new scenarios. Summary of the Invention

[0003] At least one embodiment of the present application provides a beam prediction method, apparatus, device, and medium based on a multimodal large model to address the problem that the optimal beam acquisition method in the prior art will lead to increased training overhead and lack of generalization capability for new scenarios.

[0004] In order to solve the above technical problems, this application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a beam prediction method based on a multimodal large model, comprising:

[0006] Using the first multimodal environment-beam dataset, a beam prediction model based on the multimodal large model is fine-tuned and trained to obtain a beam prediction model based on the multimodal large model after the fine-tuning training; wherein the first multimodal environment-beam dataset is obtained by matching each multimodal environment data in the first multimodal environment dataset with the beam index of the maximum received signal power at the corresponding position;

[0007] The beam prediction model based on the multimodal large model after the first fine-tuning training is fine-tuned for the second time using the small sample training set to obtain the beam prediction model based on the multimodal large model after the second fine-tuning training;

[0008] Obtaining a beam corresponding to the target terminal at the first moment based on a beam prediction model based on a multimodal large model after secondary fine-tuning training and the collected target multimodal environment data of the target terminal at the first moment;

[0009] Among them, the beam prediction model based on the multimodal large model is used to obtain text embedding features according to the position information in the target multimodal environment data and image embedding features according to the multi-view images in the target multimodal environment data, and obtain multimodal features related to beam selection based on the text embedding features and the image embedding features, and map the multimodal features to the probability distribution of all beams in the predefined beam codebook, and select the beam corresponding to the beam index with the largest probability as the beam corresponding to the target terminal at the first moment.

[0010] Optionally, the beam prediction method based on a multimodal large model further includes:

[0011] Matching each multimodal environment data in the second multimodal environment data set with the beam index of the maximum received signal power at the corresponding position to obtain a second multimodal environment-beam data set;

[0012] Different proportions of multimodal environment-beam data are selected from the second multimodal environment-beam data set to construct different small sample training sets.

[0013] Optionally, the beam prediction method based on a multimodal large model further includes:

[0014] Acquire the first multimodal environment dataset based on the location information and multi-view images of each terminal in the first wireless propagation scenario collected at intervals of a preset duration; and

[0015] The second multimodal environment data set is obtained based on the location information and multi-perspective images of each terminal in the second wireless propagation scene collected at each preset time interval, and the first wireless propagation scene and the second wireless propagation scene are different wireless propagation scenes.

[0016] Optionally, the beam prediction method based on a multimodal large model, wherein the beam prediction model based on a multimodal large model includes:

[0017] An input preprocessing module, configured to convert the location information in the target multimodal environment data into location text, and to stitch the multi-view images in the target multimodal environment data into a panoramic image;

[0018] A text embedding module, used for processing the position text to obtain text embedding features;

[0019] an image encoding module, configured to perform feature extraction on the panoramic image to obtain image embedding features;

[0020] a large model module, configured to fuse the image embedding features and the text embedding features to obtain fused features, and extract multi-modal features related to beam selection from the fused features;

[0021] an output mapping module, configured to map the multi-modal features to probability distributions of all beams in the beam codebook, select a beam index with the maximum probability, and obtain a beam according to the beam index.

[0022] Optionally, the beam prediction method based on the multi-modal large model, wherein the first multi-modal environment dataset is obtained according to position information and multi-view images of each terminal in a first wireless propagation scene collected every preset time interval, and the first multi-modal environment dataset comprises:

[0023] the first wireless propagation scene is constructed, and the first wireless propagation scene comprises scatterers and moving terminals;

[0024] every preset time interval, position information of each terminal is collected by a sensor of each terminal, and multi-view images of each terminal are collected by a multi-view image collector of each terminal;

[0025] for each terminal, multi-modal environment data of the terminal is obtained by matching each position information with corresponding multi-view images;

[0026] the first multi-modal environment dataset is obtained according to the multi-modal environment data of each terminal in the first wireless propagation scene.

[0027] Optionally, the beam prediction method based on the multi-modal large model, wherein the method further comprises:

[0028] ray tracing simulation is performed on the position of each terminal every preset time interval to generate multi-path channel state information;

[0029] the multi-path channel state information and a predefined beam codebook are used to calculate received signal power corresponding to each beam in the beam codebook;

[0030] a beam index corresponding to the maximum received signal power of the position of each terminal every preset time interval is obtained.

[0031] Optionally, the beam prediction method based on the multimodal large model, wherein the beam prediction model based on the multimodal large model is fine-tuned using the first multimodal environment-beam dataset to obtain the beam prediction model based on the multimodal large model after the fine-tuning training, includes:

[0032] Selecting a training set from the first multimodal environment-beam dataset;

[0033] The training set and LoRA (Low-Rank Adaptation) technology are used to fine-tune the weights of the beam prediction model based on the multimodal large model to obtain the beam prediction model based on the multimodal large model after one fine-tuning training.

[0034] Optionally, in the beam prediction method based on a multimodal large model, after fine-tuning the beam prediction model based on the multimodal large model using the first multimodal environment-beam dataset to obtain the beam prediction model based on the multimodal large model after fine-tuning, the method further includes at least one of the following:

[0035] The accuracy rate is used to evaluate the consistency between the beam index obtained by the beam prediction model based on the multimodal large model after a fine-tuning training and the true index;

[0036] Evaluate the complexity of a beam prediction model based on a multimodal large model after a single fine-tuning training based on at least one of the inference time, video memory usage, and computing resource consumption.

[0037] In a second aspect, an embodiment of the present application further provides a beam prediction device based on a multimodal large model, comprising:

[0038] A first training module is configured to perform a fine-tuning training on a beam prediction model based on a multimodal large model using a first multimodal environment-beam dataset to obtain a beam prediction model based on the multimodal large model after the fine-tuning training; wherein the first multimodal environment-beam dataset is obtained by matching each multimodal environment data in the first multimodal environment dataset with a beam index of a maximum received signal power at a corresponding position;

[0039] The second training module is used to perform a second fine-tuning training on the beam prediction model based on the multimodal large model after the first fine-tuning training using the small sample training set, so as to obtain the beam prediction model based on the multimodal large model after the second fine-tuning training;

[0040] A prediction module, which obtains the beam of the target terminal at the first moment based on the beam prediction model based on the multimodal large model after secondary fine-tuning training and the collected target multimodal environment data of the target terminal at the first moment;

[0041] Among them, the beam prediction model based on the multimodal large model is used to convert the location information in the target multimodal environment data into text embedding features and convert the multi-view images in the target multimodal environment data into image embedding features, and obtain multimodal features related to beam selection based on the text embedding features and the image embedding features, and map the multimodal features to the probability distribution of all beams in the predefined beam codebook, and select the beam corresponding to the beam index with the largest probability as the beam of the target terminal at the first moment.

[0042] In a third aspect, an embodiment of the present application also provides a beam prediction device based on a multimodal large model, comprising: a processor, a memory, and a program stored on the memory and executable on the processor, wherein when the program is executed by the processor, the beam prediction method based on the multimodal large model as described in the first aspect is implemented.

[0043] In a fourth aspect, an embodiment of the present application further provides a readable storage medium having a program stored thereon, and when the program is executed by a processor, the beam prediction method based on a multimodal large model as described in the first aspect is implemented.

[0044] In a fifth aspect, an embodiment of the present application further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the beam prediction method based on a multimodal large model as described in the first aspect.

[0045] Compared with the prior art, the embodiments of the present application provide a beam prediction method, apparatus, device and medium based on a multimodal large model, wherein the method comprises: using a first multimodal environment-beam data set to perform a fine-tuning training on a beam prediction model based on a multimodal large model, and obtaining a beam prediction model based on a multimodal large model after a fine-tuning training; wherein the first multimodal environment-beam data set is obtained by matching each multimodal environment data in the first multimodal environment data set with a beam index of the maximum received signal power at the corresponding position; using a small sample training set to perform a second fine-tuning training on the beam prediction model based on a multimodal large model after a fine-tuning training, and obtaining a beam prediction model based on a multimodal large model after a second fine-tuning training; according to the beam prediction model based on the multimodal large model after a second fine-tuning training and the collected target terminal at the first moment, Target multimodal environment data, obtaining the beam corresponding to the target terminal at the first moment; the beam prediction model based on the multimodal large model is used to obtain text embedding features according to the position information in the target multimodal environment data and image embedding features according to the multi-view images in the target multimodal environment data, and obtain multimodal features related to beam selection based on the text embedding features and the image embedding features, and map the multimodal features to the probability distribution of all beams in a predefined beam codebook, and select the beam corresponding to the beam index with the largest probability as the beam corresponding to the target terminal at the first moment. In this way, the beam prediction accuracy can be improved through the target multimodal environment data and the beam prediction model based on the multimodal large model, and the beam prediction model based on the multimodal large model can avoid the problems of increased training overhead and lack of generalization ability for new scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0047] Figure 1 Schematic diagram of the flow of the beam prediction method based on the multimodal large model described in an embodiment of the present application;

[0048] Figure 2 This is a schematic diagram of the architecture of the beam prediction model based on the multimodal large model described in an embodiment of the present application;

[0049] Figure 3 This is a flow chart of one implementation of the beam prediction method based on a multimodal large model described in an embodiment of the present application;

[0050] Figure 4This is a schematic diagram of the structure of the beam prediction device based on the multimodal large model described in an embodiment of the present application;

[0051] Figure 5 This is a hardware block diagram of the beam prediction device based on a multimodal large model described in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and do not limit the number of objects, for example, the first object can be one or more. In addition, "or" in this application represents at least one of the connected objects. For example, "A or B" covers three options, namely, Option 1: including A but not including B; Option 2: including B but not including A; Option 3: including both A and B. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0053] It should be noted that the above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0054] Please refer to Figure 1 , an embodiment of the present application provides a beam prediction method based on a multimodal large model, comprising:

[0055] Step 101: Fine-tune a beam prediction model based on a multimodal large model using a first multimodal environment-beam dataset to obtain a beam prediction model based on the multimodal large model after fine-tuning. The first multimodal environment-beam dataset is obtained by matching each multimodal environment data in the first multimodal environment dataset with a beam index of a maximum received signal power at a corresponding position.

[0056] Step 102: performing a second fine-tuning training on the beam prediction model based on the multimodal large model after the first fine-tuning training using the small sample training set to obtain the beam prediction model based on the multimodal large model after the second fine-tuning training;

[0057] Step 103: obtaining a beam corresponding to the target terminal at the first moment based on the beam prediction model based on the multimodal large model after secondary fine-tuning training and the collected target multimodal environment data of the target terminal at the first moment;

[0058] Among them, the beam prediction model based on the multimodal large model is used to obtain text embedding features according to the position information in the target multimodal environment data and image embedding features according to the multi-view images in the target multimodal environment data, and obtain multimodal features related to beam selection based on the text embedding features and the image embedding features, and map the multimodal features to the probability distribution of all beams in the predefined beam codebook, and select the beam corresponding to the beam index with the largest probability as the beam corresponding to the target terminal at the first moment.

[0059] It should be noted that by inputting the target multimodal environment data of the target terminal at the first moment into the beam prediction model based on the multimodal large model, the beam corresponding to the target terminal at the first moment can be output. The beam corresponding to the target terminal at the first moment is the optimal beam, which is the beam corresponding to the beam index with the highest probability in the beam codebook.

[0060] In the embodiment of the present application, optionally, before fine-tuning the beam prediction model based on the multimodal large model using the first multimodal environment-beam dataset, the method further includes:

[0061] Constructing the first wireless propagation scenario, where the first wireless propagation scenario includes a scatterer and a moving terminal;

[0062] At intervals of a preset duration, the location information of each terminal is collected by a sensor of each terminal, and the multi-view image collector of each terminal is collected by a multi-view image collector of each terminal;

[0063] For each of the terminals, matching each piece of location information with the corresponding multi-view image to obtain multimodal environment data of the terminal;

[0064] The first multimodal environment data set is acquired according to the multimodal environment data of each of the terminals in the first wireless propagation scenario.

[0065] The scatterers in the embodiment of the present application can be buildings, and the moving terminals can be vehicles, so the first wireless propagation scenario can be understood as a 3D vehicle network simulation scenario. In addition to scatterers and moving terminals, the first wireless propagation scenario can also include lanes, with multiple vehicles traveling on different lanes. Multi-view images and position information are collected regularly by on-board cameras and positioning devices to obtain multimodal environmental data containing information about scatterers around the vehicle, providing data support for subsequent beam prediction. Specifically, obtaining multimodal environmental data of the terminal includes the following steps:

[0066] Use 3D modeling software to build a city scene map. The scene size is, for example, 200 meters x 200 meters, including 8 lanes and 4 building clusters. Each building cluster contains buildings of different sizes and types to simulate the distribution of scatterers in a real communication environment. This constructed city scene map is the first wireless propagation scenario.

[0067] The constructed first wireless propagation scenario was imported into the autonomous driving simulator CARLA to simulate vehicle motion and sensor data collection. For example, three vehicles were set up to travel at different speeds in different lanes to simulate a dynamic vehicle-to-vehicle communication environment. Six onboard cameras were mounted on the roof of each vehicle, capturing multi-view images at 0°, 60°, 120°, 180°, 240°, and 300°, achieving 360° panoramic perception. The image acquisition interval was 0.02 seconds, with a resolution of 800 × 600 pixels. The positioning device also recorded the vehicle's three-dimensional position at each moment. Each vehicle collected data from 1,891 sample points, generating multimodal environmental data containing multi-view images and position information.

[0068] In the embodiment of the present application, optionally, the above method further includes:

[0069] Perform ray tracing simulation on the position of each terminal at intervals of a preset time length to generate multipath channel state information;

[0070] Calculating a received signal power corresponding to each beam in the beam codebook using the multipath channel state information and a predefined beam codebook;

[0071] Obtain a beam index of a maximum received signal power corresponding to the position of each terminal at intervals of a preset time length.

[0072] In the embodiment of the present application, the ray tracing software Wireless Insite is used to perform ray tracing simulation on the position of each terminal in the first wireless propagation scenario at each sampling time, generate multipath channel state information, and calculate the received signal power in combination with a predefined beam codebook to obtain the optimal beam index, that is, the beam index with the maximum received signal power. This is then paired with the multimodal environment data to construct a first multimodal environment-beam dataset, providing data support for subsequent beam prediction model training. Specifically, constructing the first multimodal environment-beam dataset includes the following steps:

[0073] The first wireless propagation scenario was imported into the ray tracing software Wireless Insite. The simulation frequency band was set to 28 GHz. The transmitting antenna was a uniform planar array with 64 antennas, fixed 6 meters above the ground next to the lane. The receiving antenna was a single antenna, fixed 1.5 meters above the ground on top of the vehicle. The transmitting and receiving ends used OFDM (Orthogonal Frequency Division Multiplexing) for information transmission. The downlink channel state information of the kth subcarrier can be expressed as follows:

[0074]

[0075] Among them, α l , τ l and ψ l are the attenuation, time delay and phase shift of the lth path respectively; f k is the frequency of the kth subcarrier; L is the multipath number; θ l and φ l denote the azimuth and elevation angles of the lth path respectively; a(θ l ,φ l ) is the steering vector of the transmitting antenna array. After the ray tracing simulation is completed, the downlink channel state information of each subcarrier is calculated based on the multipath channel state information.

[0076] The received signal power corresponding to each beam is calculated using a predefined beam codebook with 64 beams and the obtained multipath channel state information. The optimal beam is obtained by the beam index corresponding to the maximum received signal power, which can be expressed as follows:

[0077]

[0078] Among them, f * is the optimal beamforming vector; F is the predefined beam codebook; N s is the number of subcarriers; f is a beamforming vector in a predefined beam codebook. Based on the position of the receiving antenna, the optimal beam index is paired with the multimodal environment data at the same location to construct the first multimodal environment-beam dataset.

[0079] In an embodiment of the present application, optionally, the beam prediction model based on the multimodal large model includes:

[0080] An input preprocessing module, configured to convert the location information in the target multimodal environment data into location text, and to stitch the multi-view images in the target multimodal environment data into a panoramic image;

[0081] A text embedding module, used for processing the position text to obtain text embedding features;

[0082] An image encoding module, configured to extract features from the panoramic image to obtain image embedding features;

[0083] A large model module, configured to fuse the image embedding features and the text embedding features to obtain fused features, and extract multimodal features related to beam selection from the fused features;

[0084] An output mapping module maps the multimodal feature to the probability distribution of all beams in the beam codebook, selects the beam index with the highest probability, and obtains the beam according to the beam index.

[0085] Figure 2 This is a schematic diagram of the architecture of the beam prediction model based on the multimodal large model described in the embodiment of this application. Figure 2 As shown, an embodiment of the present application provides a beam prediction model based on a multimodal large model. The beam prediction model based on the multimodal large model can be constructed based on the DeepSeek Janus-Pro-1B model, and an input preprocessing module for position information and multi-view images is designed to convert the position information into position text, and to splice the multi-view images into a panoramic image. The position text is converted into text embedding features by the word segmenter and high-dimensional mapper in the text embedding module, and the panoramic image is converted into image embedding features by the encoder and aligner in the image encoding module. The text embedding features and the image embedding features are multimodally fused and key features are extracted through the decoder in the large model module to obtain multimodal features. An output mapping module is designed to map the multimodal features output by the large model into the probability distribution of each beam in the predefined beam codebook. The beam corresponding to the beam index with the largest probability is the optimal beam as the beam corresponding to the target terminal at the first moment. Specifically, the processing process of the beam prediction model based on the multimodal large model is as follows:

[0086] Design the input preprocessing module. Since the DeepSeek Janus-Pro-1B model requires the input image to have a fixed size, the six multi-view images are uniformly scaled to the same size and then horizontally stitched into a single panoramic image I∈R 384×384×3 , as the input of the image encoding module. The vehicle location information is converted from numerical values ​​to natural language description to obtain the location text T, which is used as the input of the text embedding module.

[0087] Use the text embedding module to process the position text T and convert it into a discrete sequence through the word segmenter Among them L pdenotes the length of the sequence and the discrete sequence is mapped to an embedding vector in a high-dimensional continuous space by a high-dimensional mapper, to obtain a text embedding feature where d m is the embedding dimension of the large model module.

[0088] The image coding module is used for feature extraction of the panoramic image I, the image is divided into 576 image blocks of the same size and embedded into a high-dimensional space, to obtain image block embedding features where d v denotes the embedding dimension of the image encoder. Subsequently, the semantic information of the image is extracted through an encoder composed of 24 encoder blocks, and the output of the encoder is projected into a space with a dimension of d m by an aligner, to obtain image embedding features

[0089] The image embedding features I e and the text embedding features T e are fused by the large model module, a decoder composed of 24 decoder blocks is used to establish the relationship between the two modalities and extract key features related to beam selection, to obtain multi-modal features output by the large model

[0090] The output mapping module is designed to project the high-dimensional output of the large model into the probability distribution of all beams in the predefined beam codebook. By calculating the average value, the output multi-modal features L o of the large model module are simplified to a matrix Subsequently, the matrix L m is obtained through an MLP (Multilayer Perceptron) with three hidden layers to obtain the final output P of the entire model, i.e., the probability distribution of the optimal beam index. Specifically, this process can be represented as follows:

[0091] P=Softmax(MLP(L m ))

[0092] In the embodiments of the present application, the beam prediction model based on the multi-modal large model can be optionally fine-tuned once using the first multi-modal environment-beam dataset to obtain the beam prediction model based on the multi-modal large model after fine-tuning once, including:

[0093] Selecting a training set from the first multi-modal environment-beam dataset;

[0094] Fine-tuning the weights of the beam prediction model based on the multi-modal large model using the training set and the LoRA technology to obtain the beam prediction model based on the multi-modal large model after fine-tuning once.

[0095] In an embodiment of the present application, the first multimodal environment-beam dataset constructed is used to fine-tune the beam prediction model based on the multimodal large model, and the LoRA technology is used to fine-tune the attention weight in the image coding module. Only some layers in the image coding module and the large model module that participate in task learning are unfrozen to reduce computational overhead. At the same time, a suitable loss function is selected to optimize the difference between the predicted beam and the true value. Specifically, the first multimodal environment-beam dataset is used to perform a fine-tuning training on the beam prediction model based on the multimodal large model, including:

[0096] The entire first multimodal environment-beam dataset is divided into a training set, a validation set, and a test set in a ratio of 7:1:2. The training set includes 3971 sample data, the validation set includes 567 sample data, and the test set includes 1135 sample data.

[0097] The LoRA technology is used to fine-tune the weights of the query matrix, key matrix, and value matrix of the self-attention part in the image encoding module. The pre-trained weight matrix is Among them, d v is the embedding dimension of the image coding module. Specifically, the process of fine-tuning the weights of the beam prediction model based on the multimodal large model using LoRA technology can be expressed as the following formula:

[0098]

[0099] Among them, the matrix and matrix Contains trainable parameters; r and α represent the rank and scaling factor of LoRA respectively. Usually, r is much smaller than d v , matrix B is initialized to a zero matrix, and matrix A is initialized using a random Gaussian distribution. During the training process of the beam prediction model based on the multimodal large model, the image encoding module and the large model module only unfreeze the weights of the normalization layer and the root mean square normalization layer to save computing resources.

[0100] The cross entropy function is used as the loss function in the model training process to minimize the difference between the predicted beam index and the true index during the model training process. The cross entropy function can be expressed as the following formula:

[0101]

[0102] Where M is the number of beams in the codebook; is the predicted best beam probability distribution; P is the true best beam probability distribution. Here, the best beam is the beam with the highest probability. The model was trained using the Adam optimizer with a batch size of 10, a learning rate of 0.0001, and 200 epochs.

[0103] In an embodiment of the present application, optionally, after fine-tuning the beam prediction model based on the multimodal large model using the first multimodal environment-beam dataset to obtain the beam prediction model based on the multimodal large model after fine-tuning, the method further includes at least one of the following:

[0104] The accuracy rate is used to evaluate the consistency between the beam index obtained by the beam prediction model based on the multimodal large model after a fine-tuning training and the true index;

[0105] Evaluate the complexity of a beam prediction model based on a multimodal large model after a single fine-tuning training based on at least one of the inference time, video memory usage, and computing resource consumption.

[0106] It should be noted that the beam prediction model based on the multimodal large model in the embodiment of the present application is comprehensively evaluated from the two dimensions of accuracy and resource consumption. The accuracy of the model is evaluated by comparing the consistency of the predicted beam with the actual beam, and the model complexity is analyzed from the aspects of inference time and video memory usage to verify its feasibility of deployment in the actual system. Specifically, the comprehensive evaluation of the beam prediction model based on the multimodal large model includes:

[0107] The Top-K accuracy is used to evaluate the accuracy of the model by calculating the probability that the correct optimal beam index appears in the top k highest prediction probabilities. The Top-K accuracy can be expressed as follows:

[0108]

[0109] Where D is the total number of samples, and I(·) is an indicator function that returns 1 if the true beam index is within the range of the top k highest prediction probabilities, and 0 otherwise. The top-1 accuracy of the beam prediction model based on the multimodal large model in the embodiment of the present application quickly reached approximately 97.5% after 20 rounds of training and remained stable during subsequent training, ultimately reaching 98.1%.

[0110] The required computing power and model inference time are used to evaluate the complexity of the model. The beam prediction model based on the multimodal large model in the embodiment of the present application requires 9.2 TFLOPs of computing power and the inference time for each sample data is 8.66ms.

[0111] In the embodiment of the present application, optionally, the above method further includes:

[0112] Matching each multimodal environment data in the second multimodal environment data set with the beam index of the maximum received signal power at the corresponding position to obtain a second multimodal environment-beam data set;

[0113] Different proportions of multimodal environment-beam data are selected from the second multimodal environment-beam dataset to construct different small sample training sets.

[0114] It should be noted that the second multimodal environment dataset can be obtained using the method for obtaining the first multimodal environment dataset described above, which will not be repeated here. The second multimodal environment-beam dataset is a dataset collected based on a second wireless propagation scenario in addition to the first wireless propagation scenario, and is used to evaluate the small sample generalization ability of the model. The second wireless propagation scenario can be a real scenario.

[0115] In the embodiments of the present application, the generalization ability of the beam prediction model based on the multimodal large model under small sample conditions is evaluated. Small sample training sets of different proportions are constructed, and the model is fine-tuned and trained on this basis. Its performance is evaluated by accuracy indicators to verify the adaptability of the model under low data volume conditions in real scenarios. Specifically, the beam prediction model based on the multimodal large model, which has been fine-tuned once, is fine-tuned again using the small sample training set, including:

[0116] From the second multimodal environment-beam dataset collected from the second wireless propagation scenario, we randomly selected 10%, 20%, and 30% of the samples to construct different small sample training sets. For each data size setting, we evaluated the model's accuracy to measure the model's generalization ability.

[0117] The model was fine-tuned and trained under the above-mentioned different small sample conditions, and the Top-1 accuracy and Top-3 accuracy were used as evaluation indicators. When the sample ratio was 10%, the Top-1 accuracy and Top-3 accuracy of the beam prediction model in the embodiment of the present application were 60.2% and 79.9% respectively. When the sample ratio was 20%, the Top-1 accuracy and Top-3 accuracy of the model were 68.7% and 85.1% respectively. When the sample ratio was 30%, the Top-1 accuracy and Top-3 accuracy of the model were 72.7% and 92.4% respectively.

[0118] Figure 3 This is a flow chart of one embodiment of the beam prediction method based on a multimodal large model described in the embodiment of the present application. Figure 3 As shown, the method includes:

[0119] Step 301: construct a first wireless propagation scenario and obtain first multimodal environment data. Specifically, construct a wireless propagation scenario map, plan the locations of lanes and building complexes in the scenario, and arrange a variety of scatterers in each building complex, including static or dynamic objects of different geometric shapes and materials, to reconstruct the first wireless propagation environment. In the constructed wireless propagation scenario, place multiple terminals in different lanes and move at different speeds, and place a certain number of cameras on the top of the terminals to capture multi-view images at fixed time intervals. Save the location information and multi-view images of each terminal as the first multimodal environment data.

[0120] Step 302 constructs a first multimodal environment-beam dataset. Specifically, using a ray tracing channel simulation tool, a base station is placed as a transmitter at a suitable location, and a receiving antenna is arranged at the terminal's location at the corresponding moment. After the ray tracing simulation is completed, channel information, including multipath channel state information and path loss, is saved. Based on a predefined beam codebook and the multipath information collected by ray tracing, the received signal power corresponding to each beamforming vector is calculated. The index corresponding to the beamforming vector with the highest received signal power is saved, and the optimal beam index is associated with the environmental information at the same location to form the first multimodal environment-beam dataset.

[0121] Step 303: Establish a beam prediction model based on a multimodal large model. Specifically, design an input preprocessing module based on the architecture of the multimodal large model, resize multiple input images and splice them into a panoramic image as input to the image encoding module. At the same time, convert the location information from a numerical value into location text as input to the text embedding module. Use the word segmenter of the text embedding module to convert the location text into a discrete sequence, and map the discrete sequence into a high-dimensional continuous representation to obtain text embedding features containing terminal location information. Use the encoder of the image encoding module to extract environmental semantic information from the panoramic image, including environmental features such as occlusion and scatterer distribution related to channel propagation, to obtain image embedding features that can reflect the wireless propagation environment. Use the decoder of the large model module to fuse different modal information from the text embedding features and image embedding features, and extract key features that affect beam selection. Design an output mapping module based on the output of the large model module and the number of beams in the predefined beam codebook, and map the high-dimensional output of the large model to the probability distribution of all beams in the beam codebook. The beam corresponding to the index with the highest probability is the optimal beam.

[0122] Step 304, using the first multimodal environment-dataset to fine-tune the beam prediction model based on the multimodal large model; specifically, the first multimodal environment-dataset is divided into a training set, a validation set, and a test set according to an appropriate ratio; the LoRA technology is used to fine-tune the weights of the multi-head self-attention part in the image coding module, and only the weights of specific layers of the image coding module and the large model module are unfrozen during the training process to save computing resources; a suitable loss function is selected to minimize the difference between the predicted beam index and the true value during the model training process.

[0123] Step 305: Evaluate the accuracy and complexity of the beam prediction model based on the multimodal large model. Specifically, use the accuracy evaluation model to evaluate the consistency between the beam index output by the model and the true optimal beam to measure its prediction accuracy. Evaluate the complexity of the model from aspects such as inference time, video memory usage, and computing resource consumption to verify its feasibility of deployment in an actual system.

[0124] Step 306: Based on the second multimodal environment-beam dataset, the small sample generalization ability of the beam prediction model based on the multimodal large model is evaluated. Specifically, a small amount of data from the second multimodal environment-beam dataset, such as 10%, 20%, 30%, etc., is randomly selected to construct small sample training sets with different data amounts. Under the corresponding small sample conditions, the model is fine-tuned and trained, and its performance is evaluated using accuracy to verify the generalization ability of the model under low data amount conditions.

[0125] In summary, the beam prediction method based on the multimodal large model described in the embodiment of the present application is adopted to make full use of the location information and multi-view images collected by the terminal, and to perform joint modeling through the multimodal large model architecture, so that the model can understand and characterize complex communication environment characteristics. Environmental characteristics mainly include factors such as the distribution of obstructions and building structures that have a significant impact on the signal propagation path. Through the deep fusion of multi-view images and location information, the adaptability of the beam prediction model to environmental changes is effectively improved, and the limitations of a single modality in characterization capabilities are overcome. In addition, the embodiment of the present application adopts a multimodal large model architecture based on DeepSeek Janus-Pro-1B, and combines LoRA technology, so that the model can more accurately capture the key environmental features that affect beam selection while retaining pre-training knowledge. Through the fusion of multi-view images and location information, and the powerful knowledge transfer capability of the large model, the discrimination performance of the optimal beam is enhanced, and the model has higher prediction accuracy. In addition, the embodiment of the present application uses parameter fine-tuning based on a large model to perform transfer learning, so that the model can still maintain good prediction performance when facing new scenarios with limited data. This small sample adaptability significantly enhances the practicality of the model, making it suitable for communication needs in changing environments such as real vehicle networks, and effectively alleviating the problem of current deep learning-based beam prediction methods relying on large-scale data.

[0126] Please refer to Figure 4 , the embodiment of the present application further provides a beam prediction device based on a multimodal large model, comprising:

[0127] A first training module 401 is configured to perform a fine-tuning training on a beam prediction model based on a multimodal large model using a first multimodal environment-beam dataset to obtain a beam prediction model based on the multimodal large model after the fine-tuning training; wherein the first multimodal environment-beam dataset is obtained by matching each multimodal environment data in the first multimodal environment dataset with a beam index of a maximum received signal power at a corresponding position;

[0128] The second training module 402 is configured to perform a second fine-tuning training on the beam prediction model based on the multimodal large model after the first fine-tuning training using the small sample training set, thereby obtaining the beam prediction model based on the multimodal large model after the second fine-tuning training;

[0129] The prediction module 403 obtains a beam of the target terminal at the first time according to a multi-modal large model-based beam prediction model trained by secondary fine tuning and target multi-modal environment data of the target terminal collected at the first time; wherein the multi-modal large model-based beam prediction model is used to convert position information in the target multi-modal environment data into text embedding features and convert multi-view images in the target multi-modal environment data into image embedding features, obtain multi-modal features related to beam selection according to the text embedding features and the image embedding features, and map the multi-modal features to a probability distribution of all beams in a predefined beam codebook, and select a beam corresponding to a beam index with the maximum probability as the beam of the target terminal at the first time.

[0130] Optionally, the multi-modal large model-based beam prediction device, wherein the device further comprises:

[0131] The obtaining module is configured to match each multi-modal environment data in the second multi-modal environment data set with a beam index of maximum received signal power of a corresponding position to obtain a second multi-modal environment-beam data set.

[0132] The construction module is configured to select different proportions of multi-modal environment-beam data from the second multi-modal environment-beam data set to construct different small sample training sets.

[0133] Optionally, the multi-modal large model-based beam prediction device, wherein the device further comprises:

[0134] The first obtaining module is configured to obtain a first multi-modal environment data set according to position information and multi-view images of each terminal in a first wireless propagation scene collected every interval of a preset time length; and

[0135] The second multi-modal environment data set is obtained according to position information and multi-view images of each terminal in a second wireless propagation scene collected every interval of the preset time length, and the first wireless propagation scene and the second wireless propagation scene are different wireless propagation scenes.

[0136] Optionally, the multi-modal large model-based beam prediction device, wherein the multi-modal large model-based beam prediction model comprises:

[0137] The input preprocessing module is configured to convert position information in the target multi-modal environment data into position text and splice multi-view images in the target multi-modal environment data into panoramic images.

[0138] The text embedding module is configured to process the position text to obtain text embedding features.

[0139] An image encoding module, configured to extract features from the panoramic image to obtain image embedding features;

[0140] A large model module, configured to fuse the image embedding features and the text embedding features to obtain fused features, and extract multimodal features related to beam selection from the fused features;

[0141] An output mapping module maps the multimodal feature to the probability distribution of all beams in the beam codebook, selects the beam index with the highest probability, and obtains the beam according to the beam index.

[0142] Optionally, in the beam prediction device based on a multimodal large model, the first acquisition module is specifically configured to:

[0143] Constructing the first wireless propagation scenario, where the first wireless propagation scenario includes a scatterer and a moving terminal;

[0144] At intervals of the preset time length, the location information of each terminal is collected by a sensor of each terminal, and the multi-view image collector of each terminal is collected by a multi-view image collector of each terminal;

[0145] For each of the terminals, matching each piece of location information with the corresponding multi-view image to obtain multimodal environment data of the terminal;

[0146] The first multimodal environment data set is acquired according to the multimodal environment data of each of the terminals in the first wireless propagation scenario.

[0147] Optionally, the beam prediction device based on a multimodal large model further comprises:

[0148] A generation module, configured to perform ray tracing simulation on the position of each terminal at intervals of a preset time length to generate multipath channel state information;

[0149] a calculation module, configured to calculate a received signal power corresponding to each beam in the beam codebook using the multipath channel state information and a predefined beam codebook;

[0150] The second acquisition module is used to obtain the beam index of the maximum received signal power corresponding to the position of each terminal at intervals of a preset time length.

[0151] Optionally, in the beam prediction device based on a multimodal large model, the first training module is specifically used to:

[0152] Selecting a training set from the first multimodal environment-beam dataset;

[0153] The training set and the low-rank adaptive technology are used to fine-tune the weights of the beam prediction model based on the multimodal large model to obtain the beam prediction model based on the multimodal large model after one fine-tuning training.

[0154] Optionally, the beam prediction device based on a multimodal large model further comprises at least one of the following:

[0155] A first evaluation module is used to evaluate the consistency between the beam index obtained by the beam prediction model based on the multimodal large model after a fine-tuning training and the true index using the accuracy rate;

[0156] The second evaluation module is used to evaluate the complexity of the beam prediction model based on the multimodal large model after one fine-tuning training based on at least one of the inference time, video memory occupancy and computing resource consumption.

[0157] It should be noted that the above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0158] The embodiment of the present application also provides a beam prediction device based on a multimodal large model, such as Figure 5 Shown, including:

[0159] A processor 501, a memory 502, a transceiver 503, and a program or instruction stored on the memory 502 and executable on the processor 501; when the processor 501 executes the program or instruction, each process of the embodiment of the beam prediction method based on the multimodal large model is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0160] The transceiver 503 is configured to receive and send data under the control of the processor 501 .

[0161] Among them, Figure 5In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically various circuits of one or more processors represented by processor 501 and memory represented by memory 502, which are linked together. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 503 may be a plurality of components, i.e., a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. For different user devices, the user interface 504 may also be an interface capable of connecting external or internal devices as required, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, etc.

[0162] The processor 501 is responsible for managing the bus architecture and general processing, and the memory 502 can store data used by the processor 501 when performing operations.

[0163] The present application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the embodiment of the beam prediction method based on a multimodal large model, and can achieve the same technical effect. To avoid repetition, the description is omitted here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0164] An embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned beam prediction method embodiment based on a multimodal large model are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0165] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0166] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0167] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A beam prediction method based on a multimodal large model, characterized in that: include: Using the first multimodal environment-beam dataset, a beam prediction model based on the multimodal large model is fine-tuned and trained to obtain a beam prediction model based on the multimodal large model after the fine-tuning training; wherein the first multimodal environment-beam dataset is obtained by matching each multimodal environment data in the first multimodal environment dataset with the beam index of the maximum received signal power at the corresponding position; The beam prediction model based on the multimodal large model after the first fine-tuning training is fine-tuned for the second time using the small sample training set to obtain the beam prediction model based on the multimodal large model after the second fine-tuning training; Obtaining a beam corresponding to the target terminal at the first moment based on a beam prediction model based on a multimodal large model after secondary fine-tuning training and the collected target multimodal environment data of the target terminal at the first moment; Among them, the beam prediction model based on the multimodal large model is used to obtain text embedding features according to the position information in the target multimodal environment data and image embedding features according to the multi-view images in the target multimodal environment data, and obtain multimodal features related to beam selection based on the text embedding features and the image embedding features, and map the multimodal features to the probability distribution of all beams in the predefined beam codebook, and select the beam corresponding to the beam index with the largest probability as the beam corresponding to the target terminal at the first moment.

2. The beam prediction method based on a multimodal large model according to claim 1, characterized in that: The method further comprises: Matching each multimodal environment data in the second multimodal environment data set with the beam index of the maximum received signal power at the corresponding position to obtain a second multimodal environment-beam data set; Different proportions of multimodal environment-beam data are selected from the second multimodal environment-beam data set to construct different small sample training sets.

3. The beam prediction method based on a multimodal large model according to claim 2, characterized in that: The method further comprises: Acquire the first multimodal environment dataset based on the location information and multi-view images of each terminal in the first wireless propagation scenario collected at intervals of a preset duration; and The second multimodal environment data set is obtained based on the location information and multi-perspective images of each terminal in the second wireless propagation scene collected at each preset time interval, and the first wireless propagation scene and the second wireless propagation scene are different wireless propagation scenes.

4. The beam prediction method based on a multimodal large model according to claim 1, characterized in that: The beam prediction model based on the multimodal large model includes: An input preprocessing module, configured to convert the location information in the target multimodal environment data into location text, and to stitch the multi-view images in the target multimodal environment data into a panoramic image; A text embedding module, used for processing the position text to obtain text embedding features; An image encoding module, configured to extract features from the panoramic image to obtain image embedding features; A large model module, configured to fuse the image embedding features and the text embedding features to obtain fused features, and extract multimodal features related to beam selection from the fused features; An output mapping module maps the multimodal feature to the probability distribution of all beams in the beam codebook, selects the beam index with the highest probability, and obtains the beam according to the beam index.

5. The beam prediction method based on a multimodal large model according to claim 3, characterized in that: Acquiring the first multimodal environment dataset according to the location information and multi-view images of each terminal in the first wireless propagation scenario collected at intervals of a preset duration includes: Constructing the first wireless propagation scenario, where the first wireless propagation scenario includes a scatterer and a moving terminal; At intervals of the preset time length, the location information of each terminal is collected by a sensor of each terminal, and the multi-view image collector of each terminal is collected by a multi-view image collector of each terminal; For each of the terminals, matching each piece of location information with the corresponding multi-view image to obtain multimodal environment data of the terminal; The first multimodal environment data set is acquired according to the multimodal environment data of each of the terminals in the first wireless propagation scenario.

6. The beam prediction method based on a multimodal large model according to claim 1, characterized in that: The method further comprises: Perform ray tracing simulation on the position of each terminal at intervals of a preset time length to generate multipath channel state information; Calculating a received signal power corresponding to each beam in the beam codebook using the multipath channel state information and a predefined beam codebook; Obtain a beam index of a maximum received signal power corresponding to the position of each terminal at intervals of a preset time length.

7. The beam prediction method based on a multimodal large model according to claim 1, characterized in that: Performing a fine-tuning training on the beam prediction model based on the multimodal large model using the first multimodal environment-beam dataset to obtain a beam prediction model based on the multimodal large model after the fine-tuning training, including: Selecting a training set from the first multimodal environment-beam dataset; The training set and the low-rank adaptive technology are used to fine-tune the weights of the beam prediction model based on the multimodal large model to obtain the beam prediction model based on the multimodal large model after one fine-tuning training.

8. The beam prediction method based on a multimodal large model according to claim 1, characterized in that: After fine-tuning the beam prediction model based on the multimodal large model using the first multimodal environment-beam dataset to obtain the beam prediction model based on the multimodal large model after fine-tuning, the method further includes at least one of the following: The accuracy rate is used to evaluate the consistency between the beam index obtained by the beam prediction model based on the multimodal large model after a fine-tuning training and the true index; Evaluate the complexity of a beam prediction model based on a multimodal large model after a single fine-tuning training based on at least one of the inference time, video memory usage, and computing resource consumption.

9. A beam prediction device based on a multimodal large model, characterized in that: include: A first training module is configured to perform a fine-tuning training on a beam prediction model based on a multimodal large model using a first multimodal environment-beam dataset to obtain a beam prediction model based on the multimodal large model after the fine-tuning training; wherein the first multimodal environment-beam dataset is obtained by matching each multimodal environment data in the first multimodal environment dataset with a beam index of a maximum received signal power at a corresponding position; The second training module is used to perform a second fine-tuning training on the beam prediction model based on the multimodal large model after the first fine-tuning training using the small sample training set, so as to obtain the beam prediction model based on the multimodal large model after the second fine-tuning training; A prediction module, which obtains the beam of the target terminal at the first moment based on the beam prediction model based on the multimodal large model after secondary fine-tuning training and the collected target multimodal environment data of the target terminal at the first moment; Among them, the beam prediction model based on the multimodal large model is used to convert the location information in the target multimodal environment data into text embedding features and convert the multi-view images in the target multimodal environment data into image embedding features, and obtain multimodal features related to beam selection based on the text embedding features and the image embedding features, and map the multimodal features to the probability distribution of all beams in the predefined beam codebook, and select the beam corresponding to the beam index with the largest probability as the beam of the target terminal at the first moment.

10. A beam prediction device based on a multimodal large model, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, the beam prediction method based on a multimodal large model as described in any one of claims 1 to 8 is implemented.

11. A readable storage medium, characterized in that: The readable storage medium stores a program, and when the program is executed by the processor, the beam prediction method based on the multimodal large model according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implements the beam prediction method based on a multimodal large model as claimed in any one of claims 1 to 8.

Images