Multi-modal feature fusion driven millimeter wave intelligent beam prediction method and system

Abstract

The application discloses a kind of multi-modal feature fusion driven millimeter wave intelligent beam prediction method and system, it is related to wireless communication technical field, including: the image data and position information data of target unmanned equipment are acquired;Feature vector of image data is extracted based on ViM encoder, position information data is nonlinearly mapped based on multilayer perception machine, and high-dimensional feature vector is obtained;Pre-training is carried out based on multi-modal momentum contrast learning framework, and pre-trained image stem network parameter and pre-trained position encoding network parameter are obtained, then migrate into supervised beam classification model, using preset quantity of labeled data is fine-tuned, and the beam prediction model trained is obtained;The beam of target unmanned equipment is predicted based on the beam prediction model trained.The application alleviates the problems that the existing technology has large calculation and time overhead, which leads to significant increase in communication initialization delay, and that it is difficult to obtain labels in existing communication scenarios.

Description

Technical Field

[0001] This invention relates to the field of wireless communication technology, and in particular to a millimeter-wave intelligent beam prediction method and system driven by multi-modal feature fusion. Background Technology

[0002] With the rapid development of fifth-generation (5G) and sixth-generation (6G) mobile communication technologies, millimeter-wave communication has become one of the core technologies for future wireless communication systems due to its extremely wide available spectrum resources and high data transmission rates. Millimeter-wave signals have high frequencies and short wavelengths, enabling them to support higher data rates compared to the Sub-6G band. However, they also exhibit significant characteristics such as high path loss, strong directivity, and sensitivity to obstructions during propagation. These characteristics necessitate extremely high beam alignment accuracy in establishing and maintaining stable links in millimeter-wave communication systems. Therefore, efficient beam prediction technology has become a key research direction in millimeter-wave communication.

[0003] Existing millimeter-wave beam prediction and selection primarily rely on beam training or beam scanning techniques. Traditional beam training methods typically require the transmitter and receiver to perform a one-to-one matching within a predefined set of beam directions, searching for the direction with the maximum channel gain by traversing different beam combinations. While this exhaustive search strategy can achieve relatively ideal beam selection results in static environments, it incurs enormous computational and time overhead. In multi-user or high-speed mobile scenarios, communication initialization delays increase significantly, and system resources such as pilot overhead and time overhead are severely consumed, making it difficult to meet real-time requirements. Furthermore, tag acquisition is challenging in current communication scenarios. In addition, existing beam prediction schemes often employ one or two modal sensing data, lacking in multimodal feature extraction and multimodal data fusion, and their semantic understanding of the wireless transmission environment is still insufficient. Summary of the Invention

[0004] To address the aforementioned technical problems in existing technologies, this invention provides a millimeter-wave intelligent beam prediction method and system driven by multimodal feature fusion. The technical solution is as follows: On one hand, a millimeter-wave intelligent beam prediction method driven by multimodal feature fusion is provided and applied to a base station. The method includes: acquiring image data and location information data of a target UAV device; the location information data includes location coordinates, altitude, and distance information; the distance information is the distance between the base station and the target UAV device; extracting feature vectors from the image data based on a ViM encoder, and performing nonlinear mapping on the location information data based on a multilayer perceptron to obtain high-dimensional feature vectors; pre-training using a multimodal momentum contrastive learning framework, using a combination of feature vectors and high-dimensional feature vectors from image data at the same time as positive sample pairs, and feature vectors or high-dimensional feature vectors from image data at different times as negative samples, to obtain pre-trained image backbone network parameters and pre-trained location encoding network parameters; transferring the pre-trained image backbone network parameters and the pre-trained location encoding network parameters to a supervised beam classification model, fine-tuning them using a preset number of labeled data, to obtain a trained beam prediction model; and predicting the beam of the target UAV device based on the trained beam prediction model.

[0005] Optionally, acquiring image data and location information data of the target drone device includes: acquiring image data of the target drone device based on an RGB camera; acquiring the location coordinate information of the target drone device based on a GPS module; and acquiring the altitude and distance information of the target drone device based on an inertial measurement unit.

[0006] Optionally, the loss function of the multimodal momentum contrastive learning framework includes a symmetric cross-modal InfoNCE loss function; the symmetric cross-modal InfoNCE loss function includes:

[0007]

[0008]

[0009] In the formula, Let be the symmetric cross-modal InfoNCE loss function. Let the image be the loss function for the position orientation. Let the position-to-image orientation loss function be... It queries image features. These are key position features from the same sample. It is the key position feature of the i-th sample. It is the temperature coefficient. It queries image features. These are key position features from the same sample. It is the key position feature of the i-th sample.

[0010] Optionally, predicting the beam of the target UAV device based on the trained beam prediction model includes: acquiring current image data and current location information data of the target UAV device; extracting feature vectors from the current image data based on a ViM encoder; performing nonlinear mapping on the current location information data based on a multilayer perceptron to obtain a high-dimensional feature vector of the target; fusing the feature vectors from the current image data and the high-dimensional feature vector of the target based on the Concat method to obtain a joint feature vector; using the joint feature vector as input, outputting the probability distribution of each candidate beam becoming the optimal beam based on the prediction head, and selecting the candidate beam with the highest probability as the optimal beam.

[0011] On the other hand, a multimodal feature fusion-driven millimeter-wave intelligent beam prediction system is also provided to implement the multimodal feature fusion-driven millimeter-wave intelligent beam prediction method provided in this embodiment of the invention, applied to a base station; the system includes: an acquisition module, an extraction module, a pre-training module, a fine-tuning module, and a prediction module; wherein, the acquisition module is used to acquire image data and location information data of a target UAV device; the location information data includes location coordinate information, altitude information, and distance information; the distance information is the distance between the base station and the target UAV device; the extraction module is used to extract feature vectors from the image data based on a ViM encoder, and perform nonlinear mapping on the location information data based on a multilayer perceptron. The high-dimensional feature vector is obtained. The pre-training module is used to pre-train based on the multimodal momentum contrastive learning framework, using the combination of feature vectors and high-dimensional feature vectors of image data at the same time as positive sample pairs and feature vectors or high-dimensional feature vectors of image data at different times as negative samples, to obtain pre-trained image backbone network parameters and pre-trained position encoding network parameters. The fine-tuning module is used to transfer the pre-trained image backbone network parameters and the pre-trained position encoding network parameters to the supervised beam classification model, and to fine-tune them using a preset number of labeled data to obtain a trained beam prediction model. The prediction module is used to predict the beam of the target UAV based on the trained beam prediction model.

[0012] Optionally, the prediction module is further configured to: acquire current image data and current location information data of the target UAV device; extract feature vectors from the current image data based on a ViM encoder; perform nonlinear mapping on the current location information data based on a multilayer perceptron to obtain a high-dimensional feature vector of the target; fuse the feature vectors of the current image data and the high-dimensional feature vector of the target based on the Concat method to obtain a joint feature vector; use the joint feature vector as input, output the probability distribution of each candidate beam becoming the optimal beam based on the prediction head, and select the candidate beam with the highest probability as the optimal beam.

[0013] On the other hand, an electronic device is also provided, including: a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the processor executes the computer program to implement the method provided in the embodiments of the present invention.

[0014] On the other hand, a computer-readable storage medium is also provided, wherein program code is stored in the computer-readable storage medium, and the program code can be called by a processor to execute the method provided in the embodiments of the present invention.

[0015] This invention provides a millimeter-wave intelligent beam prediction method and system driven by multimodal feature fusion. By autonomously learning features and patterns in the environment, it avoids the problem of relying on a large amount of labeled data. This invention can effectively reduce the complexity of the training and inference process and improve computational efficiency, thereby significantly improving the generalization ability of the millimeter-wave beam prediction system without sacrificing prediction accuracy. It also alleviates the technical problem of large computational and time overhead leading to a significant increase in communication initialization delay in the prior art. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a millimeter-wave intelligent beam prediction method driven by multimodal feature fusion provided in an embodiment of the present invention; Figure 2 This is a flowchart of a multimodal fusion feature-driven millimeter-wave intelligent beam prediction method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of a communication model between a base station and a target drone device provided in an embodiment of the present invention; Figure 4This is a comparison curve of the loss between the ViM method provided in this embodiment of the invention and the traditional ViT method during the training and validation phases; Figure 5 This is a schematic diagram comparing the accuracy of the ViM method provided in this embodiment of the invention with that of the traditional ViT method; Figure 6 This is a performance comparison diagram between the ViM method provided in this embodiment of the invention and the traditional ViT method; Figure 7 This is a comparison curve of the loss of the MoCo multimodal method and the method without MoCo under different training ratios provided in the embodiments of the present invention. Figure 8 This is a curve comparing the accuracy of the MoCo multimodal method used in the embodiments of the present invention with that of a method without MoCo at different training ratios. Figure 9 This is a schematic diagram comparing the performance of the MoCo multimodal method and the method without MoCo under different training ratios provided in the embodiments of the present invention. Figure 10 This is a comparison curve of the loss between the MoCo loading method and the local pre-trained weight loading method provided in the embodiments of the present invention; Figure 11 This is a curve comparing the accuracy of the MoCo loading method and the local pre-trained weight loading method provided in this embodiment of the invention. Figure 12 This is a performance comparison diagram of the MoCo loading method and the local pre-trained weight loading method provided in the embodiments of the present invention; Figure 13 This is a schematic diagram of a millimeter-wave intelligent beam prediction system driven by multimodal feature fusion provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0019] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0020] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0021] Figure 1 This is a flowchart of a millimeter-wave intelligent beam prediction method driven by multimodal feature fusion according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating a multi-modal fusion feature-driven millimeter-wave smart beam prediction method according to an embodiment of the present invention, which is applied to a base station. Figure 1 and Figure 2 As shown, the method specifically includes the following steps: Step S102: Acquire image data and location information data of the target drone device; the location information data includes location coordinates, altitude, and distance.

[0022] Specifically, image data of the target drone device is acquired based on an RGB camera; The GPS module is used to obtain the location coordinates of the target UAV, and the inertial measurement unit (IMU) is used to obtain the altitude and distance information of the target UAV.

[0023] Step S104: Extract feature vectors from image data based on ViM (Vision Mamba) encoder, and perform nonlinear mapping on location information data based on Multi-Layer Perceptron (MLP) to obtain high-dimensional feature vectors.

[0024] Step S106: Based on the multimodal momentum contrast (MoCo) learning framework, the combination of feature vectors and high-dimensional feature vectors of image data at the same time moment is used as positive sample pairs, and the feature vectors or high-dimensional feature vectors of image data at different times moment are used as negative samples for pre-training, so as to obtain the pre-trained image backbone network parameters and the pre-trained position encoding network parameters.

[0025] Step S108: Transfer the pre-trained image backbone network parameters and pre-trained location encoding network parameters to the supervised beam classification model, and fine-tune them using a preset number of labeled data to obtain the trained beam prediction model.

[0026] Step S110: Predict the beam of the target UAV based on the trained beam prediction model. Specifically, as follows... Figure 2 As shown, it also includes the following steps: Step S1101: Obtain the current image data and current location information data of the target drone device; Step S1102: Extract the feature vector of the current image data based on the ViM encoder; Step S1103: Based on MLP, perform nonlinear mapping on the current location information data to obtain the target high-dimensional feature vector; Step S1104: Based on the Concat method, the feature vector of the current image data and the target high-dimensional feature vector are fused to obtain a joint feature vector; Step S1105: Using the joint feature vector as input, output the probability distribution of each candidate beam becoming the optimal beam based on the prediction head, and select the candidate beam with the highest probability as the optimal beam.

[0027] Figure 3 This is a schematic diagram of a communication model between a base station and a target unmanned aerial vehicle (UAV) according to an embodiment of the present invention. Figure 3 As shown, the millimeter-wave communication system of this invention comprises two main units: a base station (BS) and an unmanned aerial vehicle (UAV). The base station is deployed on roads or roadside facilities and is equipped with (1) an N antenna array, (2) a 60 GHz millimeter-wave receiver, and (3) an RGB camera external sensing sensor. The BS is used to receive uplink pilot signals, collect multimodal environmental sensing information, and perform beam selection and downlink transmission control to provide millimeter-wave communication services for the UAV; the UAV carries a 60 GHz millimeter-wave transmitter, a GPS module, and an IMU. The antenna array of the base station and the sensor of the UAV work together to align visual and positioning information in time to construct multimodal sensing features and provide the communication system with sensing-driven beam prediction capabilities.

[0028] Assuming in discrete time slots Within this network, the BS–UAV link can be approximated as a narrowband flat fading channel, and its downlink equivalent channel vector is denoted as:

[0029] Where N is the number of antennas in BS, Represents the complex field. The system predefines a set of beamforming codebooks: , , , in For the number of candidate beams, For the first A beamforming vector is used, and normalization constraints are satisfied to ensure energy consistency between different beams. If beamforming is used for downlink... The received signal is:

[0030] in These are the symbols to be transmitted. Variables This indicates the noise at the receiving end. This represents noise power.

[0031] Millimeter-wave links are highly sensitive to non-line-of-sight occlusion, UAV movement, and array pointing mismatch; traditional exhaustive and hierarchical training requires a large number of pilot and link probes, resulting in high overhead and latency. To reduce overhead, the system utilizes external multimodal sensing image data (e.g., RGB images), location information data (e.g., GPS information), and IMU data to obtain environmental perception, and directly predicts the optimal beam through beamforming prediction using a deep learning model.

[0032] In a given time slot Perceived information and context, from Select the optimal beam To maximize instantaneous received power:

[0033] in This represents the index of the optimal beam. Due to real-time acquisition... The cost is high, so this invention does not directly estimate it. Instead, it learns the mapping from multimodal sensing to beam index, without needing to estimate the channel state in real time.

[0034] Specifically, the system first constructs feature vectors using RGB images and GPS four-dimensional multi-source sensing information:

[0035] in: : Represents the RGB image features (i.e., the feature vector of the image data) extracted by the visual encoder. : Represents the location features obtained based on a high-precision positioning system and the distance and altitude data (i.e., high-dimensional feature vectors) based on IMU data.

[0036] Deep learning models With feature vectors As input, the output length is Beam selection probability distribution:

[0037] in, Indicates the first The probability that each beam is the optimal beam is determined by the maximum probability term in the final predicted beam index:

[0038] The corresponding beamforming vector is This is used for downlink transmission in this time slot.

[0039] To realize the millimeter-wave intelligent beam prediction method driven by multimodal feature fusion, this invention proposes a deep learning prediction framework that combines RGB images and GPS information. This method achieves intelligent selection and dynamic optimization of millimeter-wave beams through three main stages: multimodal feature extraction, cross-modal feature fusion, and beam prediction.

[0040] Specifically, the feature extraction steps for each data source are as follows: 1) RGB Image Feature Extraction: The base station acquires environmental image data through an RGB camera and uses a ViM encoder to extract feature vectors from the image data. Specifically, after the image data is input, it is first segmented into several image patches, and each image patch is mapped to a low-dimensional embedding vector. Then, these embedding vectors are recursively processed by the SSM to extract visual features related to signal propagation. The final feature vector of the image data is:

[0041] in, I For the original image data, These are the learning parameters for the ViM encoder.

[0042] Location information encoding: The target UAV obtains its latitude and longitude information via GPS, and its distance and altitude information via IMU unit. These are then concatenated with the GPS information to represent its location. The location information is transformed into a high-dimensional feature vector through a non-linear mapping process using an MLP.

[0043] in, P This is the original location information data. These are the parameters for the MLP.

[0044] 2) Feature fusion: This invention employs... The method integrates image, spatial structure, and location information for a comprehensive representation:

[0045] in, These are the learnable parameters for the fusion module.

[0046] 3) Beam prediction: Joint eigenvectors The signal is fed into the prediction head, which consists of several feedforward neural network layers and outputs the probability distribution of each candidate beam becoming the optimal beam.

[0047] in, Indicates the first The probability of a beam being the optimal beam.

[0048] 4) Beam selection: Ultimately, the beam with the highest probability is selected as the optimal beam.

[0049] And select the corresponding beamforming vector Used for downlink transmission.

[0050] 5) Training and Optimization: To address the scarcity of labeled data, this invention designs a self-supervised learning pre-training framework based on MoCo. This framework utilizes synchronously acquired RGB image data, location coordinates, and height and distance information from real datasets to perform comparative learning between image modalities and location modalities without beam labels, thereby learning a general feature representation with scene discrimination capabilities. After pre-training, the pre-trained image backbone network parameters and location encoding network parameters are transferred to a supervised beam classification model, and fine-tuned using a small amount of labeled data to improve the model's performance in beam selection tasks. The MoCo framework uses the InfoNCE loss function for optimization, maximizing the consistency of positive sample pairs within the same scene and minimizing the differences between features from different scenes.

[0051] Specific steps for using MoCo: 1) First, read the RGB image from the input data. The location information, along with distance and altitude information, are concatenated to form a four-dimensional location vector. The data is then standardized based on training set statistics. For image modalities, the same RGB image undergoes two independent data augmentations to obtain two image views, View1 and View2. The image views are input to a ViM encoder to extract image features; the position vectors are input to an MLP position encoder to extract position features. Subsequently, the image features and position features are mapped to a uniform-dimensional contrastive learning embedding space via projection heads.

[0052] In this invention, image features and location features from the same sample are considered positive sample pairs because they describe multimodal observation information at the same time and in the same scene; image features or location features from other samples are considered negative samples to enhance the model's ability to distinguish different scene representations.

[0053] 2) Momentum Negative Sample Dictionary: To improve the stability of contrastive learning and the efficiency of negative sample utilization, this invention employs MoCo's momentum encoding mechanism. The model consists of two networks: a query encoder and a key encoder. The query encoder is updated through backpropagation, while the key encoder does not directly participate in gradient updates but is gradually updated by the query encoder parameters using momentum.

[0054] in, This indicates a query for encoder parameters. Indicates key encoder parameters, This represents the momentum coefficient, typically 0.999.

[0055] Specifically, MoCo maintains a momentum contrastive dictionary containing negative samples from historical models. Each sample is mapped into this dictionary and used to calculate the contrastive loss against positive samples. This dictionary is updated using momentum, allowing it to include more negative samples and improving the stability of contrastive learning.

[0056] 3) Cross-modal contrast loss calculation: In this embodiment of the invention, the loss function of the multimodal MoCo learning framework includes a symmetric cross-modal InfoNCE loss function; the symmetric cross-modal InfoNCE loss function includes two parts: an image-to-location method loss function and a location-to-image orientation loss function.

[0057] Among them, the image-to-location orientation loss function is used to query image features. As input, key position features of the same sample These form positive sample pairs, which, together with other positional features in the positional negative sample queue, form negative sample pairs. This can be represented as:

[0058] in: It queries image features. These are key position features from the same sample. It is the key position feature of the i-th sample. It is a temperature coefficient used to control the discrimination between positive and negative sample pairs.

[0059] For the location-to-image orientation loss function, to query location features As input, key image features of the same sample These form positive sample pairs, which, together with other image features in the negative sample queue, form negative sample pairs. This can be represented as:

[0060] in: It queries image features. These are key position features from the same sample. It is the key position feature of the i-th sample.

[0061] The goal of the InfoNCE loss function is to maximize the consistency between positive sample pairs (i.e., maximize their dot product) and minimize the similarity between negative samples. The final MoCo pre-training loss is the average of the losses in both directions:

[0062] 4) Pre-training parameter transfer and supervised fine-tuning: After completing the self-supervised pre-training of MoCo, the ViM image backbone network parameters, MLP position encoding network parameters, and position normalization statistics are saved. In the subsequent beam selection task, a supervised classification model is constructed: RGB images are processed by ViM to extract image features, and the normalized position vectors are processed by MLP to extract position features. The two features are concatenated and then input into a fully connected classification head, outputting classification results for 32 candidate beams. In the validation phase, the Top-1, Top-2, and Top-3 accuracies are calculated based on the 32 beam logits output by the model to evaluate the model's performance in the beam selection task. Specific results are shown in the figure below. Figure 4 , Figure 5 , Figure 6 .

[0063] Specifically, Figure 4 This is a comparison curve of the loss between the ViM method provided in this embodiment and the traditional ViT (VisionTransformer) method during the training and validation phases. Figure 5 This is a schematic diagram comparing the accuracy of the ViM method provided by an embodiment of the present invention with that of the traditional ViT method. Figure 6 This is a performance comparison diagram between the ViM method provided according to an embodiment of the present invention and the traditional ViT method. During training, the number of training parameters for ViM is 7,185,856, while the number of training parameters for ViT is 86,439,328. Figures 4-6 It can be seen that ViM reduces computational complexity while achieving slightly better accuracy than the ViT method, demonstrating the feasibility of applying the ViM method proposed in this invention to the field of communication.

[0064] Specifically, Figure 7 This is a comparison curve of the loss of the MoCo multimodal method and the method without MoCo under different training ratios, according to embodiments of the present invention. Figure 8 This is a curve comparing the accuracy of the MoCo multimodal method and the method without MoCo at different training ratios, according to embodiments of the present invention. Figure 9This is a schematic diagram comparing the performance of the MoCo multimodal method and the method without MoCo under different training ratios, according to embodiments of the present invention. Figures 7-9 As shown, the MoCo multimodal method used in this invention differs significantly in accuracy from methods that do not use MoCo, and the difference increases with the smaller the proportion of training samples.

[0065] Figure 10 This is a comparison curve of the loss of the MoCo loading method and the local pre-trained weight loading method provided by the embodiments of the present invention. Figure 11 This is a comparison curve of the accuracy of the MoCo loading method and the local pre-trained weight loading method provided by the embodiments of the present invention. Figure 12 This is a performance comparison diagram of the MoCo loading method and the local pre-trained weight loading method provided by embodiments of the present invention. Figures 10-12 As shown, the beam prediction accuracy after loading multimodal MoCo pre-trained weights is close to that after loading local pre-trained weights. However, the local pre-trained weights are selected from the ImageNet pre-trained weights provided during Mamba training, which use a large number of labels during pre-training. The multimodal MoCo method used in this invention does not use labels during pre-training, which is unsupervised training, reducing the dependence on labels and verifying the feasibility and effectiveness of the method proposed in this invention.

[0066] The complexity of the beam prediction model provided in this embodiment of the invention and the complexity of the traditional multi-head attention mechanism are calculated as follows: First, in the complex computation of Multi-Head Attention for processing multimodal data, to analyze the computational complexity of this mechanism, we assume the input data matrix... The shape is ,in: Batch size : The length of the sequence (i.e. how many tokens an image is divided into). : Feature dimensions of each token.

[0067] For each layer of computation, the query, key, and value matrices are first calculated, followed by dot product, normalization, and weighted averaging. The computational complexity analysis for each step is as follows: Step 1: Calculation Q , K , V : Input matrix The query is obtained through linear transformation ( ),key( ) and value ( ):

[0068] in, It is the learned weight matrix, with the shape of... The computational complexity of each linear transformation is:

[0069] Step 2: Calculation : Calculate the dot product of query and key. :

[0070] The computational complexity of this step is:

[0071] Step 3: Calculate Softmax and weighted summation: right Perform Softmax normalization and compare it with the value matrix. Perform a weighted summation:

[0072] The computational complexity is:

[0073] Step 4: Calculate the output matrix: Finally, the calculated attention matrix and output weight matrix are linearly transformed:

[0074] The computational complexity of this step is:

[0075] Combining the above steps, the computational complexity of the entire attention mechanism is:

[0076] generally Larger, usually It will dominate. The final computational complexity is:

[0077] Then, the ViM method introduced in this invention employs SSM. To analyze the computational complexity of this mechanism, we assume the input data matrix... The shape is After linear transformation, the shape of x is , The shape is The computational complexity is as follows, where: : The length of the sequence (i.e. how many tokens an image is divided into). : Feature dimensions of each token.

[0078] Assume the state of each layer It can be represented by the following parameterized form:

[0079]

[0080] in: It is the state transition matrix; M ,2 D , N ) It is a control input matrix; M ,2 D , N ) It is a control output matrix; M ,2 D , N ) The computational complexity is

[0081] The computational complexity is

[0082] The computational complexity is

[0083] Combining the above steps, the total complexity is 3.

[0084] Therefore, this invention introduces ViM into beam prediction technology in the field of wireless communication. The SSM in ViM reduces the computational complexity of each layer from the traditional Reduce to Therefore, the overall computational complexity is from Reduced to approximately linear This significantly reduces computational overhead and time consumption (proven below). This optimization maintains high accuracy and low latency even in the face of dynamic environmental changes and large-scale data (such as images and location information), greatly improving the predictive capabilities of communication systems.

[0085] At the same time, the MoCo framework is extended to multimodal. Through self-supervised learning, the MoCo framework enables the model to learn feature representations with high generalization ability from unlabeled multimodal data (such as camera images and location information), reducing the dependence on manually labeled data.

[0086] As described above, the multimodal feature fusion-driven millimeter-wave intelligent beam prediction method provided in this embodiment of the invention selects beams using multimodal sensing data. Based on ViM data processing, it significantly reduces communication overhead and improves real-time performance, while also reducing the computational complexity of the model. Through self-supervised learning using the MoCo framework, the model can learn robust scene representations from a large amount of unlabeled data, reducing reliance on manually labeled data. By fusing deep learning with multimodal sensing, the system can automatically optimize beam selection, improve beam training accuracy and communication quality, and adapt to complex environments.

[0087] Figure 13 This is a schematic diagram of a millimeter-wave intelligent beam prediction system driven by multimodal feature fusion according to an embodiment of the present invention, applied to a base station; such as Figure 13 As shown, the system includes: an acquisition module 10, an extraction module 20, a pre-training module 30, a fine-tuning module 40, and a prediction module 50.

[0088] Specifically, the acquisition module 10 is used to acquire image data and location information data of the target drone device; the location information data includes location coordinates, altitude information and distance information; Extraction module 20 is used to extract feature vectors from image data based on ViM encoder and to perform nonlinear mapping on position information data based on MLP to obtain high-dimensional feature vectors. The pre-training module 30 is used to pre-train based on the multimodal MoCo learning framework, using the combination of feature vectors and high-dimensional feature vectors of image data at the same time as positive sample pairs and feature vectors or high-dimensional feature vectors of image data at different times as negative samples, to obtain the pre-trained image backbone network parameters and the pre-trained location encoding network parameters. The fine-tuning module 40 is used to transfer the pre-trained image backbone network parameters and the pre-trained position encoding network parameters to the supervised beam classification model, and to fine-tune the model using a preset number of labeled data to obtain the trained beam prediction model. The prediction module 50 is used to predict the beam of the target UAV equipment based on the trained beam prediction model.

[0089] Specifically, the prediction module 50 is also used for: Acquire the current image data and current location information of the target drone device; Extracting feature vectors from the current image data based on the ViM encoder; Based on MLP, nonlinear mapping is performed on the current location information data to obtain the target high-dimensional feature vector; The Concat method is used to fuse the feature vector of the current image data and the target high-dimensional feature vector to obtain a joint feature vector; Using the joint feature vector as input, the probability distribution of each candidate beam becoming the optimal beam is output based on the prediction head, and the candidate beam with the highest probability is selected as the optimal beam.

[0090] The present invention also provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the processor executes the computer program to implement the method provided in the embodiments of the present invention.

[0091] The present invention also provides a computer-readable storage medium storing program code, which can be called by a processor to execute the method provided in the embodiments of the present invention.

[0092] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0093] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0094] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0095] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0097] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0098] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0099] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A millimeter-wave intelligent beam prediction method driven by multimodal feature fusion, characterized in that, Applied to a base station; the method includes: The system acquires image data and location information data of the target drone device; the location information data includes location coordinates, altitude, and distance; the distance information is the distance between the base station and the target drone device. The image data feature vector is extracted based on the ViM encoder, and the location information data is nonlinearly mapped based on the multilayer perceptron to obtain a high-dimensional feature vector. Based on the multimodal momentum contrastive learning framework, the combination of feature vectors and high-dimensional feature vectors of image data at the same time is used as positive sample pairs, and the feature vectors or high-dimensional feature vectors of image data at different times are used as negative samples for pre-training, so as to obtain the pre-trained image backbone network parameters and the pre-trained position encoding network parameters. The pre-trained image backbone network parameters and the pre-trained position encoding network parameters are transferred to the supervised beam classification model, and fine-tuned using a preset number of labeled data to obtain the trained beam prediction model. The beam of the target UAV is predicted based on the trained beam prediction model.

2. The method according to claim 1, characterized in that, Acquire image data and location information of the target drone equipment, including: Image data of the target drone device is acquired using an RGB camera; The location coordinates of the target drone are obtained using a GPS module, and the altitude and distance information of the target drone are obtained using an inertial measurement unit.

3. The method according to claim 1, characterized in that, The loss function of the multimodal momentum contrastive learning framework includes a symmetric cross-modal InfoNCE loss function; the symmetric cross-modal InfoNCE loss function includes: In the formula, Let be the symmetric cross-modal InfoNCE loss function. Let the image be the loss function for the position orientation. Let the position-to-image orientation loss function be... It queries image features. These are key position features from the same sample. It is the key position feature of the i-th sample. It is the temperature coefficient. It queries image features. These are key position features from the same sample. It is the key position feature of the i-th sample.

4. The method according to claim 1, characterized in that, Based on the trained beam prediction model, the beam of the target UAV device is predicted, including: Acquire the current image data and current location information data of the target drone device; The feature vector of the current image data is extracted based on the ViM encoder; A high-dimensional feature vector of the target is obtained by performing a nonlinear mapping on the current location information data based on a multilayer perceptron. The feature vector of the current image data and the target high-dimensional feature vector are fused using the Concat method to obtain a joint feature vector; Using the joint feature vector as input, the probability distribution of each candidate beam becoming the optimal beam is output based on the prediction head, and the candidate beam with the highest probability is selected as the optimal beam.

5. A millimeter-wave intelligent beam prediction system driven by multimodal feature fusion, characterized in that, This system is used to implement a multimodal feature fusion-driven millimeter-wave intelligent beam prediction method as described in any one of claims 1-4, and is applied to a base station; the system includes: an acquisition module, an extraction module, a pre-training module, a fine-tuning module, and a prediction module; wherein... The acquisition module is used to acquire image data and location information data of the target drone device; the location information data includes location coordinates, altitude, and distance; the distance information is the distance between the base station and the target drone device. The extraction module is used to extract the feature vector of the image data based on the ViM encoder, and to perform nonlinear mapping on the location information data based on the multilayer perceptron to obtain a high-dimensional feature vector. The pre-training module is used to pre-train based on a multimodal momentum contrastive learning framework, using a combination of feature vectors and high-dimensional feature vectors of image data at the same time as positive sample pairs and feature vectors or high-dimensional feature vectors of image data at different times as negative samples, to obtain pre-trained image backbone network parameters and pre-trained location encoding network parameters. The fine-tuning module is used to transfer the pre-trained image backbone network parameters and the pre-trained position encoding network parameters to the supervised beam classification model, and to fine-tune the model using a preset number of labeled data to obtain a trained beam prediction model. The prediction module is used to predict the beam of the target UAV device based on the trained beam prediction model.

6. The system according to claim 5, characterized in that, The prediction module is also used for: Acquire the current image data and current location information data of the target drone device; The feature vector of the current image data is extracted based on the ViM encoder; A high-dimensional feature vector of the target is obtained by performing a nonlinear mapping on the current location information data based on a multilayer perceptron. The feature vector of the current image data and the target high-dimensional feature vector are fused using the Concat method to obtain a joint feature vector; Using the joint feature vector as input, the probability distribution of each candidate beam becoming the optimal beam is output based on the prediction head, and the candidate beam with the highest probability is selected as the optimal beam.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor, when executing the computer program, implements the method as described in any one of claims 1-4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1-4.

Images