Millimeter wave beam tracking method based on contrast learning

Abstract

The invention relates to a millimeter wave beam tracking method based on comparative learning, and belongs to the field of wireless communication. The method is oriented to a millimeter wave communication system, combines vision and laser radar information in an environment collected by multiple sensors installed in a millimeter wave base station, utilizes a machine learning model to carry out feature processing on data of different modes, and adopts a multi-mode fusion strategy based on comparative learning to enhance a synergistic effect among the data of each mode, so as to achieve a multi-mode fusion effect. A method of predicting a current beam in real time and tracking a plurality of future beams is obtained. According to the invention, multi-modal data fusion and comparative learning are combined for beam tracking, the precision and stability of beam tracking can be effectively guaranteed, and the communication efficiency of a millimeter wave communication system is ensured.

Description

Technical Field

[0001] The invention belongs to the field of wireless communications and relates to a millimeter wave beam tracking method based on contrast learning. Background Art

[0002] With the rapid development of artificial intelligence and communication technology, the next generation of mobile communication networks has begun to use millimeter waves and even higher frequency bands to meet the growing demand for mobile services. Compared with the low-frequency signals used in traditional communication systems, the propagation loss of millimeter wave signals is more serious. Due to the short wavelength of millimeter waves, a large number of antenna arrays and narrow directional beams are required to ensure a sufficiently high received signal strength. However, applying the traditional beam acquisition method based on a large number of antenna array vector training to the millimeter wave band will generate huge beam acquisition overhead. Therefore, it is necessary to find new ways to reduce this overhead in order to reduce the communication delay and consumption of millimeter wave systems.

[0003] Machine learning-based methods bring new ideas to solve the beam management and channel assessment overhead in high-frequency communication systems. Information obtained by sensor devices in the environment, such as user location, images of user location in the environment taken by base station cameras, lidar point clouds, and radar can be used to predict and track the beam of the millimeter-wave communication system. However, a single modality can only extract information from a single data source, which may lead to loss or insufficiency of information, especially in complex scenes, where single-modality data cannot fully capture scene features. Therefore, how to invent a millimeter-wave beam tracking method suitable for multimodal data fusion, using the complementarity of data from different modalities to make up for the shortcomings of single-modality data and provide more accurate and stable prediction and tracking has become an important challenge. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a millimeter-wave beam tracking method based on contrastive learning, which utilizes the complementarity of data from different modalities to make up for the shortcomings of single-modality data and provide more accurate and stable prediction and tracking.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A millimeter wave beam tracking method based on contrastive learning constructs a millimeter wave communication beam tracking multimodal data fusion neural network model, trains the neural network using the contrastive learning method, optimizes the synergy between different modes, uses the fused features for beam tracking, predicts the current optimal beam in real time, and simultaneously predicts the optimal indexes of multiple optimal beams in the future.

[0007] The method specifically comprises the following steps:

[0008] S1: Obtain the model parameters of the multi-sensor millimeter wave communication system, build a multi-modal sensing-assisted millimeter wave communication beam tracking problem model, and transform the beam tracking problem into an optimization problem based on machine learning;

[0009] S2: Construct a millimeter wave communication beam tracking multimodal data fusion neural network model, including: feature extraction module, recurrent neural network module and fusion classification module; the model obtains perception data of different modes from multiple sensors equipped in the system, uses the feature extraction module to extract features from the acquired multimodal data, and models the multimodal time series features through the recurrent neural network module. The fusion classification module uses a multi-head attention mechanism to fuse the extracted features;

[0010] S3: Use contrastive learning mechanism to train multimodal data fusion neural network model, optimize the synergy between different modalities, use the fused features for beam tracking, and predict the best indexes of current and future multiple optimal beams in real time.

[0011] Further, step S1 specifically includes the following steps:

[0012] S11: Consider a millimeter wave communication system model around a single mobile user device, where a base station provides services for the mobile user device; the base station contains N antennas, a lidar sensor, and an RGB camera (visual data sensor) to provide awareness of the surrounding environment; assuming that the base station has only one antenna, the base station uses a predefined beamforming codebook Where Q is the total number of beamforming vectors in the codebook, is the beamforming vector in the codebook;

[0013] S12: At time step t, if each mmWave user u is beamformed by the base station using the beamforming vector f q [t] Service, channel The signal from the base station to the user is expressed as:

[0014]

[0015] in, P T is the average transmit power, To transmit complex symbols, (·) H represents the conjugate transpose, n[t] represents a noise sample extracted from a complex Gaussian distribution;

[0016] S13: Multimodal beam tracking task using lidar and visual information in the single-user case. Based on the available perception data up to a given time t-1, the base station aims to predict the next The best beam for the upcoming step, in particular, the best beam from time t to time (t+α-1); the goal of this task is to obtain the optimal beam from the codebook Select the best beam f from the accessible candidate beams ★ [t] to maximize the received signal power; this problem can be expressed mathematically as:

[0017]

[0018] S14: From a predetermined codebook The best beam index is selected from the codebook, which is usually associated with a large acquisition overhead; the signal is concentrated in a certain direction in space, which is the mechanism for guiding a narrow beam; the spatial dimension of the scene is divided into many possibly overlapping sectors by the beam vector, and a unique value is assigned to each sector; therefore, the machine learning task can be converted into a classification task using a predetermined codebook, where the user's real-time position in the visual and lidar scene determines which beam index in the codebook to assign; under the specified codebook constraints, the optimal beam f can be identified solely by the matching codebook index ＊ [t]; At time t, the optimal beam index can be expressed as:

[0019]

[0020] Among them, |Q| represents The optimal beam index is recorded as g[t]. It should be noted that finding the optimal beam is the same as finding the optimal beam index under codebook constraints. Therefore, the beam tracking problem can be expressed as:

[0021]

[0022] in, represents the conditional probability, represents the predicted optimal beam index at time step t, o t-α is the auxiliary information collected before time step (t-α+1), which contains some detailed information of the best beam at time t;

[0023] S15: The RGB image (visual perception information) obtained at time step t is defined as Where H, W and C represent the height, width and number of color channels of the image respectively; definition is the corresponding user information obtained by the laser radar sensor at time step t, where D represents the angle in the laser radar perception field, and each angle has a corresponding distance value; the present invention aims to use the input visual information x u and lidar information u , track the optimal beam index g ★To serve user u; To achieve this, the present invention proposes to use machine learning methods to learn a function f Θ (S), where S = (x u ,l u ); the observed visual and lidar information is fed into the function, and its output probability In the codebook The index of the most likely element in In form it can be expressed as:

[0024]

[0025] in, represents the optimal beam index, Indicates the beam index value;

[0026] S16: In order to use these visual and lidar data for beam tracking, the goal of the present invention is to develop a function that can predict the optimal future beam starting from time step t based on the perception information collected up to time step t-1; let represents the visual and lidar data sequence, where i represents the time step in the observation window; then the visual and lidar assisted beam tracking optimization problem can be expressed as:

[0027]

[0028] S17: In order to complete the beam tracking task, the present invention uses a prediction function, which is expressed as where a set of model parameters is denoted as Θ; using a labeled dataset The training estimation function is represents the vision and lidar input pair, g * represents the actual optimal beam index; following standard machine learning conventions, the goal of this task is to increase the probability of accurately predicting the best beam and improve the overall success probability; from a mathematical point of view, it can be expressed as:

[0029]

[0030] Further, step S2 specifically includes the following steps:

[0031] S21: Process the raw input data using the feature extraction module. Since these data usually contain irrelevant information that does not contribute to the beam tracking task, the feature extraction module aims to reduce the dimensionality of the data by extracting key features, thereby minimizing the feature space. The main goal of this module is to identify and capture the basic features that are critical to the subsequent beam tracking process. By eliminating unnecessary information, the training process is made more stable and the beam prediction and tracking efficiency is improved.

[0032] For raw image data, the feature extraction module uses object detection to identify and annotate key objects in the image, such as vehicles. Given that beam tracking tasks require rapid processing of visual data in time series, YOLOv4 is particularly suitable for this application due to its real-time detection capabilities and low computational complexity. YOLOv4 processes the input visual (RGB) image and generates bounding boxes for all detected objects in the image, represented as Each bounding box vector S bbox =[x c ,y c ,w,h] T denote the x-center, y-center, width, and height of the detected object, respectively; therefore, the bounding box obtained from the original image is used as the key feature for the beam tracking task;

[0033] LiDAR sensors generate a two-dimensional point cloud of their surroundings by emitting and receiving laser pulses. Raw LiDAR data usually contains a lot of redundant or noisy information. The main goal of feature extraction is to extract key geometric features related to the task and effectively reduce the dimensionality of the data. This feature extraction process is crucial to support subsequent beam tracking tasks.

[0034] S22: For visual features (bounding boxes) and lidar features, the present invention uses an embedding layer to Convert to Will Convert to To embed the a priori beam index metric, the present invention applies a trainable lookup table containing the embedding vector |Q|, where the embedding vector corresponds to the beam index q;

[0035] S23: The data processed by the feature extraction module enters the recurrent neural network module, which uses the long short-term memory network model to process time series data; the long short-term memory network model can effectively capture the temporal dependencies in the input data, which is crucial for the beam tracking task; by processing features from vision and lidar data, the long short-term memory network handles the relationship between time steps and facilitates accurate prediction of future beams;

[0036] S24: The data processed by the recurrent neural network module enters the fusion classification module, and the feature vectors of the visual (RGB) and lidar data temporal features output by the long short-term memory network are spliced ​​to form a multimodal fusion feature vector; the fused feature vector is transmitted through the multi-head attention mechanism; this mechanism can assign different attention weights to different features, thereby enhancing the model's attention to key information and ignoring irrelevant information; finally, the feature vector is input into the fully connected network for classification prediction; the classifier is responsible for predicting the current and future optimal beams, that is, outputting the optimal beam index.

[0037] Further, step S3 specifically includes:

[0038] In the present invention, a contrastive learning mechanism is introduced to enhance the feature alignment capability between multimodal data (visual RGB images and lidar data) and improve the beam tracking accuracy of the millimeter wave communication system; specifically, the present invention uses the normalized temperature scale cross entropy loss function as the core of contrastive learning, which is defined as follows:

[0039]

[0040] Among them, NT-Xent(z i ,z j ) represents sample z i With z j The loss, sim(z i ,z j ) represents sample z i With z j The similarity of , τ is the temperature coefficient;

[0041] During the training process of the model, feature representations are first extracted from visual (RGB) images and lidar data respectively; in order to introduce contrastive learning, the present invention takes the visual (RGB) features and the corresponding lidar features of each data sample as positive pairs; the cosine similarity of these positive pairs is calculated using the normalized temperature scale cross entropy loss function, and the temperature parameter is used for scaling to ensure stable training at different feature scales; in addition, the present invention takes the feature pairs of all other samples in the same batch during the machine learning training process as negative pairs; the goal of contrastive learning is to maximize the similarity between positive pairs and minimize the similarity between negative pairs, thereby enhancing the consistency between features of different modalities;

[0042] During the training process, the total loss function of the model is composed of multiple parts, namely, classification loss, classification loss of visual (RGB) features and lidar features, and normalized temperature-scaled cross-entropy contrastive learning loss; this combination strategy not only improves the classification performance, but also effectively aligns the multimodal feature representation, thereby achieving higher tracking accuracy in millimeter-wave communication beam tracking tasks; by introducing contrastive learning, the model can better fuse multimodal data and achieve accurate beam tracking; a multimodal data fusion millimeter-wave communication beam tracking method based on contrastive learning is obtained.

[0043] The beneficial effects of the present invention are:

[0044] (1) The present invention combines the visual and lidar data acquired by the multi-sensors of the millimeter wave base station and utilizes a multi-modal fusion strategy of comparative learning to significantly improve the accuracy and stability of beam tracking, thereby improving the communication efficiency of the millimeter wave communication system.

[0045] (2) The present invention transforms the beam tracking problem into an optimization problem and uses machine learning to calculate and update the beam selection in real time, thereby reducing the communication delay and computational overhead of the millimeter wave system and adapting to the low-latency requirements under the multi-sensor fusion framework. This enables the tracking method to be deployed in large-scale millimeter wave communication systems.

[0046] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0048] Figure 1 This is a schematic diagram of a millimeter wave communication system model equipped with multiple sensors according to the present invention;

[0049] Figure 2 This is a schematic diagram of the multimodal data fusion neural network model proposed in the present invention;

[0050] Figure 3 This is a flow chart of the multimodal data fusion beam tracking based on contrastive learning of the present invention. DETAILED DESCRIPTION

[0051] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0052] See also Figure 1 to Figure 3 The present invention provides a millimeter wave beam tracking method based on multimodal sensing assistance. In a multi-sensor millimeter wave communication system model, various key parameters in the system are obtained, and sensing data of different modes are obtained from multiple sensors equipped in the system. A multimodal sensing-assisted millimeter wave communication beam tracking problem model is constructed, and the beam tracking problem is converted into an optimization problem based on machine learning. By establishing a multimodal data fusion neural network model for feature extraction and time series modeling, a multi-head attention mechanism is used to fuse the extracted features, and a contrastive learning mechanism is used to train the multimodal data fusion neural network model, the synergy between different modalities is optimized, and the fused features are used for beam tracking, so as to realize real-time prediction of the optimal beam at the current and multiple moments in the future, thereby reducing delays and computational overhead while ensuring millimeter wave communication efficiency.

[0053] Figure 1 This is a schematic diagram of the multi-sensor millimeter wave communication system model. Figure 1 The figure shows a model of a millimeter wave communication system around a single mobile user device, where a base station serves the mobile user device. The base station contains N antennas, a lidar sensor, and an RGB camera (visual data sensor) to provide awareness of the surrounding environment. Assume that the base station has only one antenna and the base station uses a predefined beamforming codebook. Where Q is the total number of beamforming vectors in the codebook, is the beamforming vector in the codebook.

[0054] At time step t, if each mmWave user u is beamformed by the base station using the beamforming vector f q [t] Service, channel The signal from the base station to the user is expressed as:

[0055]

[0056] in, P T is the average transmit power, For complex symbol transmission, n[t] represents a noise sample drawn from a complex Gaussian distribution.

[0057] Figure 2 The figure is a schematic diagram of the multimodal data fusion neural network model. The model mainly consists of three parts: feature extraction module, recurrent neural network module and fusion classification module. First, the original input data is processed using the feature extraction block. The main goal of this module is to identify and capture the basic features that are critical to the subsequent beam tracking process.

[0058] The data processed by the feature extraction module enters the recurrent neural network module, which uses a long short-term memory network to process time series data. The long short-term memory network model can effectively capture the time dependency in the input data.

[0059] The data processed by the recurrent neural network module enters the fusion and classification module, which splices the feature vectors of the image output by the long short-term memory network and the time characteristics of the lidar data to form a multimodal fusion feature vector. The fused feature vector is transmitted through the multi-head attention mechanism. This mechanism can assign different attention weights to different features, thereby enhancing the model's attention to key information and ignoring irrelevant information. Finally, the feature vector is input into the fully connected network for classification prediction. The classifier is responsible for predicting the current and future optimal beams, that is, outputting the optimal beam index.

[0060] Figure 3 The flowchart of the multimodal data fusion beam tracking based on the contrastive learning mechanism of the present invention includes the following steps:

[0061] V1~V5: Obtain the model parameters of the multi-sensor millimeter wave communication system, build a multi-modal perception-assisted millimeter wave communication beam tracking problem model, and transform the beam tracking problem into an optimization problem based on machine learning.

[0062] V6~V12: Construct a multimodal data fusion neural network model for millimeter wave communication beam tracking, use neural networks to extract features from the acquired multimodal data, and pair image features and lidar features into positive and negative pairs. Model multimodal time series features through recurrent neural networks, use normalized temperature scale cross entropy loss function for comparative learning, maximize positive similarity, and minimize negative similarity. Use fusion classification module to fuse time series multimodal feature vectors for training.

[0063] V13~V15: Calculate the loss function and update the model parameters by combining the classification loss and contrastive learning loss.

[0064] V16~V17: After the training termination conditions are met, the neural network parameters are generated, and the multi-sensor millimeter wave communication system performs beam tracking based on the trained beam tracking network.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A millimeter wave beam tracking method based on contrastive learning, characterized in that: The method specifically comprises the following steps: S1: Obtain the model parameters of the multi-sensor millimeter wave communication system, build a multi-modal sensing-assisted millimeter wave communication beam tracking problem model, and transform the beam tracking problem into an optimization problem based on machine learning; S2: Construct a millimeter wave communication beam tracking multimodal data fusion neural network model, including: feature extraction module, recurrent neural network module and fusion classification module; the model obtains perception data of different modes from multiple sensors equipped in the system, uses the feature extraction module to extract features from the acquired multimodal data, and models the multimodal time series features through the recurrent neural network module. The fusion classification module uses a multi-head attention mechanism to fuse the extracted features; S3: Use contrastive learning mechanism to train multimodal data fusion neural network model, optimize the synergy between different modalities, use the fused features for beam tracking, and predict the best indexes of current and future multiple optimal beams in real time.

2. The millimeter wave beam tracking method based on contrastive learning according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11: Consider a millimeter wave communication system model around a single mobile user device, where a base station provides services for the mobile user device; the base station contains N antennas, a lidar sensor, and a visual data sensor to provide awareness of the surrounding environment; assuming that the base station has only one antenna, the base station uses a predefined beamforming codebook Where Q is the total number of beamforming vectors in the codebook, is the beamforming vector in the codebook; S12: At time step t, if each mmWave user u is beamformed by the base station using the beamforming vector f q [t] Service, channel The signal from the base station to the user is expressed as: in, P T is the average transmit power, To transmit complex symbols, (·) H represents the conjugate transpose, n[t] represents a noise sample extracted from a complex Gaussian distribution; S13: Multimodal beam tracking task using lidar and visual information in the single-user case. Based on the available perception data up to a given time t-1, the base station aims to predict the next The best beam for the upcoming step, in particular, the best beam from time t to time (t+α-1); the goal of this task is to obtain the optimal beam from the codebook Select the best beam f from the accessible candidate beams * [t] to maximize the received signal power; mathematically expressed as: S14: From a predetermined codebook The best beam index is selected in the image, and the signal is concentrated in a certain direction in space; the spatial dimension of the scene is divided into many possibly overlapping sectors by beam vectors, and a unique value is assigned to each sector; therefore, the machine learning task can be converted into a classification task using a predetermined codebook, where the user's real-time position in the visual and lidar scene determines which beam index in the codebook to assign; under the specified codebook constraints, the optimal beam f is individually identified by the matching codebook index * [t]; At time t, the optimal beam index is expressed as: Among them, |Q| represents The optimal beam index is recorded as g[t]. It should be noted that finding the optimal beam is the same as finding the optimal beam index under codebook constraints. Therefore, the beam tracking problem is expressed as: in, represents the conditional probability, represents the predicted optimal beam index at time step t, o t-α is the auxiliary information collected before time step (t-α+1), which contains some detailed information of the best beam at time t; S15: The visual perception information obtained at time step t is defined as Where H, W and C represent the height, width and number of color channels of the image respectively; definition is the corresponding user information obtained by the LiDAR sensor at time step t, where D represents the angle in the LiDAR perception field of view, and each angle has a corresponding distance value; using the input visual information x u and lidar information u , track the optimal beam index g ＊ Serve user u; To achieve this, use machine learning methods to learn a function f Θ (S), where S = (x u ,l u ); the observed visual and lidar information is fed into the function, and its output probability In the codebook The index of the most likely element in In form, it is: in, represents the optimal beam index, Indicates the beam index value; S16: To perform beam tracking using vision and lidar data, the goal is to develop a function that predicts the optimal future beam starting at time step t based on the perception information collected up to time step t-1; let represents the visual and lidar data sequence, where i represents the time step within the observation window; then the visual and lidar assisted beam tracking optimization problem is expressed as: S17: In order to complete the beam tracking task, a prediction function is used, which is expressed as where a set of model parameters is denoted as Θ; using a labeled dataset The training estimation function is represents the visual and lidar input pair, and g represents the actual optimal beam index; following standard machine learning conventions, the goal of this task is to increase the probability of accurately predicting the best beam and improve the overall success probability; from a mathematical point of view, it is expressed as:

3. The millimeter wave beam tracking method based on contrastive learning according to claim 2, characterized in that: Step S2 specifically includes the following steps: S21: Processing the original input data using a feature extraction module; the feature extraction module reduces the dimension of the data by extracting key features, thereby minimizing the feature space; For raw image data, the feature extraction module uses object detection to identify and annotate key objects in the image. Specifically, the input visual image is processed using YOLOv4 to generate bounding boxes for all detected objects in the image, represented as Each bounding box vector S bbox =[x c ,y c ,w,h] T denote the x-center, y-center, width, and height of the detected object, respectively; therefore, the bounding box obtained from the original image is used as the key feature for the beam tracking task; The LiDAR sensor generates a two-dimensional point cloud of its surroundings by emitting and receiving laser pulses; the goal of feature extraction is to extract key geometric features relevant to the task; S22: For visual features and lidar features, an embedding layer is used to Convert to Will Convert to To embed the prior beam index metric, a trainable lookup table containing the embedding vector |Q| is applied, where the embedding vector corresponds to the beam index q; S23: The data processed by the feature extraction module enters the recurrent neural network module, which uses a long short-term memory network model to process time series data. By processing features from vision and lidar data, the long short-term memory network handles the relationship between time steps and facilitates accurate prediction of future beams. S24: The data processed by the recurrent neural network module enters the fusion classification module, and the feature vectors of the temporal features of the visual and lidar data output by the long short-term memory network are spliced ​​to form a multimodal fusion feature vector; the fused feature vector is transmitted through the multi-head attention mechanism; finally, the feature vector is input into the fully connected network for classification prediction; the classifier is responsible for predicting the current and future optimal beams, that is, outputting the optimal beam index.

4. The millimeter wave beam tracking method based on contrastive learning according to claim 3, characterized in that: Step S3 specifically includes: introducing a contrastive learning mechanism to enhance the feature alignment capability between multimodal data and improve the beam tracking accuracy of the millimeter wave communication system; specifically, using a normalized temperature scale cross entropy loss function as the core of contrastive learning, which is defined as follows: Among them, NT-Xent(z i ,z j ) represents sample z i With z j The loss, sim(z i ,z j ) represents sample z i With z j The similarity of , τ is the temperature coefficient; During the training of the model, feature representations are first extracted from visual images and lidar data respectively. In order to introduce contrastive learning, the visual features and corresponding lidar features of each data sample are taken as positive pairs. The cosine similarity of these positive pairs is calculated using the normalized temperature-scaled cross-entropy loss function and scaled with a temperature parameter to ensure stable training at different feature scales. In addition, the feature pairs of all other samples in the same batch during machine learning training are taken as negative pairs. The goal of contrastive learning is to maximize the similarity between positive pairs and minimize the similarity between negative pairs, thereby enhancing the consistency between features of different modalities. During the training process, the total loss function of the model consists of multiple parts, namely, classification loss, classification loss of visual features and lidar features, and normalized temperature-scaled cross-entropy contrast learning loss.

Images