A vision-assisted millimeter-wave beam prediction method for low-light environments

Abstract

The present invention relates to a visually assisted millimeter-wave beam prediction method for low-light environments, belonging to the field of wireless communications. This method is targeted at millimeter-wave communication systems. A camera installed at a base station collects image data of the communication environment under low-light conditions, constructs a millimeter-wave beam prediction system model, and uses a deep neural network model and curriculum training strategy to learn from the collected image data and predict the optimal communication beam. This method can improve model performance in low-light environments, accelerate the convergence of the network model, increase the accuracy of beam prediction, and enhance the robustness and reliability of the millimeter-wave communication system in low-light environments.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communications and relates to a vision-assisted millimeter wave beam prediction method for low-light environments. Background Art

[0002] Millimeter waves, with their large bandwidth and low latency, enable efficient data transmission and are widely used in communication systems such as 5G and 6G. However, due to their short transmission range and susceptibility to obstruction, millimeter waves can easily reduce the reliability and stability of millimeter wave communications. To ensure the reliability of millimeter wave communications, large-scale antenna arrays are typically deployed to form narrow beams for long-distance wireless communication. However, such large-scale antenna arrays incur significant beam training overhead and waste resources. To address this issue, wireless communication solutions assisted by sensory data have attracted considerable attention. These solutions use artificial intelligence (AI) to learn the changing patterns of sensory data in the communication environment and predict the optimal communication beam.

[0003] Video and image data contain rich information closely related to wireless communications, such as user location, and are therefore widely used for millimeter-wave beam prediction. However, in low-light environments, the image information captured by the camera is significantly reduced, which leads to a significant decrease in the accuracy of beam prediction based on visual information. Taking industrial manufacturing scenarios as an example, limited ambient lighting conditions significantly increase the difficulty of vision-based millimeter-wave prediction, resulting in unstable data transmission. Therefore, improving the accuracy of beam prediction in low-light environments has become a critical issue that needs to be addressed. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a visually assisted millimeter-wave beam prediction method for low-light environments. Aiming at the problem of visually assisted millimeter-wave beam prediction in low-light scenarios, a course training method is adopted to ensure high-accuracy beam prediction in low-light communication environments.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A vision-assisted millimeter-wave beam prediction method for low-light environments addresses the increased difficulty of beam prediction in low-light communication environments due to the reduction of wireless communication-related information in image data. A curriculum-based beam prediction method is proposed. By adding difficult (low-light) samples to the training set in stages based on the difficulty level of the training samples, the prediction model learns from easy to difficult levels, thereby improving beam prediction accuracy in low-light environments.

[0007] The method specifically comprises the following steps:

[0008] S1: Build a millimeter-wave wireless communication system. Use a neural network model to extract features from image data under low-light conditions, optimize beamforming vectors, and select the optimal communication beam from a beam codebook to maximize received signal power and reduce beam prediction complexity.

[0009] S2: Collect image data of millimeter-wave wireless communication scenarios and the corresponding optimal beamforming dataset in the codebook, preprocess the collected image data, build a millimeter-wave beam prediction deep neural network model based on the collected dataset, and define the model loss function and model optimizer to train the model;

[0010] The preprocessing of the collected image data specifically comprises: dividing the collected image dataset into a low-light image subset and a normal-light image subset, adding the low-light image dataset to the normal-light image training set in stages based on a curriculum training method, using the normal-light image dataset as the original training set in the first stage of training, uniformly adding low-light images to the training set in subsequent stages, and ultimately predicting the optimal beam in the entire dataset;

[0011] S3: Input the entire dataset into the trained millimeter-wave beam prediction deep neural network model to predict the optimal beam.

[0012] Furthermore, step S1 specifically includes the following steps:

[0013] S11: Construct a millimeter wave wireless communication system, including a mobile user device equipped with an omnidirectional antenna to receive downlink signals from a base station; a fixed base station equipped with a camera to capture real-time environmental images; the base station includes a uniform linear antenna array consisting of M antenna elements and a beamforming vector w q transmitting a signal to a mobile user equipment, wherein represents the beamforming vector in the codebook, codebook Q is the number of beam vectors in the codebook; the system uses orthogonal frequency division multiplexing (OFDM) technology to transmit signals through K subcarriers, and the communication channel of each subcarrier is represented by h k , where k = 1, 2, ..., K;

[0014] S12: Define the beam prediction optimization problem and predict the optimal beamforming vector w through the neural network model ＊ , so that the received signal power on a given subcarrier is maximized, then the optimization objective formula is:

[0015]

[0016] The beam prediction goal can be defined as selecting the optimal communication beam from the beam codebook Q to connect and communicate with the user; if represents the communication scene image captured by the base station camera, and w represents the optimal beam predicted by the neural network. Then the beam prediction function can be expressed as:

[0017]

[0018] Among them, θ w represents the optimization parameter of the network model. Since there is a one-to-one correspondence between the beam vector and its index in the codebook, the prediction difficulty can be further reduced by directly predicting the beam index. The original formula can be expressed as:

[0019]

[0020] in, Represents the predicted optimal beam vector index.

[0021] Furthermore, in step S2, a millimeter wave beam prediction deep neural network model is constructed and trained, specifically including the following steps:

[0022] S21: Use a base station equipped with a camera to capture real-time environmental images. The images are in RGB format and contain multiple feature information in the communication environment. The base station uses the captured image information from the preset beam codebook. Select the optimal beamforming vector w from ＊ , the beam codebook includes multiple beamforming vectors To cover the entire communication scenario; the collected data sets include normal light and low light data sets;

[0023] S22: Build a millimeter-wave beam prediction neural network model. Use the pre-trained neural network model ResNet50 to perform beam prediction tasks. Input the captured image into the ResNet network model, which processes the image data and extracts deep features of the image through multi-layer convolution and pooling operations.

[0024] S23: After image feature extraction is completed, the model uses a fully connected layer to map the features to the beamforming vector space and calculates the probability distribution of each beamforming vector through the Softmax function. The model is trained using the cross entropy loss function. The optimization goal is to maximize the probability of correctly predicting the beamforming vector. The loss function is expressed as:

[0025]

[0026] Among them, L cross-entropy is the loss function, n is the number of training samples, P(s n |X n ) is the model in a given image X n In the case of , the prediction beam vector index sn To further improve the generalization ability of the model, the Adam optimizer is used to dynamically adjust the model parameters to ensure that the model converges quickly during training;

[0027] S24: Selecting the beamforming vector with the highest probability as the output result to ensure the best communication quality in the current communication environment and guarantee the effectiveness and reliability of the connection.

[0028] Furthermore, in step S2, the collected image data is preprocessed, specifically including the following steps:

[0029] S201: Divide the training data set into simple samples and difficult samples, wherein the simple samples include images taken under normal lighting conditions, and the difficult samples include images taken under low light conditions. represents the divided normal light image dataset, Represents the divided low-light dataset; the sample distributions of these two subsets are P normal , P low ;

[0030] S202: Using a curriculum training method, the training process starts with image data under normal lighting conditions and gradually introduces images taken in low-light environments to improve the beam prediction accuracy of the neural network model in low-light environments. By adding low-light data to the training set in stages, the network model's ability to learn complex low-light samples is enhanced;

[0031] Specifically, the training process is divided into V stages. In each stage, a low-light dataset with more difficult learning is added to the training set. Let λ i represents the random sampling probability of adding low-light samples to the training set at the i-th stage; represents the training set of the i-th stage, P i Represents the distribution of training set samples in the i-th stage, then and P i It can be expressed as:

[0032]

[0033] P i ∝P normal +λ i ·P low

[0034] in, Indicated by λ i The sampling probability is used to randomly sample the data set The new data set is formed, so in the final stage, the training set contains all the normal light and low light data sets;

[0035] Let λ1 = 0 and This means that in the first stage of training, all normal light images are used as the initial training set to train the model, and in subsequent stages, a uniform amount of low light samples is added to the training set. to the training set to form a new training set; finally, the optimal beam is predicted on the entire dataset.

[0036] The beneficial effects of the present invention are:

[0037] (1) The present invention takes into account the problem of reduced accuracy of millimeter wave beam prediction in low-light environments, optimizes the prediction in low-light environments, and ensures the stability and reliability of 5G millimeter wave communications.

[0038] (2) The present invention adopts a course training method to improve the beam prediction performance. The phased learning method of course training is used to enable the model to learn more common features from simple samples, which effectively improves the accuracy of model prediction and enables industrial 5G communication to maintain stable and reliable implementation.

[0039] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0041] Figure 1 The visually assisted millimeter wave wireless communication system model provided by the present invention;

[0042] Figure 2 This is a schematic diagram of deep learning training based on course training in the present invention;

[0043] Figure 3 This is a flow chart of the visually assisted millimeter wave beam prediction method based on course training of the present invention. DETAILED DESCRIPTION

[0044] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0045] See also Figures 1 to 3 This paper provides a vision-assisted millimeter-wave beam prediction method for low-light environments. For a vision-assisted millimeter-wave communication system, a camera installed at the base station captures real-time low-light images of the communication environment to predict which communication beam in the codebook the base station should select for communication with the user. This method uses the ResNet50 network model for beam prediction, and utilizes a curriculum learning training method to accelerate training and improve prediction accuracy. This invention presents a deep learning-based beam prediction solution.

[0046] Figure 1 This is a model diagram of a visually assisted millimeter wave communication system. It includes a fixed base station, a mobile user, and other mobile scatterers. The base station is equipped with a camera and an antenna array with M antennas. The base station captures and collects real-time images of the communication environment to form a raw data set. In this system, the base station uses a predefined beam steering codebook. Communicate with the user connection.

[0047] Figure 2 The following diagram shows the deep neural network prediction framework for curriculum-based training. The entire training process is divided into V stages, each of which contains v training rounds. Normal-light images are used as the initial training set, and low-light datasets are gradually and uniformly randomly added to the training set in subsequent stages. For the beam prediction task, a deep residual network ResNet50 is used for prediction. In the final classification output layer, the predicted output categories are fine-tuned to the number of beam codebooks. When image data is input into the model, the convolutional layers and residual blocks in ResNet50 extract deep features from the image, which are then passed through a fully connected layer to produce the predicted output.

[0048] Figure 3 This is a flow chart of the visually assisted millimeter wave beam prediction method based on course training of the present invention, which specifically includes the following steps:

[0049] V1~V4: Obtain relevant data sets for beam prediction, divide the collected data sets into two subsets: low-light image data sets and normal-light image data sets for course training to initialize model hyperparameters, the number of course training stages, and the number of training rounds for each stage.

[0050] V4-V10: Build a ResNet50 network model for beamforming training and define the loss function and optimizer in the network model. Use normal-light images as the original training set for the first stage of training. In subsequent stages of training, gradually and evenly add low-light image datasets for course training.

[0051] V10~V15: Network training and prediction process. During the training process, the course is completed in stages. The entire network is trained by inputting images, the loss function is calculated, and the network parameters are updated by backpropagation. After the training termination conditions are met, the predicted optimal beam is finally output.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. A vision-assisted millimeter-wave beam prediction method for low-light environments, characterized in that: The method specifically comprises the following steps: S1: Build a millimeter-wave wireless communication system. Use a neural network model to extract features from image data under low-light conditions, optimize the beamforming vector, and select the optimal communication beam from the beam codebook to maximize the received signal power. This includes the following steps: S11: Construct a millimeter wave wireless communication system, including a mobile user device equipped with an omnidirectional antenna to receive downlink signals from a base station; a fixed base station equipped with a camera to capture real-time environmental images; the base station includes a uniform linear antenna array consisting of M antenna elements and a beamforming vector w q transmitting a signal to a mobile user equipment, wherein represents the beamforming vector in the codebook, codebook Q is the number of beam vectors in the codebook; the system uses orthogonal frequency division multiplexing technology to transmit signals through K subcarriers, and the communication channel of each subcarrier is represented by h k , where k = 1, 2, ..., K; S12: Define the beam prediction optimization problem and predict the optimal beamforming vector w through the neural network model ＊ , so that the received signal power on a given subcarrier is maximized, then the optimization objective formula is: The beam prediction goal is defined as selecting the optimal communication beam from the beam codebook Q to connect and communicate with the user; if represents the communication scene image captured by the base station camera, and w represents the optimal beam predicted by the neural network. Then the beam prediction function is expressed as: Among them, θ w represents the optimization parameter of the network model. Since there is a one-to-one correspondence between the beam vector and its index in the codebook, the prediction difficulty is further reduced by directly predicting the beam index. The original formula is expressed as: in, represents the predicted optimal beam vector index; S2: Collect image data of millimeter-wave wireless communication scenarios and the corresponding optimal beamforming dataset in the codebook, preprocess the collected image data, build a millimeter-wave beam prediction deep neural network model based on the collected dataset, and define the model loss function and model optimizer to train the model; The preprocessing of the collected image data specifically comprises: dividing the collected image dataset into a low-light image subset and a normal-light image subset, adding the low-light image dataset to the normal-light image training set in stages based on a curriculum training method, using the normal-light image dataset as the original training set in the first stage of training, uniformly adding low-light images to the training set in subsequent stages, and ultimately predicting the optimal beam in the entire dataset; Build and train a deep neural network model for mmWave beam prediction, which includes the following steps: S21: Use a base station equipped with a camera to capture real-time environmental images. The images are in RGB format and contain multiple feature information in the communication environment. The base station uses the captured image information from the preset beam codebook. Select the optimal beamforming vector w from ＊ , the beam codebook includes multiple beamforming vectors To cover the entire communication scenario; the collected data sets include normal light and low light data sets; S22: Build a millimeter-wave beam prediction neural network model. Use the pre-trained neural network model ResNet50 to perform beam prediction tasks. Input the captured image into the ResNet network model, which processes the image data and extracts deep features of the image through multi-layer convolution and pooling operations. S23: After image feature extraction is completed, the model uses a fully connected layer to map the features to the beamforming vector space and calculates the probability distribution of each beamforming vector through the Softmax function. The model is trained using the cross entropy loss function. The optimization goal is to maximize the probability of correctly predicting the beamforming vector. The loss function is expressed as: Among them, L cross-entropy is the loss function, n is the number of training samples, P(s n |X n ) is the model in a given image X n In the case of , the prediction beam vector index s n To further improve the generalization ability of the model, the Adam optimizer is used to dynamically adjust the model parameters to ensure that the model converges quickly during training; S24: Select the beamforming vector with the highest probability as the output result; S3: Input the entire dataset into the trained millimeter-wave beam prediction deep neural network model to predict the optimal beam.

2. The visually assisted millimeter wave beam prediction method according to claim 1, characterized in that: In step S2, the collected image data is preprocessed, specifically including the following steps: S201: Divide the training data set into simple samples and difficult samples, wherein the simple samples include images taken under normal lighting conditions, and the difficult samples include images taken under low light conditions. represents the divided normal light image dataset, Represents the divided low-light dataset; the sample distributions of the two subsets are P normal , P low ; S202: Using a curriculum training method, the training process starts with image data under normal lighting conditions and gradually introduces images taken in low-light environments to improve the beam prediction accuracy of the neural network model in low-light environments. By adding low-light data to the training set in stages, the network model's ability to learn complex low-light samples is enhanced; Specifically, the training process is divided into V stages. In each stage, a low-light dataset with more difficult learning is added to the training set. Let λ i represents the random sampling probability of adding low-light samples to the training set at the i-th stage; represents the training set of the i-th stage, P i Represents the distribution of training set samples in the i-th stage, then and P i Expressed as: P i ∝P normal +λ i ·P low in, Indicated by λ i The sampling probability is used to randomly sample the data set The new data set is formed, so in the final stage, the training set contains all the normal light and low light data sets; Let λ1 = 0 and This means that in the first stage of training, all normal light images are used as the initial training set to train the model, and in subsequent stages, a uniform amount of low light samples is added to the training set. to the training set to form a new training set; finally, the optimal beam is predicted on the entire dataset.

Images