Visual assistance millimeter wave beam prediction method for low-light environment

Abstract

The invention relates to a vision-assisted millimeter wave beam prediction method for a low-light environment, and belongs to the field of wireless communication. The method is oriented to a millimeter wave communication system, image data of a communication environment under a low light condition is acquired through a camera installed on a base station, a millimeter wave beam prediction system model is constructed, the acquired image data is learned by using a deep neural network model and a course training strategy, and an optimal communication beam is predicted. According to the method, the performance of the model can be improved in the low-light environment, the convergence speed of the network model is increased, the accuracy of beam prediction can be improved, and the robustness and reliability of the millimeter wave communication system in the low-light environment are enhanced.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communications and relates to a vision-assisted millimeter wave beam prediction method for low-light environments. Background Art

[0002] Millimeter waves have the characteristics of large bandwidth and low latency, and can achieve efficient data transmission. They can be widely used in communication systems such as 5G and 6G. However, due to the short transmission distance and susceptibility to obstruction, the reliability and stability of millimeter wave communication can be easily reduced. In order to ensure the reliability of millimeter wave communication, it is usually necessary to deploy large-scale antenna arrays to form narrow beams to achieve long-distance wireless communication. However, this large-scale antenna array brings huge beam training overhead and resource waste. In order to solve this problem, wireless communication solutions based on perception data assistance have received much attention. Through artificial intelligence technology, the law of changes in perception data in the communication environment is learned to predict the optimal communication beam.

[0003] Video and image data contain rich information closely related to wireless communications, such as user location, so they are widely used in millimeter wave beam prediction. However, when in a low-light environment, the image information captured by the camera will be significantly reduced, which will cause the accuracy of beam prediction based on visual information to drop significantly. Taking industrial manufacturing scenarios as an example, limited ambient lighting conditions will significantly increase the difficulty of vision-based millimeter wave prediction, thereby causing unstable data transmission. Therefore, how to improve the accuracy of beam prediction in low-light environments has become an important issue that needs to be solved urgently. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a vision-assisted millimeter-wave beam prediction method for low-light environments. Aiming at the problem of vision-assisted millimeter-wave beam prediction in low-light scenes, a course training method is adopted to ensure high-accuracy beam prediction in low-light communication environments.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A vision-assisted millimeter-wave beam prediction method for low-light environments. In order to solve the problem of increased difficulty in beam prediction due to the reduction of wireless communication related information in image data in low-light communication environments, a beam prediction method based on curriculum training is proposed. According to the difficulty level of the training samples, difficult (low-light) samples are added to the training set in stages, so that the prediction model learns from easy to difficult, thereby improving the beam prediction accuracy in low-light environments.

[0007] The method specifically comprises the following steps:

[0008] S1: Build a millimeter wave wireless communication system, extract features from image data under low light conditions through a neural network model, optimize the beamforming vector, and select the optimal communication beam from the beam codebook to maximize the received signal power and reduce the complexity of beam prediction;

[0009] S2: Collect image data of millimeter wave wireless communication scenarios and the corresponding optimal beamforming data set in the codebook, preprocess the collected image data, build a millimeter wave beam prediction deep neural network model based on the collected data set, and define the model loss function and model optimizer to train the model;

[0010] The preprocessing of the collected image data specifically comprises: dividing the collected image data set into a low-light image subset and a normal-light image subset, adding the low-light image data set to the normal-light image training set in stages based on the course training method, using the normal-light image data set as the original training set in the first stage of training, uniformly adding the low-light images to the training set in subsequent stages, and finally predicting the optimal beam in the entire data set;

[0011] S3: Input the entire data set into the trained millimeter-wave beam prediction deep neural network model to predict the optimal beam.

[0012] Further, step S1 specifically includes the following steps:

[0013] S11: Construct a millimeter wave wireless communication system, including a mobile user device equipped with an omnidirectional antenna to receive downlink signals from a base station; a fixed base station equipped with a camera to capture real-time environmental images; the base station includes a uniform linear antenna array consisting of M antenna elements and a beamforming vector w q transmitting a signal to a mobile user equipment, wherein represents the beamforming vector in the codebook, codebook Q is the number of beam vectors in the codebook; the system uses orthogonal frequency division multiplexing (OFDM) technology to transmit signals through K subcarriers, and the communication channel of each subcarrier is represented by h k , where k = 1, 2, ..., K;

[0014] S12: Define the beam prediction optimization problem and predict the optimal beamforming vector w through the neural network model ＊ , so that the received signal power on a given subcarrier is maximized, then the optimization objective formula is:

[0015]

[0016] The beam prediction objective can be defined as selecting the best communication beam from the beam codebook Q to connect and communicate with the user; if represents the communication scene image captured by the base station camera, and w represents the optimal beam predicted by the neural network. Then the beam prediction function can be expressed as:

[0017]

[0018] Among them, θ w represents the optimization parameter of the network model. Since there is a one-to-one correspondence between the beam vector and its index in the codebook, the prediction difficulty can be further reduced by directly predicting the beam index. The original formula can be expressed as:

[0019]

[0020] in, Represents the predicted optimal beam vector index.

[0021] Further, in step S2, a millimeter wave beam prediction deep neural network model is constructed and trained, specifically including the following steps:

[0022] S21: Use a base station equipped with a camera to capture real-time environmental images. The images are in RGB format and contain multiple feature information in the communication environment. The base station uses the captured image information from the preset beam codebook. Select the optimal beamforming vector w from ＊ , the beam codebook includes multiple beamforming vectors To cover the entire communication scenario; the collected data sets include normal light and low light data sets;

[0023] S22: Build a millimeter wave beam prediction neural network model, use the pre-trained neural network model ResNet50 to perform the beam prediction task, input the collected image into the ResNet network model, and the ResNet network model processes the image data and extracts the deep features of the image through multi-layer convolution and pooling operations;

[0024] S23: After the image feature extraction is completed, the model uses a fully connected layer to map the features to the beamforming vector space, and calculates the probability distribution of each beamforming vector through the Softmax function. The training of the model adopts the cross entropy loss function. The optimization goal is to maximize the probability of correctly predicting the beamforming vector. The loss function is expressed as:

[0025]

[0026] Among them, L cross-entropy is the loss function, n is the number of training samples, P(s n |X n ) is the model in a given image X n In the case of , the prediction beam vector index sn To further improve the generalization ability of the model, the Adam optimizer is used to dynamically adjust the model parameters to ensure that the model converges quickly during training;

[0027] S24: Select the beamforming vector with the highest probability as the output result to ensure the best communication quality in the current communication environment and guarantee the effectiveness and reliability of the connection.

[0028] Further, in step S2, the collected image data is preprocessed, specifically including the following steps:

[0029] S201: Divide the training data set into simple samples and difficult samples, where the simple samples include images taken under normal lighting conditions and the difficult samples include images taken under low light conditions. represents the divided normal light image dataset, Represents the divided low-light data set; the sample distributions of these two subsets are P normal , P low ;

[0030] S202: adopting a course training method, wherein the training process starts with image data under normal lighting conditions, and gradually introduces images taken under low-light conditions to improve the beam prediction accuracy of the neural network model under low-light conditions, and increases the learning ability of the network model for complex low-light samples by adding low-light data to the training set in stages;

[0031] Specifically, the training process is divided into V stages. In each stage, a low-light dataset with more difficult learning is added to the training set. Let λ i represents the random sampling probability of adding low-light samples to the training set in the i-th stage; represents the training set of the i-th stage, P i represents the sample distribution of the training set in the i-th stage, then and P i It can be expressed as:

[0032]

[0033] P i ∝P normal +λ i ·P low

[0034] in, Represents a randomly sampled data set The total number of λ i To form a new data set, the training set in the final stage contains all normal light and low light data sets;

[0035] Let λ 1 =0 and This indicates that in the first stage of training, all normal light images are used as the initial training set to train the model, and in subsequent stages, low light samples are uniformly added to the training set. to the training set to form a new training set; finally, the optimal beam is predicted on the entire data set.

[0036] The beneficial effects of the present invention are:

[0037] (1) The present invention takes into account the problem of reduced accuracy of millimeter wave beam prediction in low-light environments, and optimizes the prediction in low-light environments, thereby ensuring the stability and reliability of 5G millimeter wave communications.

[0038] (2) The present invention adopts a course training method to improve the beam prediction performance, and utilizes the phased learning method of course training to enable the model to learn more common features from simple samples, which effectively improves the accuracy of model prediction and enables industrial 5G communication to maintain stable and reliable implementation.

[0039] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0041] Figure 1 A model of a visually assisted millimeter wave wireless communication system provided by the present invention;

[0042] Figure 2 This is a schematic diagram of deep learning training based on course training of the present invention;

[0043] Figure 3 This is a flow chart of the visually assisted millimeter wave beam prediction method based on course training of the present invention. DETAILED DESCRIPTION

[0044] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0045] See also Figure 1 to Figure 3 The present invention provides a vision-assisted millimeter wave beam prediction method for low-light environments. For a vision-assisted millimeter wave communication system, a camera installed at the base station captures real-time low-light images of the communication environment to predict which communication beam in the codebook the base station should select to communicate with the user. The ResNet50 network model is used for beam prediction, and the curriculum learning training method is used to speed up the training speed and improve the prediction accuracy. A beam prediction scheme based on deep learning is invented.

[0046] Figure 1 This is a model diagram of a visually assisted millimeter wave communication system. It includes a fixed base station, a mobile user, and other mobile scatterers. The base station is equipped with a camera and an antenna array, and the antenna array has M antennas. The base station captures and collects real-time images of the communication environment to form an original data set. In this system, the base station uses a predefined beam steering codebook Communicate with the user connection.

[0047] Figure 2 The following is a diagram of the deep neural network prediction framework based on course training. The entire training process is divided into V stages, each of which contains v training rounds. Normal light images are used as the initial training set, and low light data sets are gradually and uniformly randomly added to the training set in subsequent stages. For the beam prediction task, the deep residual network ResNet50 is used for prediction. In the final classification output layer, the predicted output category is fine-tuned to the number of beam codebooks. When the image data is input into the model, the convolutional layer and residual block in ResNet50 extract the deep features in the image, and then pass through the fully connected layer to generate the predicted output.

[0048] Figure 3 This is a flow chart of the visually assisted millimeter wave beam prediction method based on course training of the present invention, which specifically includes the following steps:

[0049] V1~V4: Obtain relevant data sets for beam prediction, and divide the collected data sets into two subsets: low-light image data sets and normal-light image data sets for course training to initialize model hyperparameters, the number of course training stages, and the number of training rounds for each stage.

[0050] V4~V10: Build a ResNet50 network model for beam training and define the loss function and optimizer in the network model. Use normal light images as the original training set for the first stage of training. In the subsequent training stages, gradually and evenly add low-light image data sets for course training.

[0051] V10~V15: Network training and prediction process. During the training process, the course is completed in stages. The entire network is trained by inputting images, the loss function is calculated, and the network parameters are updated by backpropagation. After the training termination conditions are met, the predicted optimal beam is finally output.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A vision-assisted millimeter-wave beam prediction method for low-light environments, characterized in that: The method specifically comprises the following steps: S1: Build a millimeter wave wireless communication system, extract features from image data under low light conditions through a neural network model, optimize the beamforming vector, and select the optimal communication beam from the beam codebook to maximize the received signal power; S2: Collect image data of millimeter wave wireless communication scenarios and the corresponding optimal beamforming data set in the codebook, preprocess the collected image data, build a millimeter wave beam prediction deep neural network model based on the collected data set, and define the model loss function and model optimizer to train the model; The preprocessing of the collected image data specifically comprises: dividing the collected image data set into a low-light image subset and a normal-light image subset, adding the low-light image data set to the normal-light image training set in stages based on the course training method, using the normal-light image data set as the original training set in the first stage of training, uniformly adding the low-light images to the training set in subsequent stages, and finally predicting the optimal beam in the entire data set; S3: Input the entire data set into the trained millimeter-wave beam prediction deep neural network model to predict the optimal beam.

2. The visually assisted millimeter wave beam prediction method according to claim 1, characterized in that: Step S1 specifically includes the following steps: S11: Construct a millimeter wave wireless communication system, including a mobile user device equipped with an omnidirectional antenna to receive downlink signals from a base station; a fixed base station equipped with a camera to capture real-time environmental images; the base station includes a uniform linear antenna array consisting of M antenna elements and a beamforming vector w q transmitting a signal to a mobile user equipment, wherein represents the beamforming vector in the codebook, codebook Q is the number of beam vectors in the codebook; the system uses orthogonal frequency division multiplexing technology to transmit signals through K subcarriers, and the communication channel of each subcarrier is represented by h k , where k = 1, 2, ..., K; S12: Define the beam prediction optimization problem and predict the optimal beamforming vector w through the neural network model ★ , so that the received signal power on a given subcarrier is maximized, then the optimization objective formula is: The beam prediction objective is defined as selecting the best communication beam from the beam codebook Q to connect and communicate with the user; if represents the communication scene image captured by the base station camera, and w represents the optimal beam predicted by the neural network. Then the beam prediction function is expressed as: Among them, θ w represents the optimization parameter of the network model. Since there is a one-to-one correspondence between the beam vector and its index in the codebook, the prediction difficulty is further reduced by directly predicting the beam index. The original formula is expressed as: in, Represents the predicted optimal beam vector index.

3. The visually assisted millimeter wave beam prediction method according to claim 2, characterized in that: In step S2, a millimeter wave beam prediction deep neural network model is constructed and trained, which specifically includes the following steps: S21: Use a base station equipped with a camera to capture real-time environmental images. The images are in RGB format and contain multiple feature information in the communication environment. The base station uses the captured image information from the preset beam codebook. Select the optimal beamforming vector w from ★ , the beam codebook includes multiple beamforming vectors To cover the entire communication scenario; the collected data sets include normal light and low light data sets; S22: Build a millimeter wave beam prediction neural network model, use the pre-trained neural network model ResNet50 to perform the beam prediction task, input the collected image into the ResNet network model, and the ResNet network model processes the image data and extracts the deep features of the image through multi-layer convolution and pooling operations; S23: After the image feature extraction is completed, the model uses a fully connected layer to map the features to the beamforming vector space, and calculates the probability distribution of each beamforming vector through the Softmax function. The training of the model adopts the cross entropy loss function. The optimization goal is to maximize the probability of correctly predicting the beamforming vector. The loss function is expressed as: Among them, L cross-entropy is the loss function, n is the number of training samples, P(s n |X n ) is the model in a given image X n In the case of , the prediction beam vector index s n To further improve the generalization ability of the model, the Adam optimizer is used to dynamically adjust the model parameters to ensure that the model converges quickly during training; S24: Select the beamforming vector with the highest probability as the output result.

4. The visually assisted millimeter wave beam prediction method according to claim 2, characterized in that: In step S2, the collected image data is preprocessed, specifically including the following steps: S201: Divide the training data set into simple samples and difficult samples, where the simple samples include images taken under normal lighting conditions and the difficult samples include images taken under low light conditions. represents the divided normal light image dataset, Represents the divided low-light data set; the sample distributions of the two subsets are P normal , P low ; S202: adopting a course training method, wherein the training process starts with image data under normal lighting conditions, and gradually introduces images taken under low-light conditions to improve the beam prediction accuracy of the neural network model under low-light conditions, and increases the learning ability of the network model for complex low-light samples by adding low-light data to the training set in stages; Specifically, the training process is divided into V stages. In each stage, a low-light dataset with more difficult learning is added to the training set. Let λ i represents the random sampling probability of adding low-light samples to the training set in the i-th stage; represents the training set of the i-th stage, P i represents the sample distribution of the training set in the i-th stage, then and P i It is expressed as: P i ∝P normal +λ i ·P low in, Represents a randomly sampled data set The total number of λ i To form a new data set, the training set in the final stage contains all normal light and low light data sets; Let λ1 = 0 and This indicates that in the first stage of training, all normal light images are used as the initial training set to train the model, and in subsequent stages, low light samples are uniformly added to the training set. to the training set to form a new training set; finally, the optimal beam is predicted on the entire data set.

Images