A method of environmental semantic extraction and channel prediction based on computer vision

Abstract

The present invention provides a method for environmental semantic extraction and channel prediction based on computer vision. The method comprises: deploying a channel measurement system in the environment to be tested, wherein the channel measurement system obtains channel detection data and RGB image data in the environment to be tested; preprocessing the channel detection data and RGB image data to obtain preprocessed channel characteristic parameters and RGB images; constructing an environmental semantic extraction model, inputting the preprocessed RGB image data into the environmental semantic extraction model, and obtaining image data with environmental semantics; constructing and training a channel prediction model, inputting image data with environmental semantics into the trained channel prediction model, and obtaining corresponding channel prediction results. The method of the present invention explores the influence of the physical environment on the channel and makes full use of environmental semantic information, which not only saves computing overhead and storage overhead, but also improves the prediction accuracy, speed and generalization performance of the model.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and in particular to a computer vision-based environment semantic extraction and channel prediction method. Background Art

[0002] The future 6G network will be committed to building an intelligent wireless communication system with efficient use of spectrum resources and greatly improved energy efficiency. The integration of communication and AI (Artificial Intelligence) has also become an important technological development trend. With the explosion of data and the vigorous development of computer technology, a large number of top AI algorithms represented by CV (Computer Vision) have emerged. At present, visually assisted wireless communication with the help of CV technology has been applied in many communication tasks, and this method has been proven to achieve better system performance. Channel prediction is one of the typical applications in the field of wireless communications. Accurate channel prediction can provide key guidance for power control, link budget, network planning, etc.

[0003] At present, most of the channel prediction methods in the prior art rely on simple but low-precision channel statistical measurements or high-precision but complex channel deterministic simulations, which cannot take into account the computational overhead, prediction accuracy, and implementation efficiency at the same time. In addition, since the channel is determined by the propagation environment, the mapping relationship between the channel and the environment should exist objectively. However, the classical method can only characterize the channel from the time, space, and frequency domains, and cannot further explore the mapping relationship between channel characteristics and the propagation environment.

[0004] Due to the limitations of the channel prediction methods in the existing technology, CV technology is applied to wireless channel research. On the one hand, CV technology introduces visual perception data, which is an out-of-band data and does not occupy additional spectrum resources, which is conducive to reducing communication overhead and improving spectrum efficiency; on the other hand, the characteristics of the propagation environment (such as the size, shape and position of the backscatterer, etc.) can be recorded in the form of visual data, which can effectively indicate the wireless propagation characteristics.

[0005] At present, some scholars have carried out research on vision-assisted wireless communications, but most of these works directly use raw environmental visual data to handle various communication tasks without deeply exploring the semantic features of the environment. This will undoubtedly generate a large amount of redundant information, thereby reducing training efficiency, leading to greater computing and storage burdens, and limiting system performance. Although there are a small number of works aimed at extracting environmental semantics to assist wireless communications research, these studies all focus on tasks such as beam selection, active switching, and blocking prediction for millimeter wave or terahertz multi-antenna systems. They do not have universality across all frequency bands and multiple scenarios, and almost do not involve research on channel prediction. Summary of the invention

[0006] The present invention provides an environmental semantic extraction and channel prediction method based on computer vision, so as to effectively explore the influence of the physical environment on the channel.

[0007] In order to achieve the above object, the present invention adopts the following technical scheme.

[0008] A computer vision-based environment semantic extraction and channel prediction method, comprising:

[0009] Deploy a channel measurement system in the environment to be tested, wherein the channel measurement system obtains channel detection data and RGB image data in the environment to be tested;

[0010] Preprocessing the channel detection data and the RGB image data to obtain preprocessed channel characteristic parameters and an RGB image;

[0011] Construct an environmental semantics extraction model, input the preprocessed RGB image data into the environmental semantics extraction model, and obtain image data with environmental semantics;

[0012] Construct and train a channel prediction model, input image data with environmental semantics into the trained channel prediction model, and obtain the corresponding channel prediction results.

[0013] Preferably, a channel measurement system is deployed in the environment to be tested, and the channel measurement system obtains channel detection data and RGB image data in the environment to be tested.

[0014] A channel detection system is provided, including a signal transmitting end, a signal receiving end and an RGB camera, wherein the signal transmitting end and the RGB camera are placed on one side of a measuring road, and the RGB camera is fixed on one side of the signal transmitting end through a camera bracket with an adjustable height;

[0015] The signal transmitting end is used to generate a vector detection signal and transmit the vector detection signal into the propagation environment. The signal receiving end is fixed on the car and moves back and forth along the measurement road to continuously receive and store the vector detection signal. The RGB camera is used to capture an RGB image containing propagation environment information.

[0016] Preferably, the preprocessing of the channel detection data and the RGB image data to obtain the preprocessed channel characteristic parameters and the RGB image includes:

[0017] Performing data synchronization and data filtering on the channel sounding data and the RGB image data, wherein the data synchronization refers to making one-to-one correspondence between the channel sounding data and the RGB image data with inconsistent data sampling rates by aligning timestamps, and the data filtering refers to deleting the channel sounding data and the RGB image data that do not meet the requirements;

[0018] The pre-processed channel detection data is converted into channel transfer function and channel impulse response through Fourier transform and inverse Fourier transform to obtain relevant channel characteristic parameters. The calculation formulas of path loss and received power are as follows:

[0019]

[0020] P r =P t -PL-L system +G (2)

[0021] Where PL is the path loss, N is the number of sampling points, H is the channel transfer function, and P r and P t are the receiving power and transmitting power, L system is the system loss and G is the antenna gain.

[0022] Preferably, the step of constructing an environmental semantics extraction model, inputting preprocessed RGB image data into the environmental semantics extraction model to obtain image data with environmental semantics, includes:

[0023] A YOLOv8 network with pre-trained weights is pre-trained on the COCO dataset and used as an environmental semantic extraction model. The pre-trained weights include the parameters of all neural network layers of the YOLOv8 network.

[0024] The preprocessed RGB image data is input into the pre-trained YOLOv8 network. The pre-trained YOLOv8 network uses computer vision methods to identify dynamic antiscatterers and static antiscatterers in the test environment, and obtains image data with environmental semantics. Antiscatterers whose positions change continuously in RGB images of different frames are called dynamic antiscatterers; antiscatterers whose positions remain unchanged in RGB images of different frames are called static antiscatterers.

[0025] The image data with environmental semantics includes a type based on target recognition, a type based on instance segmentation, and a type with a binary mask added. The environmental semantic images of the type based on target recognition, the type based on instance segmentation, and the type with a binary mask added are sequentially combined with corresponding channel characteristic parameters to obtain an environmental semantic image-channel data set. The environmental semantic image-channel data set is divided into a training set, a verification set, and a test set according to a certain division ratio. The training set, the verification set, and the test set are used to train, verify, and test a pre-trained YOLOv8 network to obtain a trained YOLOv8 network as an environmental semantic extraction model.

[0026] Preferably, the environmental semantic image based on target recognition includes adding a rectangular bounding box along the contour edge of the dynamic antiscatterer, and no processing is performed on the static antiscatterer; the environmental semantic image based on instance segmentation includes using pixel blocks of different colors to cover the contours of dynamic antiscatterers of different categories, and the static antiscatterer remains unchanged; the environmental semantic image with added binary mask includes adding a binary mask to the preprocessed RGB image, wherein white pixels are used to fill the contour of the dynamic scatterer, and the static scatterer is completely covered by black pixels.

[0027] Preferably, the constructing and training of the channel prediction model, inputting image data with environmental semantics into the trained channel prediction model, and obtaining corresponding channel prediction results, includes:

[0028] A channel prediction model based on the ResNet-18 network is constructed. The ResNet-18 network is an 18-layer convolutional neural network based on residual blocks, in which the convolutional layer has 17 layers, including 1 7×7 convolution kernel and 16 3×3 convolution kernels, and the fully connected layer has 1 layer and only 1 output;

[0029] The ResNet-18 network is trained, verified and tested using the training set, the validation set and the test set, the environment semantic image is used as input data of the ResNet-18 network, and the channel characteristic parameters output by the ResNet-18 network are used as labels. During the training process, the loss function is a root mean square loss function, and a trained YOLOv8 network as an environment semantic extraction model is obtained;

[0030] The image data with environmental semantics corresponding to the channel detection data to be identified collected in the test environment is input into the trained YOLOv8 network, and the trained YOLOv8 network outputs the channel characteristic parameters of the channel detection data to be identified.

[0031] It can be seen from the technical solutions provided by the above embodiments of the present invention that the method of the present invention explores the impact of the physical environment on the channel and makes full use of the environmental semantic information, which not only saves computing and storage overheads, but also improves the prediction accuracy, speed and generalization performance of the model, thereby realizing the characterization of the channel from a new dimension. In addition, the method is not limited by frequency bands and communication scenarios, and can be applied to a variety of downstream communication tasks, with a relatively broad application prospect.

[0032] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0034] Figure 1 A processing flow chart of an environmental semantic extraction and channel prediction method based on computer vision provided in an embodiment of the present invention.

[0035] Figure 2 A flowchart of a computer vision-based environmental semantic extraction and channel prediction method provided in Embodiment 1 of the present invention.

[0036] Figure 3 A schematic diagram of the composition of an actual measurement scene photo and RGB image data set provided in Example 1 of the present invention.

[0037] Figure 4 An image with three types of environmental semantics is provided in the first embodiment of the present invention.

[0038] Figure 5 This is a diagram of the received power prediction results of Experiment 1 provided in Example 1 of the present invention.

[0039] Figure 6 This is a graph of RMSE results of Experiment 1 provided in Example 1 of the present invention.

[0040] Figure 7 This is a diagram of the received power prediction results of Experiment 2 provided in Example 1 of the present invention.

[0041] Figure 8 This is a graph of the RMSE results of Experiment 2 provided in Example 1 of the present invention.

[0042] Fig. 9This is a diagram of the received power prediction results of Experiment 3 provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0043] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0044] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0045] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0046] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0047] The processing flow of a computer vision-based environment semantic extraction and channel prediction method provided by an embodiment of the present invention is as follows: Figure 1 As shown, the following steps are included:

[0048] Step S1: deploying a channel measurement system in the environment to be tested, and the channel measurement system acquires channel detection data and RGB image data in the environment to be tested.

[0049] Step S2: preprocessing the channel detection data and RGB (red (R), green (G), blue (B)) image data to obtain preprocessed channel characteristic parameters and RGB images;

[0050] Step S3: construct an environmental semantics extraction model, input the preprocessed RGB image data into the environmental semantics extraction model, and obtain image data with environmental semantics;

[0051] Step S4: construct and train a channel prediction model, input image data with environmental semantics into the trained channel prediction model, and obtain corresponding channel prediction results.

[0052] Furthermore, the above step S1 includes the following sub-steps:

[0053] Step S101: Building a channel measurement system.

[0054] The channel detection system includes a signal transmitting end, a signal receiving end and an RGB camera. The signal transmitting end is used to generate a vector detection signal and transmit the vector detection signal to the propagation environment, the signal receiving end is used to receive the vector detection signal and store the vector detection signal on a hard disk, and the RGB camera is fixed to one side of the signal transmitting end through a camera bracket with adjustable height, and is used to capture an RGB image containing propagation environment information.

[0055] Step S102: Carry out vehicle-to-infrastructure (V2I) channel measurement activities to obtain channel detection data and RGB image data.

[0056] The vehicle-to-infrastructure (V2I) channel measurement activity specifically refers to: the signal transmitter and the RGB camera are statically placed on one side of the measurement road, continuously transmitting vector detection signals and taking RGB images; the signal receiver is fixed on the car and moves back and forth along the measurement road, and continuously receives the vector detection signal.

[0057] The acquisition of channel detection data and RGB image data specifically refers to: the signal transmitting end, the signal receiving end and the RGB camera are turned on and operated in sequence within a short time interval, the vector detection signal is received and stored by the signal receiving end after being propagated through the physical environment, and the RGB image data containing the physical environment information is stored in the RGB camera.

[0058] Furthermore, the above step S2 includes the following sub-steps:

[0059] Step S201: Data synchronization and data filtering.

[0060] Human errors will inevitably be introduced in actual V2I channel measurement activities. To ensure data availability, the raw data needs to be preprocessed, mainly including data synchronization and data filtering.

[0061] The data synchronization refers to the one-to-one correspondence between the channel detection data and RGB image data with inconsistent data sampling rates by means of timestamp alignment. The data filtering refers to the deletion of the channel detection data and RGB image data that do not meet the requirements, mainly including all channel detection data and RGB image data that cannot be aligned by timestamp, RGB image data of the signal receiving end that does not appear in the viewing angle and the channel detection data at the corresponding time, and the channel detection data of the signal receiving end in a stationary state and the RGB image data at the corresponding time.

[0062] Step S202: Acquire channel characteristic parameters.

[0063] Acquiring the channel characteristic parameters refers to converting the preprocessed channel detection data into a channel transfer function and a channel impulse response through Fourier transform and inverse Fourier transform, thereby obtaining relevant channel characteristic parameters, such as received power, path loss, etc. The calculation formulas for path loss and received power are shown below:

[0064]

[0065] P r =P t -PL-L system +G (2)

[0066] Where PL is the path loss, N is the number of sampling points, H is the channel transfer function, and P r and P t are the receiving power and transmitting power, L system is the system loss and G is the antenna gain.

[0067] Furthermore, the above step S3 includes the following sub-steps:

[0068] Step S301: construct an environmental semantic extraction model based on the YOLOv8 network.

[0069] The environmental semantic extraction model specifically refers to a YOLOv8 network with pre-trained weights that is pre-trained on the open source COCO (Microsoft Common Objects in Context) dataset.

[0070] The YOLOv8 network is a deep learning-based, open source, top-level computer vision model that is widely used in visual processing tasks such as target detection, instance segmentation, and adding masks.

[0071] The COCO dataset is an open source, large-scale, rich object detection, segmentation, and captioning dataset. The dataset contains 80 categories of objects commonly seen in daily life and has more than 330,000 images. It is widely used to train various CV models and is one of the most commonly used large datasets in the world.

[0072] The pre-trained weights include the parameters of all neural network layers such as Head, Neck, and Backbone of the YOLOv8 network, which are obtained after large-scale training on the COCO dataset. By loading these pre-trained weights, the convergence speed of the YOLOv8 network can be accelerated, and a transfer learning method is provided, so that the YOLOv8 network can realize functions such as target detection on the constructed image dataset, thereby reducing the complicated training overhead.

[0073] Step S302: define environmental semantics to characterize the propagation environment.

[0074] The environmental semantics refers to the propagation environment features captured in the RGB image, which mainly refers to the shape, size, number and spatial position of the antiscatterers that will have a significant impact on the propagation of radio waves. The antiscatterers whose positions change continuously in the RGB images of different frames are called dynamic antiscatterers; on the contrary, the antiscatterers whose positions do not change in the RGB images of different frames are static antiscatterers.

[0075] The extraction of environmental semantics refers to highlighting dynamic antiscatterers through computer vision methods so that deep neural networks can quickly and accurately identify and learn the characteristics of dynamic antiscatterers. At the same time, static antiscatterers will also present different characteristics with different computer vision methods.

[0076] Step S303: Based on the YOLOv8 network, the preprocessed RGB image data is input to obtain image data with environmental semantics.

[0077] The actual V2I channel measurement scenario is relatively simple, with only pedestrians and cars appearing. Therefore, the pre-trained YOLOv8 network can be directly used to identify all dynamic scatterers in the propagation environment without the need for additional network training. When the pre-processed RGB image data is input into the YOLOv8 network, image data with environmental semantics can be generated.

[0078] The YOLOv8 network has different visual processing methods, based on which there are three types of image data with environmental semantics. The first is a method based on target recognition, which adds a rectangular bounding box (BBox) along the contour edge of the dynamic antiscatterer to achieve the purpose of highlighting, and does not do any processing on the static antiscatterer; the second is a method based on instance segmentation, that is, using pixel blocks of different colors to cover the contours of different categories of dynamic antiscatterers, and the static antiscatterer remains unchanged; the third is to add a binary mask to the preprocessed RGB image, in which white pixels are used to fill the contours of the dynamic scatterer, while the static scatterer is completely covered by black pixels.

[0079] Step S304: construct three types of environmental semantic image-channel datasets and divide them into a training set, a validation set and a test set in sequence.

[0080] The environmental semantic image-channel dataset refers to combining three types of environmental semantic images with corresponding channel characteristic parameters in sequence to obtain an environmental semantic image-channel dataset, and the environmental semantic image-channel dataset is divided into three datasets of training, verification and testing according to a certain division ratio. In each dataset, the division ratios of the training set, verification set and test set are 0.8, 0.1 and 0.1 respectively.

[0081] The pre-trained YOLOv8 network is trained, verified and tested using the training set, validation set and test set to obtain a trained YOLOv8 network as an environmental semantic extraction model. In all data sets, the environmental semantic image will be used as the "input" of the deep neural network in the environmental semantic extraction model, and the channel characteristic parameters will be used as the "output" or "label" of the deep neural network in the environmental semantic extraction model.

[0082] Furthermore, the above step S4 includes the following sub-steps:

[0083] Step S401: construct a channel prediction model based on the ResNet-18 network.

[0084] The channel prediction model specifically refers to the ResNet-18 network, which is widely used in regression tasks. The ResNet-18 network is an 18-layer convolutional neural network based on residual blocks, in which the convolution layer has 17 layers, including 1 7×7 convolution kernel and 16 3×3 convolution kernels, and the fully connected layer has 1 layer and only 1 output. By introducing residual connections, the problems of gradient vanishing and gradient exploding in the training process can be effectively solved. The residual block can be expressed as:

[0085] H(x)=F(x)+x (3)

[0086] Among them, H(x) is the output value or observation value of the residual block, x is the input of the residual block, and F(·) is the activation function, which is generally ReLu in regression tasks, and its expression is:

[0087] y=max(0,x) (4)

[0088] Here, y refers to the output of the activation function.

[0089] Step S402: Train the constructed ResNet-18 network.

[0090] Based on the constructed ResNet-18 network, the training set and validation set in the three constructed environmental semantic image-channel datasets are taken out in turn and used for the training of the ResNet-18 network, where the environmental semantic image is the input and the channel characteristic parameters are the labels. Finally, three trained ResNet-18 networks that can process three different types of environmental semantic images can be obtained.

[0091] During the training process, the loss function is the mean square loss function (MSE). The definition of MSE is the average of the squares of the differences between the predicted values ​​and the true values. Its expression is:

[0092]

[0093] Where n is the number of samples, Y i is the predicted value, is the true value, i.e., the label. ReLu and Adam functions are selected as activation function and optimizer respectively, and the learning rate is set to 0.001; the training batch size is 64, and the number of training rounds is 50.

[0094] Step S403: Based on the trained ResNet-18 network, image data with environmental semantics is input to obtain corresponding channel prediction results.

[0095] The trained ResNet-18 network can realize the mapping relationship between environmental semantic images and wireless channels. By taking out the test set images from the three constructed environmental semantic image-channel data sets in turn and inputting them into the trained ResNet-18 network, the corresponding channel characteristic parameters can be obtained.

[0096] The image data with environmental semantics corresponding to the channel detection data to be identified collected in the test environment is input into the trained YOLOv8 network, and the trained YOLOv8 network outputs the channel characteristic parameters of the channel detection data to be identified.

[0097] Embodiment 1:

[0098] Table 1 Example parameter configuration

[0099]

[0100] This embodiment provides a method for environmental semantic extraction and channel prediction based on computer vision. The complete channel detection process is as follows: Figure 2 Specifically, the process includes the following steps:

[0101] Step S1: deploy a channel measurement system to obtain channel detection data and RGB image data.

[0102] Step S101: Building a channel measurement system.

[0103] In an embodiment of the present invention, the channel detection system includes a signal transmitting end, a signal receiving end and an RGB camera. The signal transmitting end is used to generate a vector detection signal and transmit it to a propagation environment, the signal receiving end is used to receive the vector detection signal and store it on a hard disk, and the RGB camera is fixed to one side of the signal transmitting end through a camera bracket with adjustable height, and is used to capture an RGB image containing propagation environment information.

[0104] Step S102: Carry out vehicle-to-infrastructure (V2I) channel measurement activities to obtain channel detection data and RGB image data.

[0105] In the embodiments of the present invention, channel measurement activities are carried out in three typical V2I street scenarios, in which the transmitter and the camera are fixed on one side of the street, and the receiver is installed on a car and moves back and forth along a fixed route. During this process, the receiver continuously receives and stores channel detection signals, and the camera continuously captures the dynamic behavior of the receiving vehicle and the characteristic information of the surrounding environment and generates RGB images. Therefore, in each V2I scenario, a set of data sets about channel detection signals and RGB images will be obtained, and the data sets in all scenarios constitute the database for subsequent model training and testing. Figure 3 A schematic diagram of the composition of an actual measurement scene photo and RGB image data set provided in Example 1 of the present invention.

[0106] Furthermore, in order to fully train the model to obtain higher prediction accuracy and verify the generalization of the model, a large amount of RGB image data and channel detection data were obtained in three different street scenes, such as Figure 3 As shown, in streets 1 and 2, the RGB images are from daytime and nighttime scenes respectively, while in street 3, the RGB images are only from daytime scenes.

[0107] Step S2: Perform data preprocessing to obtain preprocessed channel characteristic parameters and RGB images.

[0108] Step S201: Data synchronization and data filtering.

[0109] In an embodiment of the present invention, data preprocessing mainly includes data synchronization and data filtering. Since the sampling rates of channel detection data and RGB image data are not consistent, which are 8 snapshots / second and 30 frames / second respectively, it is necessary to use the method of timestamp alignment to make the two types of data correspond one by one, so as to facilitate subsequent model training. Channel detection data and RGB image data that do not meet the requirements are directly deleted, that is, data filtering, mainly including channel detection data and RGB image data that cannot be aligned by timestamp, RGB image data and its corresponding channel detection data that can achieve timestamp alignment but the receiving end vehicle does not appear in the field of view, and channel detection data and its corresponding RGB image data that can achieve timestamp alignment but are in a stationary state.

[0110] Step S202: Acquire channel characteristic parameters.

[0111] In the embodiment of the present invention, the received power is used as the channel characteristic parameter to be predicted, which can be extracted from the channel detection data. The calculation formula of the received power is as follows:

[0112]

[0113] P r =P t -PL-L system +G (2)

[0114] Where PL is the path loss, N is the number of sampling points, H is the channel transfer function, and P r and P t are the receiving power and transmitting power, L system is the system loss and G is the antenna gain.

[0115] After data synchronization, filtering, and received power extraction, a data set of channel characteristic parameters and RGB images that can be mapped to each other and meet the requirements of model training and testing is obtained.

[0116] Step S3: construct an environmental semantics extraction model, input the preprocessed RGB image data, and obtain image data with environmental semantics.

[0117] Step S301: construct an environmental semantic extraction model based on the YOLOv8 network.

[0118] As a top open source CV model, the YOLOv8 network has powerful image processing capabilities and is widely used in visual processing tasks such as target detection, instance segmentation, and mask addition. In an embodiment of the present invention, a YOLOv8 network pre-trained on the COCO dataset with pre-trained weights is used as an environmental semantic extraction model.

[0119] The COCO dataset is an open source, large, and rich object detection, segmentation, and captioning dataset. The dataset contains 80 categories of objects commonly seen in daily life and has more than 330,000 images. It is widely used to train various CV models and is one of the most commonly used large datasets in the world.

[0120] The pre-trained weights include the parameters of all neural network layers of the YOLOv8 network, such as Head, Neck, and Backbone. These parameters are obtained after large-scale training on the COCO dataset. By loading these pre-trained weights, the convergence speed of the YOLOv8 network can be accelerated, and a transfer learning method is provided, so that the YOLOv8 network can achieve functions such as target detection on the constructed image dataset, thereby reducing the complicated training overhead.

[0121] It should be noted that the YOLOv8 network used in the embodiments of the present invention is consistent with the standard architecture, and the hyperparameters, network structure, etc. are not changed.

[0122] Step S302: define environmental semantics to characterize the propagation environment.

[0123] In the embodiments of the present invention, environmental semantics refers to the propagation environment features captured in the RGB image, which mainly refers to the features of the antiscatterers that will affect the propagation of radio waves, including shape, size, quantity and spatial position, etc., wherein the antiscatterers are divided into dynamic (such as vehicles, pedestrians) and static (such as streets, buildings) antiscatterers. Environmental semantics extraction refers to highlighting the dynamic antiscatterers through the CV method, so that the subsequent channel prediction model can quickly identify and learn the features of the antiscatterers. At the same time, the static antiscatterers will also show different features with different CV methods.

[0124] Step S303: Based on the YOLOv8 network, the preprocessed RGB image data is input to obtain image data with environmental semantics.

[0125] The actual V2I channel measurement scenario is relatively simple, with basically only pedestrians and cars appearing. Therefore, the pre-trained YOLOv8 network can be directly used to identify all dynamic scatterers in the propagation environment without the need for additional network training. The input of the YOLOv8 network is pre-processed RGB image data, which contains feature information of anti-scatterers in the environment that can have a greater impact on radio wave propagation; the output of the model is image data with environmental semantics. In an embodiment of the present invention, based on different visual processing methods of the YOLOv8 network, image data with three types of environmental semantics are generated in sequence for the same original RGB image data for subsequent prediction of received power, such as Figure 4 shown.

[0126] The first is a method based on target recognition, which adds a rectangular bounding box along the edge of the contour of the dynamic antiscatterer to achieve the purpose of highlighting, while no processing is done on the static antiscatterer; the second is a method based on instance segmentation, which uses pixel blocks of different colors to cover the contours of dynamic antiscatterers of different categories, while the static antiscatterer remains unchanged; the third is to add a binary mask to the preprocessed RGB image, in which white pixels are used to fill the contours of the dynamic scatterer, while the static scatterer is completely covered by black pixels.

[0127] Step S304: construct three types of environmental semantic image-channel datasets and divide them into a training set, a validation set and a test set in sequence.

[0128] The environmental semantic image-channel dataset refers to three types of environmental semantic images and the same channel characteristic parameters, which can be used for deep neural network training, verification and testing. In each dataset, the ratio of training set, verification set and test set is 0.8, 0.1 and 0.1. In all datasets, environmental semantic images will be used as the "input" of the deep neural network, and the channel characteristic parameters will be used as the "output" or "label" of the deep neural network.

[0129] Step S4: construct and train a channel prediction model, and based on the trained channel prediction model, input image data with environmental semantics to obtain corresponding channel prediction results.

[0130] Step S401: construct a channel prediction model based on the ResNet-18 network.

[0131] In an embodiment of the present invention, a ResNet-18 network based on residual blocks is used as a channel prediction model to extract features of images with environmental semantics and realize the prediction of received power.

[0132] Since the received power prediction can be regarded as a regression task, the output of the channel prediction model should have only one specific value, namely the received power value. Therefore, the last fully connected layer of the ResNet-18 network used was replaced with a fully connected layer with only one neuron output, and the rest of the structure was consistent with the standard architecture, that is, it had 18 layers of neural networks, of which there were 17 convolutional layers, including 1 7×7 convolution kernel and 16 3×3 convolution kernels, and only 1 fully connected layer.

[0133] The ResNet-18 network can effectively solve the problems of gradient vanishing and gradient exploding during training by introducing residual connections. The residual block can be expressed as

[0134] H(x)=F(x)+x (3)

[0135] Among them, H(x) is the output value or observation value of the residual block, x is the input of the residual block, and F(·) is the activation function, which is generally ReLu in regression tasks, and its expression is

[0136] y=max(0,x) (4)

[0137] Here, y refers to the output of the activation function.

[0138] Step S402: Train the constructed ResNet-18 network.

[0139] Based on the constructed ResNet-18 network, the training set and validation set of the three constructed environmental semantic image-channel datasets are taken out in turn and used for network training, where the environmental semantic image is the input and the channel characteristic parameters are the labels. Finally, three trained ResNet-18 networks that can process three different types of environmental semantic images can be obtained.

[0140] During the training process, the hyperparameters of the model are shown in Table 1, where the loss function is the mean square loss function (MSE). The definition of MSE is the average of the squares of the differences between the predicted value and the true value, and its expression is

[0141]

[0142] Where n is the number of samples, Y i is the predicted value, is the true value, i.e., the label. ReLu and Adam functions are selected as activation function and optimizer respectively, and the learning rate is set to 0.001; the training batch size is 64, and the number of training rounds is 50.

[0143] Step S403: Based on the trained ResNet-18 network, image data with environmental semantics is input to obtain corresponding channel prediction results.

[0144] The trained ResNet-18 network can realize the mapping relationship between the environmental semantic image and the wireless channel. The test set images in the three constructed environmental semantic image-channel data sets are taken out in turn and input into the trained ResNet-18 network to obtain the corresponding received power value.

[0145] In order to verify the performance of the method in the embodiment of the present invention, a series of verification experiments were carried out. Three types of images with environmental semantics were generated by the trained YOLOv8 network and input into the trained ResNet-18 network to predict the received power. In this way, the influence of the three types of environmental semantics on the prediction accuracy of the model can be compared. The root mean square error (RMSE) is used to measure the prediction performance of the model.

[0146] Experiment 1 is a scene self-verification. The data from the daytime scenes of 3 streets constitute the total data set, with a total of 13905 sets of data. The ratios of the training set, validation set and test set are 0.8, 0.1 and 0.1 respectively. The test set is composed of independent data from the 3 streets, while the training set and validation set are a mixture of other data from the 3 streets. Taking Street 1 as an example, the prediction results of the received power and the RMSE results are as follows: Figure 5 and Figure 6 As shown. For Street 1, when images with three types of environmental semantics, such as bounding boxes, instance segmentation, and binary masks, are used as inputs to the channel prediction model to predict the received power, the RMSEs are 2.27, 1.93, and 3.71 dB, respectively. For Streets 2 and 3, the RMSEs are 1.53, 1.47, 2.83 and 1.85, 1.51, 3.26 dB, respectively. It can be found that the received power is predicted relatively accurately, and the RMSE is no more than 4 dB, which means that when using the method in the embodiment of the present invention, the received power can be predicted relatively accurately only by using the RGB image of the propagation environment. In addition, compared with images with bounding boxes and binary masks, the best received power prediction accuracy can be obtained when the image after instance segmentation is used as the input of ResNet-18, which means that the image after instance segmentation can better represent the environmental semantics, and the characteristic information of the propagation environment contained therein can be more easily learned by ResNet-18.

[0147] Experiment 2 is a day / night cross-validation. All data from the daytime scenes of the three streets constitute the training set, a total of 13,905 sets of data, and all data from the night scenes of streets 1 and 2 are used as the test set, a total of 4,841 sets of data. Taking street 1 as an example, the prediction results of received power and RMSE results are as follows: Figure 7 and Figure 8In order to show the results more clearly and intuitively, only 10% of the data was randomly selected and placed in Figure 6 In. From Figure 6 It can be seen that although the ResNet-18 network is trained on the daytime data set, whether it is Street 1 or Street 2, when the night RGB image is input, the model can still accurately predict the received power. Moreover, when the image after instance segmentation is used as the input of the ResNet-18 network, the prediction accuracy of the received power is still the highest. Combining experiments 1 and 2, it can be seen that for the channel prediction network trained on the same data set, regardless of whether the image of the test set is from daytime or nighttime, the final model can achieve good received power prediction, and their prediction accuracy is not much different, which also proves to a certain extent that the proposed model has good generalization and robustness.

[0148] Experiment 3 is a scene cross-validation. The data from the daytime and nighttime scenes of Street 1 and Street 2 constitute the training set, with a total of 8739 sets of data, and the data from the daytime scenes of Street 3 is used as the test set, with a total of 5964 sets of data. The purpose of this experiment is to verify the generalization ability of the trained model on a dataset from a completely new scene. The prediction results of the received power are shown in Figure 2. Fig. 9 As shown in Figure 2, their RMSEs are 4.46, 3.42, and 3.76 dB, respectively. It can be seen that the image segmented by the instance still has the highest prediction accuracy when used as the input of the channel prediction model. Compared with the first two experiments, the prediction accuracy of Experiment 3 has decreased, but this difference may be caused by the data set. The actual value of the received power in the training set fluctuates between -30 dB and 0 dB, as shown in Figure 2. Figure 4 As shown, in the test set, it varies between -50dB and -20dB, as shown in Figure 8 As shown in the figure, the inconsistency of the received power range in the training and test sets leads to a decrease in the model prediction performance. However, the RMSE remains between 3.42-4.46 dB, and the model is still able to predict the general trend of the received power, which is enough to show that the model has good generalization ability.

[0149] In summary, the present invention proposes a method for environmental semantic extraction and channel prediction based on computer vision, which can predict channel characteristic parameters only by transmitting RGB images of the environment, and has high prediction accuracy and low model complexity, which is better than the classic channel prediction method. This method also explores the impact of the physical environment on the channel and makes full use of environmental semantic information, which not only saves computing and storage overheads, but also improves prediction accuracy, speed and generalization performance, thereby realizing the characterization of the channel from a new dimension. In addition, this method is not limited by frequency bands and communication scenarios, and can be applied to a variety of downstream communication tasks, with a relatively broad application prospect.

[0150] The computer vision-based environmental semantic extraction and channel prediction method of the embodiment of the present invention breaks through the limitations of time, space, and frequency domains in the classical channel prediction method, but establishes a mapping relationship from the propagation environment to the channel, and further proposes the concept and extraction method of environmental semantics for channel prediction tasks, so that the channel characteristic parameters can be predicted only through the image of the propagation environment. Compared with the classical channel prediction method, this method is simple to implement and has a low model complexity. At the same time, it can ensure a high prediction accuracy. Moreover, the visual data used by this method is out-of-band data, which does not occupy additional spectrum resources, can reduce communication overhead, and improve spectrum utilization. Compared with other visual-assisted wireless communication tasks, this method focuses on extracting semantic information from environmental images and ignores other environmental information that has little impact on radio wave propagation, greatly reducing the redundancy of visual data, saving computing overhead and storage overhead. At the same time, this method is not limited by frequency bands and communication scenarios and has strong generalization.

[0151] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0152] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0153] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0154] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for environmental semantic extraction and channel prediction based on computer vision, characterized in that: include: Deploy a channel measurement system in the environment to be tested, wherein the channel measurement system obtains channel detection data and RGB image data in the environment to be tested; Preprocessing the channel detection data and the RGB image data to obtain preprocessed channel characteristic parameters and an RGB image; Construct an environmental semantics extraction model, input the preprocessed RGB image data into the environmental semantics extraction model, and obtain image data with environmental semantics; Construct and train a channel prediction model, input image data with environmental semantics into the trained channel prediction model, and obtain the corresponding channel prediction results; The construction of the environment semantics extraction model, inputting the preprocessed RGB image data into the environment semantics extraction model to obtain image data with environment semantics, includes: A YOLOv8 network with pre-trained weights is pre-trained on the COCO dataset and used as an environmental semantic extraction model. The pre-trained weights include the parameters of all neural network layers of the YOLOv8 network. The preprocessed RGB image data is input into the pre-trained YOLOv8 network. The pre-trained YOLOv8 network uses computer vision methods to identify dynamic antiscatterers and static antiscatterers in the test environment, and obtains image data with environmental semantics. Antiscatterers whose positions change continuously in RGB images of different frames are called dynamic antiscatterers; antiscatterers whose positions remain unchanged in RGB images of different frames are static antiscatterers. The image data with environmental semantics includes a type based on target recognition, a type based on instance segmentation, and a type with a binary mask added. The environmental semantic images of the type based on target recognition, the type based on instance segmentation, and the type with a binary mask added are sequentially combined with corresponding channel characteristic parameters to obtain an environmental semantic image-channel data set. The environmental semantic image-channel data set is divided into a training set, a verification set, and a test set according to a certain division ratio. The pre-trained YOLOv8 network is trained, verified, and tested using the training set, the verification set, and the test set to obtain a trained YOLOv8 network as an environmental semantic extraction model; The target recognition-based environment semantic image includes adding a rectangular bounding box along the contour edge of the dynamic antiscatterer, and no processing is performed on the static antiscatterer; the instance segmentation-based environment semantic image includes using pixel blocks of different colors to cover the contours of dynamic antiscatterers of different categories, and the static antiscatterer remains unchanged; the binary mask-added environment semantic image includes adding a binary mask to the preprocessed RGB image, wherein white pixels are used to fill the contour of the dynamic scatterer, and the static scatterer is completely covered by black pixels; The said constructing and training the channel prediction model, inputting the image data with environmental semantics into the trained channel prediction model, and obtaining the corresponding channel prediction result, includes: A channel prediction model based on the ResNet-18 network is constructed. The ResNet-18 network is an 18-layer convolutional neural network based on residual blocks, in which the convolutional layer has 17 layers, including 1 7×7 convolution kernel and 16 3×3 convolution kernels, and the fully connected layer has 1 layer and only 1 output; The ResNet-18 network is trained, verified and tested using the training set, the validation set and the test set, the environment semantic image is used as the input data of the ResNet-18 network, and the channel characteristic parameters output by the ResNet-18 network are used as labels. During the training process, the loss function is a root mean square loss function, and the trained ResNet-18 network as an environment semantic extraction model is obtained; The image data with environmental semantics corresponding to the channel detection data to be identified collected in the test environment is input into the trained ResNet-18 network, and the trained ResNet-18 network outputs the channel characteristic parameters of the channel detection data to be identified.

2. The method according to claim 1, characterized in that The channel measurement system is deployed in the environment to be tested, and the channel measurement system obtains channel detection data and RGB image data in the environment to be tested, A channel detection system is provided, including a signal transmitting end, a signal receiving end and an RGB camera, wherein the signal transmitting end and the RGB camera are placed on one side of a measuring road, and the RGB camera is fixed on one side of the signal transmitting end through a camera bracket with an adjustable height; The signal transmitting end is used to generate a vector detection signal and transmit the vector detection signal into the propagation environment. The signal receiving end is fixed on the car and moves back and forth along the measurement road to continuously receive and store the vector detection signal. The RGB camera is used to capture an RGB image containing propagation environment information.

3. The method according to claim 2, characterized in that The preprocessing of the channel detection data and the RGB image data to obtain the preprocessed channel characteristic parameters and the RGB image includes: Performing data synchronization and data filtering on the channel sounding data and the RGB image data, wherein the data synchronization refers to making one-to-one correspondence between the channel sounding data and the RGB image data with inconsistent data sampling rates by aligning timestamps, and the data filtering refers to deleting the channel sounding data and the RGB image data that do not meet the requirements; The pre-processed channel detection data is converted into channel transfer function and channel impulse response through Fourier transform and inverse Fourier transform to obtain relevant channel characteristic parameters. The calculation formulas of path loss and received power are as follows: P r =P t -PL-L system +G (2) Where PL is the path loss, N is the number of sampling points, H is the channel transfer function, and P r and P t are the receiving power and transmitting power, L system is the system loss and G is the antenna gain.

Images